3
U'ícêt ã @ sü U d Z ddlZddlZddlmZ ddlmZ ddlm Z ddl
m Z ddl
mZ dd lmZ ejrxddlZdd
lmZ ed Zejejd f ejd
ZejdZejdej Z ejdej ej B Z ejdej ej B Z!e dZ"e dZ#e dZ$e dZ%e dZ&e dZ'e dZ(e dZ)e dZ*e dZ+e d Z,e d Z-e d Z.e d Z/e d Z0e d!Z1e d"Z2e d#Z3e d$Z4e d%Z5e d&Z6e d'Z7e d(Z8e d)Z9e d*Z:e d+Z;e d,Z<e d-Z=e d.Z>e d/Z?e d0Z@e d1ZAe d2ZBe d3ZCe d4ZDe d5ZEe d6ZFe d7ZGe d8ZHe d9ZIe d:ZJe d;ZKe d<ZLe d=ZMe d>ZNe d?ZOe d@ZPe dAZQe dBZRe"e:e&e)e2e1e5e;e-e7e.e8e,e6e(e3e*e+e/e0e#e'e$e4e%e9dCZSdDdE eSjT D ZUeVeSeVeUksÈtWdFejdGdHjXdIdJ eYeSdKdL dMD dNZZe[eHeJeIe<eMeNeOgZ\e[e<ePeJeOgZ]e^e^dOdPdQZ_dRe^dSdTdUZ`e^e^dVdWdXZae^ebdYdZd[Zcd\ejdeje^e^f d]d^d_ZeG d`da daZfG dbdR dRejgZhG dcdd ddZiG dedf dfZjd\d d]dgdhZkG didj djelZmG dkdl dlejgZnG dmd d ZodS )nzúImplements a Jinja / Python combination lexer. The ``Lexer`` class
is used to do some preprocessing. It filters out invalid operators like
the bitshift operators we don't allow in templates. It separates
template code and python code in expressions.
é N)Ú literal_eval)Údeque)Úinterné )Úpattern)ÚTemplateSyntaxError)ÚLRUCache)Ú Environmenté2 ÚLexerz\s+z (\r\n|\r|\n)z7('([^'\\]*(?:\\.[^'\\]*)*)'|"([^"\\]*(?:\\.[^"\\]*)*)")z¿
(
0b(_?[0-1])+ # binary
|
0o(_?[0-7])+ # octal
|
0x(_?[\da-f])+ # hex
|
[1-9](_?\d)* # decimal
|
0(_?0)* # decimal zero
)
zü
(?<!\.) # doesn't start with a .
(\d+_)*\d+ # digits, possibly _ separated
(
(\.(\d+_)*\d+)? # optional fractional part
e[+\-]?(\d+_)*\d+ # exponent part
|
\.(\d+_)*\d+ # required fractional part
)
ÚaddZassignZcolonÚcommaÚdivÚdotÚeqÚfloordivÚgtZgteqZlbraceZlbracketZlparenÚltZlteqÚmodÚmulÚneÚpipeÚpowZrbraceZrbracketZrparenZ semicolonÚsubÚtildeÚ
whitespaceÚfloatÚintegerÚnameÚstringÚoperatorZ block_beginZ block_endZvariable_beginZ variable_endZ raw_beginZraw_endZ
comment_beginZ comment_endÚcommentZlinestatement_beginZlinestatement_endZlinecomment_beginZlinecomment_endZ linecommentÚdataÚinitialÚeof)ú+ú-ú/z//Ú*ú%z**ú~ú[ú]ú(ú)Ú{Ú}z==z!=ú>z>=ú<z<=ú=Ú.ú:ú|ú,ú;c C s i | ]\}}||qS © r9 )Ú.0ÚkÚvr9 r9 ú./tmp/pip-build-gk9425m9/Jinja2/jinja2/lexer.pyú
<dictcomp> s r> zoperators droppedr- r6 c c s | ]}t j|V qd S )N)ÚreÚescape)r: Úxr9 r9 r= ú <genexpr> s rB c C s
t | S )N)Úlen)rA r9 r9 r= Ú<lambda> s rD )Úkeyr. )Ú
token_typeÚreturnc C sL | t krt | S tdtdtdtdtdtdtdtdt dt
d t d
t d i j
| | S ) Nzbegin of commentzend of commentr! zbegin of statement blockzend of statement blockzbegin of print statementzend of print statementzbegin of line statementzend of line statementztemplate data / textzend of template)Úreverse_operatorsÚTOKEN_COMMENT_BEGINÚTOKEN_COMMENT_ENDÚ
TOKEN_COMMENTÚTOKEN_LINECOMMENTÚTOKEN_BLOCK_BEGINÚTOKEN_BLOCK_ENDÚTOKEN_VARIABLE_BEGINÚTOKEN_VARIABLE_ENDÚTOKEN_LINESTATEMENT_BEGINÚTOKEN_LINESTATEMENT_ENDÚ
TOKEN_DATAÚ TOKEN_EOFÚget)rF r9 r9 r= Ú_describe_token_type£ s rV ÚToken)ÚtokenrG c C s | j tkr| jS t| j S )z#Returns a description of the token.)ÚtypeÚ
TOKEN_NAMEÚvaluerV )rX r9 r9 r= Údescribe_token· s
r\ )ÚexprrG c C s2 d| kr&| j dd\}}|tkr*|S n| }t|S )z0Like `describe_token` but for token expressions.r5 r )ÚsplitrZ rV )r] rY r[ r9 r9 r= Údescribe_token_expr¿ s r_ )r[ rG c C s t tj| S )zsCount the number of newline characters in the string. This is
useful for extensions that filter a stream.
)rC Ú
newline_reÚfindall)r[ r9 r9 r= Úcount_newlinesÌ s rb r )Ú environmentrG c C s° t j}t| jt|| jft| jt|| jft| jt|| jfg}| j dk rp|j
t| j t d|| j f | j dk r|j
t| j t
d|| j f dd t|ddD S )zACompiles all the rules from the environment into a list of rules.Nz ^[ \t\v]*z(?:^|(?<=\S))[^\S\r\n]*c S s g | ]}|d d
qS )r Nr9 )r: rA r9 r9 r= ú
<listcomp>ù s z!compile_rules.<locals>.<listcomp>T)Úreverse)r? r@ rC Úcomment_start_stringrI Úblock_start_stringrM Úvariable_start_stringrO Úline_statement_prefixÚappendrQ Úline_comment_prefixÚTOKEN_LINECOMMENT_BEGINÚsorted)rc ÚeÚrulesr9 r9 r= Ú
compile_rulesÓ s*
rp c @ s> e Zd ZdZefeeje ddddZe edddd Z
dS )
ÚFailurezjClass that raises a `TemplateSyntaxError` if called.
Used by the `Lexer` to specify known errors.
N)ÚmessageÚclsrG c C s || _ || _d S )N)rr Ú error_class)Úselfrr rs r9 r9 r= Ú__init__ s zFailure.__init__z te.NoReturn)ÚlinenoÚfilenamerG c C s | j | j||d S )N)rt rr )ru rw rx r9 r9 r= Ú__call__ s zFailure.__call__) Ú__name__Ú
__module__Ú __qualname__Ú__doc__r ÚstrÚtÚTyperv Úintry r9 r9 r9 r= rq ü s rq c @ sH e Zd ZU eeeedddZee dddZ
ee ddd Z d
S ) rW )rG c C s t | S )N)r\ )ru r9 r9 r= Ú__str__ s z
Token.__str__)r] rG c C s2 | j |krdS d|kr.|jdd| j | jgkS dS )z¸Test a token against a token expression. This can either be a
token type or ``'token_type:token_value'``. This can only test
against string values and types.
Tr5 r F)rY r^ r[ )ru r] r9 r9 r= Útest s
z
Token.test)ÚiterablerG c s t fdd|D S )z(Test against multiple token expressions.c 3 s | ]} j |V qd S )N)r )r: r] )ru r9 r= rB $ s z!Token.test_any.<locals>.<genexpr>)Úany)ru r r9 )ru r= Útest_any" s zToken.test_anyN) rz r{ r| r rw r~ rY r[ r Úboolr r r9 r9 r9 r= rW s
c @ s< e Zd ZdZdddddZd ddd Zedd
d ZdS ) ÚTokenStreamIteratorz`The iterator for tokenstreams. Iterate over the stream
until the eof token is reached.
Ú TokenStreamN)ÚstreamrG c C s
|| _ d S )N)r )ru r r9 r9 r= rv , s z TokenStreamIterator.__init__)rG c C s | S )Nr9 )ru r9 r9 r= Ú__iter__/ s z TokenStreamIterator.__iter__c C s. | j j}|jtkr | j j tt| j |S )N)r ÚcurrentrY rT ÚcloseÚ
StopIterationÚnext)ru rX r9 r9 r= Ú__next__2 s
z TokenStreamIterator.__next__)rz r{ r| r} rv r rW r r9 r9 r9 r= r ' s r c @ sä e Zd ZdZeje eje eje dddZ e
dddZ e ddd Z
ee dd
d Zed d
ddZedddZd!ed dddZeeje dddZee dddZeddd Zd dd d Zeedd d Zd S )"r zÛA token stream is an iterable that yields :class:`Token`\s. The
parser however does not iterate over it but calls :meth:`next` to go
one token ahead. The current active token is stored as :attr:`current`.
)Ú generatorr rx c C s> t || _t | _|| _|| _d| _tdtd| _ t
| d S )NFr Ú ) ÚiterÚ_iterr Ú_pushedr rx ÚclosedrW Ú
TOKEN_INITIALr r )ru r r rx r9 r9 r= rv C s
zTokenStream.__init__)rG c C s t | S )N)r )ru r9 r9 r= r Q s zTokenStream.__iter__c C s t | jp| jjtk S )N)r r r rY rT )ru r9 r9 r= Ú__bool__T s zTokenStream.__bool__c C s | S )z Are we at the end of the stream?r9 )ru r9 r9 r= ÚeosW s zTokenStream.eosN)rX rG c C s | j j| dS )z Push a token back to the stream.N)r rj )ru rX r9 r9 r= Úpush\ s zTokenStream.pushc C s" t | }| j}| j| || _|S )zLook at the next token.)r r r )ru Z old_tokenÚresultr9 r9 r= Úlook` s
zTokenStream.lookr )ÚnrG c C s xt |D ] }t| q
W dS )zGot n tokens ahead.N)Úranger )ru r Ú_r9 r9 r= Úskiph s zTokenStream.skip)r] rG c C s | j j|rt| S dS )zqPerform the token test and return the token if it matched.
Otherwise the return value is `None`.
N)r r r )ru r] r9 r9 r= Únext_ifm s zTokenStream.next_ifc C s | j |dk S )z8Like :meth:`next_if` but only returns `True` or `False`.N)r¡ )ru r] r9 r9 r= Úskip_ifv s zTokenStream.skip_ifc C sX | j }| jr| jj | _ n:| j jtk rTyt| j| _ W n tk
rR | j Y nX |S )z|Go one token ahead and return the old one.
Use the built-in :func:`next` instead of calling this directly.
) r r ÚpopleftrY rT r r r r )ru Úrvr9 r9 r= r z s zTokenStream.__next__c C s& t | jjtd| _tf | _d| _dS )zClose the stream.r TN)rW r rw rT r r r )ru r9 r9 r= r s
zTokenStream.closec C sn | j j|sft|}| j jtkr>td|d| j j| j| jtd|dt | j | j j| j| jt
| S )z}Expect a given token type and return it. This accepts the same
argument as :meth:`jinja2.lexer.Token.test`.
z%unexpected end of template, expected r4 zexpected token z, got ) r r r_ rY rT r rw r rx r\ r )ru r] r9 r9 r= Úexpect s
zTokenStream.expect)r )rz r{ r| r} r ÚIterablerW ÚOptionalr~ rv r r r r Úpropertyr r r r r r¡ r¢ r r r¥ r9 r9 r9 r= r = s
r c C sZ | j | j| j| j| j| j| j| j| j| j | j
| j f }t j
|}|dkrVt| t |< }|S )z(Return a lexer which is probably cached.N)rg Úblock_end_stringrh Úvariable_end_stringrf Úcomment_end_stringri rk Ú trim_blocksÚ
lstrip_blocksÚnewline_sequenceÚkeep_trailing_newlineÚ _lexer_cacherU r )rc rE Úlexerr9 r9 r= Ú get_lexerª s
r² c s$ e Zd ZdZf Z fddZ ZS )ÚOptionalLStripzWA special tuple for marking a point in the state that can have
lstrip applied.
c s t j| |S )N)ÚsuperÚ__new__)rs ÚmembersÚkwargs)Ú __class__r9 r= rµ Ë s zOptionalLStrip.__new__)rz r{ r| r} Ú __slots__rµ Ú
__classcell__r9 r9 )r¸ r= r³ Â s r³ c @ sB e Zd ZU eje ejeejedf eje f
ej e dS )Ú_Rule.N)
rz r{ r| r ÚPatternr~ r ÚUnionÚTuplerq Útokensr§ Úcommandr9 r9 r9 r= r» Ï s
r» c
@ sÐ e Zd ZdZdddddZeeddd Zdeeje eje eje e d
d d Z
dej ej e
eef eje eje eje d
ddZdeeje eje eje ejej e
eef d
ddZdS )r a
Class that implements a lexer for a given environment. Automatically
created by the environment class, usually you don't have to do that.
Note that the lexer is not automatically bound to an environment.
Multiple environments can share the same lexer.
r N)rc rG c
C s0 t j}ttjt ddd}tttd ttt d tt
t d tt t
d tttd tttd g}t|}||j}||j}||j}||j} |jrdnd}
|jr¨|dnd | _|j| _|j| _d| d| d | d
} d j | gd d
|D } dt|d| dt t ddt|dt d gt t|d| d| d | |
d
t t!fdt|dt"dfd gt#t|d| d| d | |
dt$dg| t%t|d| d | t&dg| t't|d| d| d| d | |
d
t t t(dt|dt"d fd gt)t|d t*dg| t+t|d t,t-fdgi| _.d S ) N)rA rG c S s t j| t jt jB S )N)r? ÚcompileÚMÚS)rA r9 r9 r= Úcá s zLexer.__init__.<locals>.cz\n?r z[^ \t]z(?P<raw_begin>z(\-|\+|)\s*raw\s*(?:\-z\s*|z))r6 c S s" g | ]\}}d | d| dqS )z(?P<r1 z (\-|\+|))r9 )r: r Úrr9 r9 r= rd
s z"Lexer.__init__.<locals>.<listcomp>Úrootz(.*?)(?:r. z#bygroupz.+z (.*?)((?:\+z|\-z#popz(.)zMissing end of comment tagz(?:\+z\-z (.*?)((?:z(\-|\+|))\s*endraw\s*(?:\+z Missing end of raw directivez \s*(\n|$)z(.*?)()(?=\n|$))/r? r@ r~ r r¼ r» Ú
whitespace_reÚTOKEN_WHITESPACEÚfloat_reÚ TOKEN_FLOATÚ
integer_reÚ
TOKEN_INTEGERÚname_rerZ Ú string_reÚ TOKEN_STRINGÚ operator_reÚTOKEN_OPERATORrp rg r© r« rª r¬ r Úlstrip_unless_rer® r¯ Újoinr³ rS rI rK rJ rq rM rN rO rP ÚTOKEN_RAW_BEGINÚ
TOKEN_RAW_ENDrQ rR rl rL ÚTOKEN_LINECOMMENT_ENDro )
ru rc rn rÄ Z tag_rulesZroot_tag_rulesZblock_start_reZ block_end_reZcomment_end_reZvariable_end_reZblock_suffix_reZ root_raw_reZ
root_parts_rer9 r9 r= rv Ý sz
"zLexer.__init__)r[ rG c C s t j| j|S )z`Replace all newlines with the configured sequence in strings
and template data.
)r` r r® )ru r[ r9 r9 r= Ú_normalize_newlinesV s zLexer._normalize_newlines)Úsourcer rx ÚstaterG c C s& | j ||||}t| j|||||S )z:Calls tokeniter + tokenize and wraps it in a token stream.)Ú tokeniterr Úwrap)ru rØ r rx rÙ r r9 r9 r= Útokenize\ s zLexer.tokenize)r r rx rG c
c sz xr|D ]h\}}}|t kr q|}|tkr2t}n2|tkrBt}n"|ttfkrTqn|tkrj| j|} nú|dkrx|}nì|t kr|}|j
st d|||nÈ|t kry$| j|dd
j
ddjd}W nJ tk
r } z,t|jdd
j } t | ||||W Y dd}~X nX nL|tkr6t|jd d
d }n.|tkrRt|jd d
}n|tkrdt| }t|||V qW dS )zThis is called with the stream as returned by `tokenize` and wraps
every token in a :class:`Token` and converts the value.
Úkeywordz Invalid character in identifierr ÚasciiÚbackslashreplacezunicode-escaper5 Nr r r éÿÿÿÿrà )Úignored_tokensrQ rM rR rN rÔ rÕ rS r× rZ Ú isidentifierr rÏ ÚencodeÚdecodeÚ Exceptionr~ r^ ÚstriprÌ r ÚreplacerÊ r rÑ Ú operatorsrW )
ru r r rx rw rX Z value_strr[ rn Úmsgr9 r9 r= rÛ g sD
$
z
Lexer.wrapc c sZ t j|ddd
}| j r.|d! dkr.|d"= dj|}d}d}dg}|dk rt|dkrt|d#ksftd
|j|d | j|d$ } t|}
g } | j} d}
d }x¶x®| D ]x\}}}|j ||}|dkrÊq¨| rÞ|t
t t fkrÞq¨t
|tr|j }t
|trÊ|d }td
d |ddd
D }|dkr^|j }|t|d
jd}
|f|dd
}nl|dkrÊ| dk rÊ|j jt rÊ|jdd }|dks¢|rÊ| j||sÊ|d|
f|dd
}xÄt|D ]¸\}}|jtkrô|||n|dkrPx|j j D ]0\}}|dk r |||fV ||jd7 }P q W t |dn:|| } | sh|t krt||| fV || jd|
7 }d}
qÔW nÒ|j } |t kr:| dkrº| jd n| dkrÐ| jd nj| dkræ| jd nT| d%kr:| s t!d| d|||| j" } | | kr:t!d| d| d|||| sJ|t krV||| fV || jd7 }|j d&d
dk}|j# } |dk r|d kr|j" nV|dkrêxJ|j j D ] \}}|dk r¶|j| P q¶W t |d n
|j| | j|d' } n| |kr t |d | }P q¨W ||
kr4dS t!d || d | |||q W dS )(a This method tokenizes the text and returns the tokens in a
generator. Use this method if you just want to tokenize a template.
.. versionchanged:: 3.0
Only ``\n``, ``\r\n`` and ``\r`` are treated as line
breaks.
Né r r Ú
r rÆ ÚvariableÚblockz
invalid stateZ_beginTc s s | ]}|d k r|V qd S )Nr9 )r: Úgr9 r9 r= rB ß s z"Lexer.tokeniter.<locals>.<genexpr>r& r% z#bygroupz= wanted to resolve the token dynamically but no group matchedr/ r0 r- r. r+ r, z unexpected 'ú'z
', expected 'z#popzA wanted to resolve the new state dynamically but no group matchedz* yielded empty string without stack changezunexpected char z at rà rà )rì rí rà )r0 r. r, rà rà )$r` r^ r¯ rÓ ÚAssertionErrorrj ro rC rÒ ÚmatchrP rN rR Ú
isinstanceÚtupleÚgroupsr³ r ÚrstripÚcountÚ groupdictrU rO ÚrfindÚsearchÚ enumerater¸ rq ÚitemsÚ RuntimeErrorÚignore_if_emptyÚgrouprÑ r ÚpopÚend) ru rØ r rx rÙ ÚlinesÚposrw ÚstackZ statetokensZ
source_lengthZbalancing_stackrÒ Znewlines_strippedZ
line_startingÚregexr¿ Z new_stateÚmrô ÚtextZ
strip_signÚstrippedZl_posÚidxrX rE r[ r" Z expected_opÚpos2r9 r9 r= rÚ sÎ
zLexer.tokeniter)NNN)NN)NN)rz r{ r| r} rv r~ r× r r§ r rÜ r¦ r¾ r ÚIteratorrW rÛ rÚ r9 r9 r9 r= r Õ s y &5 )pr} r? Útypingr Úastr Ú collectionsr Úsysr Z _identifierr rÍ Ú
exceptionsr Úutilsr Z
TYPE_CHECKINGZtyping_extensionsÚterc r r° ÚMutableMappingr¾ rÁ rÇ r` rà rÎ Ú
IGNORECASEÚVERBOSErË rÉ Z TOKEN_ADDZ TOKEN_ASSIGNZ TOKEN_COLONZ TOKEN_COMMAZ TOKEN_DIVZ TOKEN_DOTZTOKEN_EQZTOKEN_FLOORDIVZTOKEN_GTZ
TOKEN_GTEQZ TOKEN_LBRACEZTOKEN_LBRACKETZ TOKEN_LPARENZTOKEN_LTZ
TOKEN_LTEQZ TOKEN_MODZ TOKEN_MULZTOKEN_NEZ
TOKEN_PIPEZ TOKEN_POWZ TOKEN_RBRACEZTOKEN_RBRACKETZ TOKEN_RPARENZTOKEN_SEMICOLONZ TOKEN_SUBZ TOKEN_TILDErÈ rÊ rÌ rZ rÏ rÑ rM rN rO rP rÔ rÕ rI rJ rK rQ rR rl rÖ rL rS r rT rè rû rH rC rð rÓ rm rÐ Ú frozensetrá rý r~ rV r\ r_ r rb ÚListrp rq Z
NamedTuplerW r r r² ró r³ r» r r9 r9 r9 r= Ú<module> sø
*
) m