3
V'ícó| ã
@ sê d Z ddlZddlZddlZddlmZmZ ddlmZ ddl m
Z
m Z m Z m
Z
mZ ddlmZmZmZmZmZmZ ddlmZ dd d
d d d
dddddddg
ZejdZd?d@dAdBdCgZe d d! Z G d"d# d#e Z G d$d de d%Z G d&d d e Z!G d'd
d
e"Z#G d(d) d)Z$e$ Z%G d*d+ d+e&Z'G d,d- d-Z(d.d Z)G d/d0 d0Z*e* Z+d1d Z,G d2d dZ-G d3d deZ.G d4d5 d5e Z/G d6d d e e/d%Z0G d7d d Z1G d8d
d
e0Z2d9d: Z3G d;d< d<e/Z4G d=d> d>e0e4d%Z5dS )Dz´
pygments.lexer
~~~~~~~~~~~~~~
Base lexer classes.
:copyright: Copyright 2006-2022 by the Pygments team, see AUTHORS.
:license: BSD, see LICENSE for details.
é N)Ú
apply_filtersÚFilter)Úget_filter_by_name)ÚErrorÚTextÚOtherÚ
WhitespaceÚ
_TokenType)Ú get_bool_optÚ get_int_optÚ get_list_optÚmake_analysatorÚFutureÚ guess_decode)Ú regex_optÚLexerÚ
RegexLexerÚExtendedRegexLexerÚDelegatingLexerÚ LexerContextÚincludeÚinheritÚbygroupsÚusingÚthisÚdefaultÚwordsÚline_rez.*?
ó úutf-8ó ÿþ úutf-32ó þÿúutf-32beó ÿþúutf-16ó þÿúutf-16bec C s dS )Ng © )Úxr( r( ú2/tmp/pip-build-gk9425m9/Pygments/pygments/lexer.pyÚ<lambda>" s r+ c @ s e Zd ZdZdd ZdS )Ú LexerMetaz
This metaclass automagically converts ``analyse_text`` methods into
static methods which always return float values.
c C s( d|krt |d |d< tj| |||S )NÚ analyse_text)r
ÚtypeÚ__new__)ZmcsÚnameÚbasesÚdr( r( r* r/ + s zLexerMeta.__new__N)Ú__name__Ú
__module__Ú __qualname__Ú__doc__r/ r( r( r( r* r, % s r, c @ s^ e Zd ZdZdZdZg Zg Zg Zg Z dZ
dd Z dd Z dd Z
d
d Zdd
dZdd ZdS )r aÈ
Lexer for a specific language.
Basic options recognized:
``stripnl``
Strip leading and trailing newlines from the input (default: True).
``stripall``
Strip all leading and trailing whitespace from the input
(default: False).
``ensurenl``
Make sure that the input ends with a newline (default: True). This
is required for some lexers that consume input linewise.
.. versionadded:: 1.3
``tabsize``
If given and greater than 0, expand tabs in the input (default: 0).
``encoding``
If given, must be an encoding name. This encoding will be used to
convert the input string to Unicode, if it is not already a Unicode
string (default: ``'guess'``, which uses a simple UTF-8 / Locale /
Latin1 detection. Can also be ``'chardet'`` to use the chardet
library, if it is installed.
``inencoding``
Overrides the ``encoding`` if given.
Nr c K s || _ t|dd| _t|dd| _t|dd| _t|dd| _|jdd | _|jd
pZ| j| _g | _ x t
|d f D ]}| j | qrW d S ) NÚstripnlTÚstripallFÚensurenlÚtabsizer ÚencodingÚguessZ
inencodingÚfilters) Úoptionsr
r7 r8 r9 r r: Úgetr; r= r Ú
add_filter)Úselfr> Úfilter_r( r( r* Ú__init__b s zLexer.__init__c C s( | j rd| jj| j f S d| jj S d S )Nz <pygments.lexers.%s with %r>z<pygments.lexers.%s>)r> Ú __class__r3 )rA r( r( r* Ú__repr__n s
zLexer.__repr__c K s& t |tst|f|}| jj| dS )z8
Add a new stream filter to this lexer.
N)Ú
isinstancer r r= Úappend)rA rB r> r( r( r* r@ u s
zLexer.add_filterc C s dS )a~
Has to return a float between ``0`` and ``1`` that indicates
if a lexer wants to highlight this text. Used by ``guess_lexer``.
If this method returns ``0`` it won't highlight it in any case, if
it returns ``1`` highlighting with this lexer is guaranteed.
The `LexerMeta` metaclass automatically wraps this function so
that it works like a static method (no ``self`` or ``cls``
parameter) and the return value is automatically converted to
`float`. If the return value is an object that is boolean `False`
it's the same as if the return values was ``0.0``.
Nr( )Útextr( r( r* r- } s zLexer.analyse_textFc s¼ t ts jdkr$t\}nÜ jdkrØy ddl}W n, tk
rf } ztd|W Y dd}~X nX d}x4tD ],\}}j|rrt|d
j |d}P qrW |dkrÒ|j
dd
} j | j dpÌd d}|n(j jjd
r td
d
n jd
r td
d
j d d j d
d j
rHj n jrZjd jdkrrj j jrjd rd 7 fdd}
|
} |s¸t| j } | S )a=
Return an iterable of (tokentype, value) pairs generated from
`text`. If `unfiltered` is set to `True`, the filtering mechanism
is bypassed even if filters are defined.
Also preprocess the text, i.e. expand tabs and strip it if
wanted and applies registered filters.
r< Úchardetr NzkTo enable chardet encoding guessing, please install the chardet library from http://chardet.feedparser.org/Úreplacei r; zutf-8u z
Ú
ú
c 3 s( x" j D ]\} }}||fV q W d S )N)Úget_tokens_unprocessed)Ú_ÚtÚv)rA rH r( r* Ústreamer¾ s z"Lexer.get_tokens.<locals>.streamer)rF Ústrr; r rI Ú ImportErrorÚ
_encoding_mapÚ
startswithÚlenÚdecodeÚdetectr? rJ r8 Ústripr7 r: Ú
expandtabsr9 Úendswithr r= ) rA rH Z
unfilteredrN rI ÚeÚdecodedÚbomr; ÚencrQ Ústreamr( )rA rH r* Ú
get_tokens sN
zLexer.get_tokensc C s t dS )zú
Return an iterable of (index, tokentype, value) pairs where "index"
is the starting position of the token within the input text.
In subclasses, implement this method as a generator to
maximize effectiveness.
N)ÚNotImplementedError)rA rH r( r( r* rM Æ s z Lexer.get_tokens_unprocessed)F)r3 r4 r5 r6 r0 ÚurlÚaliasesÚ filenamesZalias_filenamesÚ mimetypesÚpriorityrC rE r@ r- ra rM r( r( r( r* r 1 s
;)Ú metaclassc @ s$ e Zd ZdZefddZdd ZdS )r a
This lexer takes two lexer as arguments. A root lexer and
a language lexer. First everything is scanned using the language
lexer, afterwards all ``Other`` tokens are lexed using the root
lexer.
The lexers from the ``template`` lexer package use this base lexer.
c K s0 |f || _ |f || _|| _tj| f| d S )N)Ú
root_lexerÚlanguage_lexerÚneedler rC )rA Z _root_lexerZ_language_lexerZ_needler> r( r( r* rC Û s zDelegatingLexer.__init__c C s d}g }g }xX| j j|D ]H\}}}|| jkrR|rH|jt||f g }||7 }q|j|||f qW |r||jt||f t|| jj|S )NÚ )rj rM rk rG rV Ú
do_insertionsri )rA rH ZbufferedÚ
insertionsZ
lng_bufferÚirO rP r( r( r* rM á s
z&DelegatingLexer.get_tokens_unprocessedN)r3 r4 r5 r6 r rC rM r( r( r( r* r Ñ s c @ s e Zd ZdZdS )r zI
Indicates that a state should include rules from another state.
N)r3 r4 r5 r6 r( r( r( r* r ø s c @ s e Zd ZdZdd ZdS )Ú_inheritzC
Indicates the a state should inherit from its superclass.
c C s dS )Nr r( )rA r( r( r* rE s z_inherit.__repr__N)r3 r4 r5 r6 rE r( r( r( r* rp ÿ s rp c @ s e Zd ZdZdd Zdd ZdS )Úcombinedz:
Indicates a state combined from multiple states.
c G s t j| |S )N)Útupler/ )ÚclsÚargsr( r( r* r/ s zcombined.__new__c G s d S )Nr( )rA rt r( r( r* rC s zcombined.__init__N)r3 r4 r5 r6 r/ rC r( r( r( r* rq s rq c @ sF e Zd ZdZdd ZdddZdddZdd d
Zd d Zd
d Z dS )Ú _PseudoMatchz:
A pseudo match object constructed from a string.
c C s || _ || _d S )N)Ú_textÚ_start)rA ÚstartrH r( r( r* rC s z_PseudoMatch.__init__Nc C s | j S )N)rw )rA Úargr( r( r* rx s z_PseudoMatch.startc C s | j t| j S )N)rw rV rv )rA ry r( r( r* Úend" s z_PseudoMatch.endc C s |r t d| jS )Nz
No such group)Ú
IndexErrorrv )rA ry r( r( r* Úgroup% s z_PseudoMatch.groupc C s | j fS )N)rv )rA r( r( r* Úgroups* s z_PseudoMatch.groupsc C s i S )Nr( )rA r( r( r* Ú groupdict- s z_PseudoMatch.groupdict)N)N)N)
r3 r4 r5 r6 rC rx rz r| r} r~ r( r( r( r* ru s
ru c s d fdd }|S )zL
Callback that yields multiple actions for each group in the match.
Nc 3 sÄ x°t D ]¤\}}|d kr q
q
t|tkrT|j|d }|r®|j|d ||fV q
|j|d }|d k r
|r~|j|d |_x.|| t|j|d ||D ]}|r|V qW q
W |rÀ|j |_d S )Né )Ú enumerater. r r| rx Úposru rz )ÚlexerÚmatchÚctxro ÚactionÚdataÚitem)rt r( r* Úcallback5 s" zbygroups.<locals>.callback)Nr( )rt r r( )rt r* r 1 s c @ s e Zd ZdZdS )Ú_ThiszX
Special singleton used for indicating the caller class.
Used by ``using``.
N)r3 r4 r5 r6 r( r( r( r* r K s r c sj i dkr:j d}t|ttfr.|d< n d|fd< tkrTdfdd }nd fdd }|S )
aÂ
Callback that processes the match with a different lexer.
The keyword arguments are forwarded to the lexer, except `state` which
is handled separately.
`state` specifies the state that the new lexer will start in, and can
be an enumerable such as ('root', 'inline', 'string') or a simple
string which is assumed to be on top of the root state.
Note: For that to work, `_other` must not be an `ExtendedRegexLexer`.
ÚstateÚstackÚrootNc 3 sn r j | j | jf }n| }|j }x0|j|j f D ]\}}}|| ||fV q>W |rj|j |_d S )N)Úupdater> rD rx rM r| rz r )r r r ÚlxÚsro rO rP )Ú gt_kwargsÚkwargsr( r* r j s zusing.<locals>.callbackc 3 sb j | j f }|j }x0|j|j fD ]\}}}|| ||fV q2W |r^|j |_d S )N)r r> rx rM r| rz r )r r r r r ro rO rP )Ú_otherr r r( r* r y s
)N)N)ÚpoprF Úlistrr r )r r r r r( )r r r r* r T s
c @ s e Zd ZdZdd ZdS )r zÑ
Indicates a state or state action (e.g. #pop) to apply.
For example default('#pop') is equivalent to ('', Token, '#pop')
Note that state tuples may be used as well.
.. versionadded:: 2.0
c C s
|| _ d S )N)r )rA r r( r( r* rC s zdefault.__init__N)r3 r4 r5 r6 rC r( r( r( r* r s c @ s" e Zd ZdZdddZdd ZdS ) r z
Indicates a list of literal words that is transformed into an optimized
regex that matches any of the words.
.. versionadded:: 2.0
rl c C s || _ || _|| _d S )N)r ÚprefixÚsuffix)rA r r r r( r( r* rC s zwords.__init__c C s t | j| j| jdS )N)r r )r r r r )rA r( r( r* r? s z words.getN)rl rl )r3 r4 r5 r6 rC r? r( r( r( r* r s
c @ sJ e Zd ZdZdd Zdd Zdd Zdd Zdd d Zd
d Z dd Z
d
S )ÚRegexLexerMetazw
Metaclass for RegexLexer, creates the self._tokens attribute from
self.tokens on the first instantiation.
c C s t |tr|j }tj||jS )zBPreprocess the regular expression component of a token definition.)rF r r? ÚreÚcompiler )rs ÚregexÚrflagsr r( r( r* Ú_process_regex¨ s
z RegexLexerMeta._process_regexc C s& t |tks"t|s"td|f |S )z5Preprocess the token component of a token definition.z2token type must be simple type or callable, not %r)r. r ÚcallableÚAssertionError)rs Útokenr( r( r* Ú_process_token® s z RegexLexerMeta._process_tokenc C s t |trd|dkrd
S ||kr$|fS |dkr0|S |dd
dkrRt|dd
S dsbtd| n¶t |trÎd | j }| jd7 _g }x4|D ],}||ks¨td
| |j| j||| qW |||< |fS t |trx(|D ] }||ksà|dksàtd | qàW |S dstd | dS )z=Preprocess the state transition action of a token definition.ú#popr ú#pushNé z#pop:Fzunknown new state %rz_tmp_%dzcircular state ref %rzunknown new state zunknown new state def %réÿÿÿÿ)r¡ r¢ ) rF rR Úintr rq Ú_tmpnameÚextendÚ_process_staterr )rs Ú new_stateÚ unprocessedÚ processedZ tmp_stateÚitokensZistater( r( r* Ú_process_new_state´ s6
z!RegexLexerMeta._process_new_statec C s t |tkstd| |d dks0td| ||kr@|| S g }||< | j}x<|| D ].}t|tr||kstd| |j| j||t| q^t|tr¨q^t|t rÜ| j
|j ||}|j t
jdjd|f q^t |tksôtd| y| j|d ||}W n> tk
rH } z td |d || | f | W Y dd} ~ X nX | j|d
}
t|d krld}n| j
|d ||}|j ||
|f q^W |S ) z%Preprocess a single state definition.zwrong state name %rr ú#zinvalid state name %rzcircular state reference %rrl Nzwrong rule def %rz+uncompilable regex %r in state %r of %r: %sr é )r. rR r ÚflagsrF r r§ r¨ rp r r r rG r r r rr r Ú ExceptionÚ
ValueErrorr rV ) rs rª r« r Útokensr Ztdefr© ÚrexÚerrr r( r( r* r¨ × s>
(
z RegexLexerMeta._process_stateNc C s@ i }| j |< |p| j| }x t|D ]}| j||| q&W |S )z-Preprocess a dictionary of token definitions.)Ú _all_tokensr³ r r¨ )rs r0 Ú tokendefsr« r r( r( r* Úprocess_tokendef s
z RegexLexerMeta.process_tokendefc
C sà i }i }xÒ| j D ]È}|jjdi }x´|j D ]¨\}}|j|}|dkr~|||< y|jt}W n tk
rr w,Y nX |||< q,|j|d}|dkrq,||||d
< y|jt} W n tk
rÆ Y q,X || ||< q,W qW |S )a
Merge tokens from superclasses in MRO order, returning a single tokendef
dictionary.
Any state that is not defined by a subclass will be inherited
automatically. States that *are* defined by subclasses will, by
default, override that state in the superclass. If a subclass wishes to
inherit definitions from a superclass, it can use the special value
"inherit", which will cause the superclass' state definition to be
included at that point in the state.
r³ Nr )Ú__mro__Ú__dict__r? ÚitemsÚindexr r² r )
rs r³ Ú inheritableÚcÚtoksr r» ZcuritemsZ inherit_ndxZ new_inh_ndxr( r( r* Ú
get_tokendefs s0
z RegexLexerMeta.get_tokendefsc O sL d| j kr:i | _d| _t| dr(| jr(n| jd| j | _tj | f||S )z:Instantiate cls after preprocessing its token definitions.Ú_tokensr Útoken_variantsrl )
rº r¶ r¦ Úhasattrr r¸ rÀ rÁ r. Ú__call__)rs rt Úkwdsr( r( r* rÄ < s
zRegexLexerMeta.__call__)N) r3 r4 r5 r6 r r r r¨ r¸ rÀ rÄ r( r( r( r* r ¢ s #,
1r c @ s$ e Zd ZdZejZi ZdddZdS )r z±
Base for simple stateful regular expression-based lexers.
Simplifies the lexing process so that you need only
provide a list of states and regular expressions.
r c c sÊ d}| j }t|}||d
}x¤x|D ]*\}}} |||}
|
r*|dk rzt|tkrj|||
j fV n|| |
E dH |
j }| dk rTt| træx°| D ]D} | dkr¾t|dkrà|j q| dkrÖ|j
|d q|j
| qW nbt| t r t | t|kr|dd
= n
|| d
= n,| dkr6|j
|d ndsHt
d| ||d
}P q*W yN|| dkrd g}|d }|tdfV |d7 }w"|t|| fV |d7 }W q" tk
rÀ P Y q"X q"W dS )z~
Split ``text`` into (tokentype, text) pairs.
``stack`` is the initial stack (default: ``['root']``)
r r Nz#popz#pushFzwrong state def: %rrK r r¤ r¤ r¤ r¤ )rÁ r r. r r| rz rF rr rV r rG r¥ Úabsr r r r{ ) rA rH r r r· Z
statestackÚ statetokensÚrexmatchr
r© Úmr r( r( r* rM m sT
z!RegexLexer.get_tokens_unprocessedN©r )rÊ ) r3 r4 r5 r6 r Ú MULTILINEr° r³ rM r( r( r( r* r J s c @ s" e Zd ZdZdddZdd ZdS )r z9
A helper object that holds lexer position data.
Nc C s* || _ || _|pt|| _|p"dg| _d S )Nr )rH r rV rz r )rA rH r r rz r( r( r* rC ° s zLexerContext.__init__c C s d| j | j| jf S )NzLexerContext(%r, %r, %r))rH r r )rA r( r( r* rE ¶ s zLexerContext.__repr__)NN)r3 r4 r5 r6 rC rE r( r( r( r* r « s
c @ s e Zd ZdZdddZdS )r zE
A RegexLexer that uses a context object to store its state.
Nc c sD | j }|s t|d}|d }n|}||jd
}|j}xxþ|D ]f\}}}|||j|j} | rB|dk r¸t|tkr|j|| j fV | j |_n$|| | |E dH |s¸||jd }|dk r¨t |t
r.xÊ|D ]T}
|
dk rþt |jdkr(|jj qÔ|
dkr |jj
|jd qÔ|jj
|
qÔW nlt |trjt|t |jkr\|jdd
= n |j|d
= n0|dkr|jj
|jd
ndstd| ||jd }P qBW yt|j|jkrÀP ||j d krþdg|_|d }|jtd fV | jd7 _w:|jt||j fV | jd7 _W q: tk
r: P Y q:X q:W dS )z
Split ``text`` into (tokentype, text) pairs.
If ``context`` is given, use this lexer context instead.
r r r Nz#popz#pushFzwrong state def: %rrK r¤ r¤ r¤ r¤ r¤ )rÁ r r rH r rz r. r r| rF rr rV r rG r¥ rÆ r r r r{ ) rA rH Úcontextr· r rÇ rÈ r
r© rÉ r r( r( r* rM À sb
z)ExtendedRegexLexer.get_tokens_unprocessed)NN)r3 r4 r5 r6 rM r( r( r( r* r » s c c s¶ t | } yt| \}}W n tk
r6 |E dH dS X d}d}xü|D ]ô\}}}|dkr\|}d} x¨|ot|t| |kr|| ||
}
|
r¤|||
fV |t|
7 }x*|D ]"\} } }
|| |
fV |t|
7 }qªW || } yt| \}}W qb tk
r d}P Y qbX qbW | t|k rF|||| d
fV |t|| 7 }qFW xr|r°|pNd}x,|D ]$\}}}|||fV |t|7 }qVW yt| \}}W n tk
rª d}P Y nX q@W dS )ag
Helper for lexers which must combine the results of several
sublexers.
``insertions`` is a list of ``(index, itokens)`` pairs.
Each ``itokens`` iterable should be inserted at position
``index`` into the token stream given by the ``tokens``
argument.
The result is a combined token stream.
TODO: clean up the code here.
NTr F)ÚiterÚnextÚ
StopIterationrV )rn r³ r¼ r¬ ÚrealposZinsleftro rO rP ZoldiZtmpvalZit_indexZit_tokenZit_valueÚpr( r( r* rm sN
rm c @ s e Zd ZdZdd ZdS )ÚProfilingRegexLexerMetaz>Metaclass for ProfilingRegexLexer, collects regex timing info.c sL t |tr t|j|j|jdn|tj|tjf fdd }|S )N)r r c s` j d jfddg}tj }j| ||}tj }|d d7 < |d || 7 < |S )Nr r g r¤ )Ú
_prof_dataÚ
setdefaultÚtimer )rH r ÚendposÚinfoÚt0ÚresÚt1)rs Úcompiledr´ r r( r* Ú
match_funcN s z:ProfilingRegexLexerMeta._process_regex.<locals>.match_func) rF r r r r r r ÚsysÚmaxsize)rs r r r rÜ r( )rs rÛ r´ r r* r F s
z&ProfilingRegexLexerMeta._process_regexN)r3 r4 r5 r6 r r( r( r( r* rÒ C s rÒ c @ s" e Zd ZdZg ZdZdddZdS ) ÚProfilingRegexLexerzFDrop-in replacement for RegexLexer that does profiling of its regexes.é r c # sÒ j jji tj ||E d H j jj }tdd |j D fdddd}tdd |D }t t d j j
t ||f t d d
t d d t dd
x|D ]}t d| q®W t d d
d S )Nc s sN | ]F\\}}\}}|t |jd jdddd
|d| d| | fV qdS )zu'z\\ú\NéA iè )ÚreprrY rJ )Ú.0r ÚrÚnrO r( r( r* ú <genexpr>d s z=ProfilingRegexLexer.get_tokens_unprocessed.<locals>.<genexpr>c s
| j S )N)Ú_prof_sort_index)r) )rA r( r* r+ g s z<ProfilingRegexLexer.get_tokens_unprocessed.<locals>.<lambda>T)ÚkeyÚreversec s s | ]}|d V qdS )é Nr( )rä r) r( r( r* rç i s z2Profiling result for %s lexing %d chars in %.3f msú=én z$%-20s %-64s ncalls tottime percallr r ú-z%-20s %-65s %5d %8.4f %8.4f)r r ) rD rÓ rG r rM r Úsortedr» ÚsumÚprintr3 rV )rA rH r Úrawdatar Z sum_totalr2 r( )rA r* rM _ s"
z*ProfilingRegexLexer.get_tokens_unprocessedN©r )ró )r3 r4 r5 r6 rÓ rè rM r( r( r( r* rß Y s rß )r r )r r! )r" r# )r$ r% )r& r' )6r6 r rÝ rÕ Zpygments.filterr r Zpygments.filtersr Zpygments.tokenr r r r r Z
pygments.utilr
r r r
r r Zpygments.regexoptr Ú__all__r r rT Ú staticmethodZ_default_analyser. r, r r rR r rp r rr rq ru r r r r r r r r r r rm rÒ rß r( r( r( r* Ú<module> sR
!'
2 )aH@