Edit File: _tokenizer.pyo
� ��abc @` s� d d l m Z m Z m Z d d l m Z d d l m Z d d l m Z d d l m Z d d l m Z m Z d d l m Z m Z m Z d d l m Z m Z d d l m Z d d l m Z d d l m Z e e � Z d e f d � � YZ d S( i ( t absolute_importt divisiont unicode_literals( t unichr( t dequei ( t spaceCharacters( t entities( t asciiLetterst asciiUpper2Lower( t digitst hexDigitst EOF( t tokenTypest tagTokenTypes( t replacementCharacters( t HTMLInputStream( t Triet HTMLTokenizerc B` s� e Z d Z dJ d � Z d � Z d � Z dJ e d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z! d � Z" d � Z# d � Z$ d! � Z% d" � Z& d# � Z' d$ � Z( d% � Z) d& � Z* d' � Z+ d( � Z, d) � Z- d* � Z. d+ � Z/ d, � Z0 d- � Z1 d. � Z2 d/ � Z3 d0 � Z4 d1 � Z5 d2 � Z6 d3 � Z7 d4 � Z8 d5 � Z9 d6 � Z: d7 � Z; d8 � Z< d9 � Z= d: � Z> d; � Z? d< � Z@ d= � ZA d> � ZB d? � ZC d@ � ZD dA � ZE dB � ZF dC � ZG dD � ZH dE � ZI dF � ZJ dG � ZK dH � ZL dI � ZM RS(K u This class takes care of tokenizing HTML. * self.currentToken Holds the token that is currently being processed. * self.state Holds a reference to the method to be invoked... XXX * self.stream Points to HTMLInputStream object. c K` sb t | | � | _ | | _ t | _ g | _ | j | _ t | _ d | _ t t | � j � d S( N( R t streamt parsert Falset escapeFlagt lastFourCharst dataStatet statet escapet Nonet currentTokent superR t __init__( t selfR R t kwargs( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR " s c c` s} t g � | _ xg | j � rx x6 | j j rV i t d d 6| j j j d � d 6Vq! Wx | j rt | j j � VqZ Wq Wd S( u� This is where the magic happens. We do our usually processing through the states and when we have a token to return we yield the token which pauses processing until the next token is requested. u ParseErroru typei u dataN( R t tokenQueueR R t errorsR t popt popleft( R ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyt __iter__1 s * c % C` s� t } d } | r! t } d } n g } | j j � } x8 | | k rp | t k rp | j | � | j j � } q9 Wt d j | � | � } | t k r� t | } | j j i t d d 6d d 6i | d 6d 6� n�d | k o� d k n s� | d k r3d } | j j i t d d 6d d 6i | d 6d 6� nrd | k oJd k n s d | k ofd k n s d | k o�d k n s d | k o�d k n s | t d d d d d d d d d d d d! d" d# d$ d% d& d' d( d) d* d+ d, d- d. d/ d0 d1 d2 d3 d4 d5 d6 d7 d g# � k rQ| j j i t d d 6d d 6i | d 6d 6� n y t | � } WnA t k r�| d8 } t d | d ?B� t d9 | d: @B� } n X| d; k r�| j j i t d d 6d< d 6� | j j | � n | S(= u� This function returns either U+FFFD or the character based on the decimal or hexadecimal representation. It also discards ";" if present. If not present self.tokenQueue.append({"type": tokenTypes["ParseError"]}) is invoked. i i u u ParseErroru typeu$ illegal-codepoint-for-numeric-entityu datau charAsIntu datavarsi � i�� i�� u �i i i i i i� i� i� i i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i i � i� u ;u numeric-entity-without-semicolon( R R R t charR t appendt intt joinR R R t frozensett chrt ValueErrort unget( R t isHext allowedt radixt charStackt ct charAsIntR% t v( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyt consumeNumberEntityA s` * c C` s� d } | j j � g } | d t k s] | d t d d f k s] | d k rt | | d k rt | j j | d � n�| d d k rpt } | j | j j � � | d d k r� t } | j | j j � � n | r� | d t k s� | r"| d t k r"| j j | d � | j | � } q7| j j i t d d 6d d 6� | j j | j � � d d j | � } n�xF | d t k r�t j d j | � � s�Pn | j | j j � � qsWy, t j d j | d � � } t | � } Wn t k r�d } n X| d k r�| d d k r@| j j i t d d 6d d 6� n | d d k r�| r�| | t k s�| | t k s�| | d k r�| j j | j � � d d j | � } q7t | } | j j | j � � | d j | | � 7} nK | j j i t d d 6d d 6� | j j | j � � d d j | � } | r[| j d d d c | 7<n= | t k rpd } n d } | j j i t | d 6| d 6� d S( Nu &i u <u #i����u xu Xu ParseErroru typeu expected-numeric-entityu datau u ;u named-entity-without-semicolonu =u expected-named-entityi u SpaceCharactersu Characters( u xu X( R R% R R R R, R R& t TrueR R R4 R R R" R( t entitiesTriet has_keys_with_prefixt longest_prefixt lent KeyErrorR R R ( R t allowedChart fromAttributet outputR0 t hext entityNamet entityLengtht tokenType( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyt consumeEntity� sf ) c C` s | j d | d t � d S( uI This method replaces the need for "entityInAttributeValueState". R; R< N( RB R5 ( R R; ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyt processEntityInAttribute� s c C` s� | j } | d t k r� | d j t � | d <| d t d k r� | d rs | j j i t d d 6d d 6� n | d r� | j j i t d d 6d d 6� q� q� n | j j | � | j | _ d S( u� This method is a generic handler for emitting the tags. It also sets the state to "data" because that's what's needed after a token has been emitted. u typeu nameu EndTagu datau ParseErroru attributes-in-end-tagu selfClosingu self-closing-flag-on-end-tagN( R R t translateR R R R& R R ( R t token( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyt emitCurrentToken� s c C` s( | j j � } | d k r* | j | _ n� | d k rE | j | _ n� | d k r� | j j i t d d 6d d 6� | j j i t d d 6d d 6� n� | t k r� t S| t k r� | j j i t d d 6| | j j t t � d 6� n8 | j j d � } | j j i t d d 6| | d 6� t S( Nu &u <u u ParseErroru typeu invalid-codepointu datau Charactersu SpaceCharacters( u &u <u ( R R% t entityDataStateR t tagOpenStateR R& R R R R t charsUntilR5 ( R t datat chars( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR � s& !c C` s | j � | j | _ t S( N( RB R R R5 ( R ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRG s c C` s( | j j � } | d k r* | j | _ n� | d k rE | j | _ n� | t k rU t S| d k r� | j j i t d d 6d d 6� | j j i t d d 6d d 6� n| | t k r� | j j i t d d 6| | j j t t � d 6� n8 | j j d � } | j j i t d d 6| | d 6� t S( Nu &u <u u ParseErroru typeu invalid-codepointu datau Charactersu �u SpaceCharacters( u &u <u ( R R% t characterReferenceInRcdataR t rcdataLessThanSignStateR R R R& R R RI R5 ( R RJ RK ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyt rcdataState s& !c C` s | j � | j | _ t S( N( RB RN R R5 ( R ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRL 1 s c C` s� | j j � } | d k r* | j | _ n� | d k r} | j j i t d d 6d d 6� | j j i t d d 6d d 6� nH | t k r� t S| j j d � } | j j i t d d 6| | d 6� t S( Nu <u u ParseErroru typeu invalid-codepointu datau Charactersu �( u <u ( R R% t rawtextLessThanSignStateR R R& R R R RI R5 ( R RJ RK ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyt rawtextState6 s c C` s� | j j � } | d k r* | j | _ n� | d k r} | j j i t d d 6d d 6� | j j i t d d 6d d 6� nH | t k r� t S| j j d � } | j j i t d d 6| | d 6� t S( Nu <u u ParseErroru typeu invalid-codepointu datau Charactersu �( u <u ( R R% t scriptDataLessThanSignStateR R R& R R R RI R5 ( R RJ RK ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyt scriptDataStateH s c C` s� | j j � } | t k r t S| d k rr | j j i t d d 6d d 6� | j j i t d d 6d d 6� n2 | j j i t d d 6| | j j d � d 6� t S( Nu u ParseErroru typeu invalid-codepointu datau Charactersu �( R R% R R R R& R RI R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyt plaintextStateZ s c C` s� | j j � } | d k r* | j | _ nr| d k rE | j | _ nW| t k r� i t d d 6| d 6g d 6t d 6t d 6| _ | j | _ n | d k r� | j j i t d d 6d d 6� | j j i t d d 6d d 6� | j | _ n� | d k r<| j j i t d d 6d d 6� | j j | � | j | _ n` | j j i t d d 6d d 6� | j j i t d d 6d d 6� | j j | � | j | _ t S( Nu !u /u StartTagu typeu nameu datau selfClosingu selfClosingAcknowledgedu >u ParseErroru' expected-tag-name-but-got-right-bracketu Charactersu <>u ?u' expected-tag-name-but-got-question-marku expected-tag-nameu <( R R% t markupDeclarationOpenStateR t closeTagOpenStateR R R R t tagNameStateR R& R R, t bogusCommentStateR5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRH i s6 " " c C` s? | j j � } | t k rS i t d d 6| d 6g d 6t d 6| _ | j | _ n� | d k r� | j j i t d d 6d d 6� | j | _ n� | t k r� | j j i t d d 6d d 6� | j j i t d d 6d d 6� | j | _ nL | j j i t d d 6d d 6i | d 6d 6� | j j | � | j | _ t S( Nu EndTagu typeu nameu datau selfClosingu >u ParseErroru* expected-closing-tag-but-got-right-bracketu expected-closing-tag-but-got-eofu Charactersu </u! expected-closing-tag-but-got-charu datavars( R R% R R R R RV R R R& R R R, RW R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRU � s( " c C` s� | j j � } | t k r* | j | _ n� | d k rC | j � n� | t k r� | j j i t d d 6d d 6� | j | _ nr | d k r� | j | _ nW | d k r� | j j i t d d 6d d 6� | j d c d 7<n | j d c | 7<t S( Nu >u ParseErroru typeu eof-in-tag-nameu datau /u u invalid-codepointu nameu �( R R% R t beforeAttributeNameStateR RF R R R& R R t selfClosingStartTagStateR R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRV � s" c C` su | j j � } | d k r3 d | _ | j | _ n> | j j i t d d 6d d 6� | j j | � | j | _ t S( Nu /u u Charactersu typeu <u data( R R% t temporaryBuffert rcdataEndTagOpenStateR R R& R R, RN R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRM � s " c C` s{ | j j � } | t k r9 | j | 7_ | j | _ n> | j j i t d d 6d d 6� | j j | � | j | _ t S( Nu Charactersu typeu </u data( R R% R RZ t rcdataEndTagNameStateR R R& R R, RN R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR[ � s " c C` s� | j o( | j d j � | j j � k } | j j � } | t k r� | r� i t d d 6| j d 6g d 6t d 6| _ | j | _ n| d k r� | r� i t d d 6| j d 6g d 6t d 6| _ | j | _ n� | d k r+| r+i t d d 6| j d 6g d 6t d 6| _ | j � | j | _ nc | t k rI| j | 7_ nE | j j i t d d 6d | j d 6� | j j | � | j | _ t S( Nu nameu EndTagu typeu datau selfClosingu /u >u Charactersu </( R t lowerRZ R R% R R R RX R RY RF R R R R& R, RN R5 ( R t appropriateRJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR\ � s2 + c C` su | j j � } | d k r3 d | _ | j | _ n> | j j i t d d 6d d 6� | j j | � | j | _ t S( Nu /u u Charactersu typeu <u data( R R% RZ t rawtextEndTagOpenStateR R R& R R, RP R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRO � s " c C` s{ | j j � } | t k r9 | j | 7_ | j | _ n> | j j i t d d 6d d 6� | j j | � | j | _ t S( Nu Charactersu typeu </u data( R R% R RZ t rawtextEndTagNameStateR R R& R R, RP R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR_ � s " c C` s� | j o( | j d j � | j j � k } | j j � } | t k r� | r� i t d d 6| j d 6g d 6t d 6| _ | j | _ n| d k r� | r� i t d d 6| j d 6g d 6t d 6| _ | j | _ n� | d k r+| r+i t d d 6| j d 6g d 6t d 6| _ | j � | j | _ nc | t k rI| j | 7_ nE | j j i t d d 6d | j d 6� | j j | � | j | _ t S( Nu nameu EndTagu typeu datau selfClosingu /u >u Charactersu </( R R] RZ R R% R R R RX R RY RF R R R R& R, RP R5 ( R R^ RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR` s2 + c C` s� | j j � } | d k r3 d | _ | j | _ n{ | d k rp | j j i t d d 6d d 6� | j | _ n> | j j i t d d 6d d 6� | j j | � | j | _ t S( Nu /u u !u Charactersu typeu <!u datau <( R R% RZ t scriptDataEndTagOpenStateR R R& R t scriptDataEscapeStartStateR, RR R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRQ s "" c C` s{ | j j � } | t k r9 | j | 7_ | j | _ n> | j j i t d d 6d d 6� | j j | � | j | _ t S( Nu Charactersu typeu </u data( R R% R RZ t scriptDataEndTagNameStateR R R& R R, RR R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRa , s " c C` s� | j o( | j d j � | j j � k } | j j � } | t k r� | r� i t d d 6| j d 6g d 6t d 6| _ | j | _ n| d k r� | r� i t d d 6| j d 6g d 6t d 6| _ | j | _ n� | d k r+| r+i t d d 6| j d 6g d 6t d 6| _ | j � | j | _ nc | t k rI| j | 7_ nE | j j i t d d 6d | j d 6� | j j | � | j | _ t S( Nu nameu EndTagu typeu datau selfClosingu /u >u Charactersu </( R R] RZ R R% R R R RX R RY RF R R R R& R, RR R5 ( R R^ RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRc 7 s2 + c C` sl | j j � } | d k rL | j j i t d d 6d d 6� | j | _ n | j j | � | j | _ t S( Nu -u Charactersu typeu data( R R% R R& R t scriptDataEscapeStartDashStateR R, RR R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRb S s " c C` sl | j j � } | d k rL | j j i t d d 6d d 6� | j | _ n | j j | � | j | _ t S( Nu -u Charactersu typeu data( R R% R R& R t scriptDataEscapedDashDashStateR R, RR R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRd ] s " c C` s | j j � } | d k rL | j j i t d d 6d d 6� | j | _ n� | d k rg | j | _ n� | d k r� | j j i t d d 6d d 6� | j j i t d d 6d d 6� nS | t k r� | j | _ n8 | j j d � } | j j i t d d 6| | d 6� t S( Nu -u Charactersu typeu datau <u u ParseErroru invalid-codepointu �( u <u -u ( R R% R R& R t scriptDataEscapedDashStateR t" scriptDataEscapedLessThanSignStateR R RI R5 ( R RJ RK ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyt scriptDataEscapedStateg s" " c C` s | j j � } | d k rL | j j i t d d 6d d 6� | j | _ n� | d k rg | j | _ n� | d k r� | j j i t d d 6d d 6� | j j i t d d 6d d 6� | j | _ nI | t k r� | j | _ n. | j j i t d d 6| d 6� | j | _ t S( Nu -u Charactersu typeu datau <u u ParseErroru invalid-codepointu �( R R% R R& R Re R Rg Rh R R R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRf { s" " " c C` sD | j j � } | d k r@ | j j i t d d 6d d 6� n | d k r[ | j | _ n� | d k r� | j j i t d d 6d d 6� | j | _ n� | d k r� | j j i t d d 6d d 6� | j j i t d d 6d d 6� | j | _ nI | t k r| j | _ n. | j j i t d d 6| d 6� | j | _ t S( Nu -u Charactersu typeu datau <u >u u ParseErroru invalid-codepointu �( R R% R R& R Rg R RR Rh R R R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRe � s& % " " c C` s� | j j � } | d k r3 d | _ | j | _ n� | t k r} | j j i t d d 6d | d 6� | | _ | j | _ n> | j j i t d d 6d d 6� | j j | � | j | _ t S( Nu /u u Charactersu typeu <u data( R R% RZ t scriptDataEscapedEndTagOpenStateR R R R& R t scriptDataDoubleEscapeStartStateR, Rh R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRg � s & " c C` su | j j � } | t k r3 | | _ | j | _ n> | j j i t d d 6d d 6� | j j | � | j | _ t S( Nu Charactersu typeu </u data( R R% R RZ t scriptDataEscapedEndTagNameStateR R R& R R, Rh R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRi � s " c C` s� | j o( | j d j � | j j � k } | j j � } | t k r� | r� i t d d 6| j d 6g d 6t d 6| _ | j | _ n| d k r� | r� i t d d 6| j d 6g d 6t d 6| _ | j | _ n� | d k r+| r+i t d d 6| j d 6g d 6t d 6| _ | j � | j | _ nc | t k rI| j | 7_ nE | j j i t d d 6d | j d 6� | j j | � | j | _ t S( Nu nameu EndTagu typeu datau selfClosingu /u >u Charactersu </( R R] RZ R R% R R R RX R RY RF R R R R& R, Rh R5 ( R R^ RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRk � s2 + c C` s� | j j � } | t t d � Bk rz | j j i t d d 6| d 6� | j j � d k rk | j | _ q� | j | _ n\ | t k r� | j j i t d d 6| d 6� | j | 7_ n | j j | � | j | _ t S( Nu /u >u Charactersu typeu datau script( u /u >( R R% R R) R R& R RZ R] t scriptDataDoubleEscapedStateR Rh R R, R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRj � s " " c C` s? | j j � } | d k rL | j j i t d d 6d d 6� | j | _ n� | d k r� | j j i t d d 6d d 6� | j | _ n� | d k r� | j j i t d d 6d d 6� | j j i t d d 6d d 6� n_ | t k r| j j i t d d 6d d 6� | j | _ n" | j j i t d d 6| d 6� t S( Nu -u Charactersu typeu datau <u u ParseErroru invalid-codepointu �u eof-in-script-in-script( R R% R R& R t scriptDataDoubleEscapedDashStateR t( scriptDataDoubleEscapedLessThanSignStateR R R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRl � s$ " " "c C` sW | j j � } | d k rL | j j i t d d 6d d 6� | j | _ n| d k r� | j j i t d d 6d d 6� | j | _ n� | d k r� | j j i t d d 6d d 6� | j j i t d d 6d d 6� | j | _ nk | t k r%| j j i t d d 6d d 6� | j | _ n. | j j i t d d 6| d 6� | j | _ t S( Nu -u Charactersu typeu datau <u u ParseErroru invalid-codepointu �u eof-in-script-in-script( R R% R R& R t$ scriptDataDoubleEscapedDashDashStateR Rn Rl R R R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRm s( " " " c C` s� | j j � } | d k r@ | j j i t d d 6d d 6� nD| d k r} | j j i t d d 6d d 6� | j | _ n| d k r� | j j i t d d 6d d 6� | j | _ n� | d k r| j j i t d d 6d d 6� | j j i t d d 6d d 6� | j | _ nk | t k rV| j j i t d d 6d d 6� | j | _ n. | j j i t d d 6| d 6� | j | _ t S( Nu -u Charactersu typeu datau <u >u u ParseErroru invalid-codepointu �u eof-in-script-in-script( R R% R R& R Rn R RR Rl R R R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRo s, % " " " c C` su | j j � } | d k rU | j j i t d d 6d d 6� d | _ | j | _ n | j j | � | j | _ t S( Nu /u Charactersu typeu datau ( R R% R R& R RZ t scriptDataDoubleEscapeEndStateR R, Rl R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRn 0 s " c C` s� | j j � } | t t d � Bk rz | j j i t d d 6| d 6� | j j � d k rk | j | _ q� | j | _ n\ | t k r� | j j i t d d 6| d 6� | j | 7_ n | j j | � | j | _ t S( Nu /u >u Charactersu typeu datau script( u /u >( R R% R R) R R& R RZ R] Rh R Rl R R, R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRp ; s " " c C` s� | j j � } | t k r1 | j j t t � nz| t k rf | j d j | d g � | j | _ nE| d k r | j � n,| d k r� | j | _ n| d k r� | j j i t d d 6d d 6� | j d j | d g � | j | _ n� | d k rH| j j i t d d 6d d 6� | j d j d d g � | j | _ nc | t k r�| j j i t d d 6d d 6� | j | _ n& | j d j | d g � | j | _ t S( Nu datau u >u /u 'u "u =u <u ParseErroru typeu# invalid-character-in-attribute-nameu u invalid-codepointu �u# expected-attribute-name-but-got-eof( u 'u "u =u <( R R% R RI R5 R R R& t attributeNameStateR RF RY R R R R ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRX K s6 c C` sv | j j � } t } t } | d k r6 | j | _ n�| t k ry | j d d d c | | j j t t � 7<t } nM| d k r� t } n8| t k r� | j | _ n | d k r� | j | _ n| d k r| j j i t d d 6d d 6� | j d d d c d 7<t } n� | d k rh| j j i t d d 6d d 6� | j d d d c | 7<t } n^ | t k r�| j j i t d d 6d d 6� | j | _ n! | j d d d c | 7<t } | rr| j d d d j t � | j d d d <x` | j d d D]M \ } } | j d d d | k r | j j i t d d 6d d 6� Pq q W| rr| j � qrn t S( Nu =u datai����i u >u /u u ParseErroru typeu invalid-codepointu �u 'u "u <u# invalid-character-in-attribute-nameu eof-in-attribute-nameu duplicate-attribute( u 'u "u <( R R% R5 R t beforeAttributeValueStateR R R RI R t afterAttributeNameStateRY R R& R R R RD R RF ( R RJ t leavingThisStatet emitTokent namet _( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRq i sR - c C` s� | j j � } | t k r1 | j j t t � n�| d k rL | j | _ nz| d k re | j � na| t k r� | j d j | d g � | j | _ n,| d k r� | j | _ n| d k r | j j i t d d 6d d 6� | j d j d d g � | j | _ n� | d k rc| j j i t d d 6d d 6� | j d j | d g � | j | _ nc | t k r�| j j i t d d 6d d 6� | j | _ n& | j d j | d g � | j | _ t S( Nu =u >u datau u /u u ParseErroru typeu invalid-codepointu �u 'u "u <u& invalid-character-after-attribute-nameu expected-end-of-tag-but-got-eof( u 'u "u <( R R% R RI R5 Rr R RF R R R& Rq RY R R R R ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRs � s: c C` s� | j j � } | t k r1 | j j t t � n�| d k rL | j | _ n�| d k rw | j | _ | j j | � nj| d k r� | j | _ nO| d k r� | j j i t d d 6d d 6� | j � n| d k r%| j j i t d d 6d d 6� | j d d d c d 7<| j | _ n� | d k r}| j j i t d d 6d d 6� | j d d d c | 7<| j | _ nd | t k r�| j j i t d d 6d d 6� | j | _ n' | j d d d c | 7<| j | _ t S( Nu "u &u 'u >u ParseErroru typeu. expected-attribute-value-but-got-right-bracketu datau u invalid-codepointi����i u �u =u <u `u" equals-in-unquoted-attribute-valueu$ expected-attribute-value-but-got-eof( u =u <u `( R R% R RI R5 t attributeValueDoubleQuotedStateR t attributeValueUnQuotedStateR, t attributeValueSingleQuotedStateR R& R RF R R R ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRr � s> c C` s� | j j � } | d k r* | j | _ n� | d k rF | j d � n� | d k r� | j j i t d d 6d d 6� | j d d d c d 7<nh | t k r� | j j i t d d 6d d 6� | j | _ n+ | j d d d c | | j j d � 7<t S( Nu "u &u u ParseErroru typeu invalid-codepointu datai����i u �u# eof-in-attribute-value-double-quote( u "u &u ( R R% t afterAttributeValueStateR RC R R& R R R R RI R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRx � s c C` s� | j j � } | d k r* | j | _ n� | d k rF | j d � n� | d k r� | j j i t d d 6d d 6� | j d d d c d 7<nh | t k r� | j j i t d d 6d d 6� | j | _ n+ | j d d d c | | j j d � 7<t S( Nu 'u &u u ParseErroru typeu invalid-codepointu datai����i u �u# eof-in-attribute-value-single-quote( u 'u &u ( R R% R{ R RC R R& R R R R RI R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRz � s c C` sm | j j � } | t k r* | j | _ n?| d k rF | j d � n#| d k r_ | j � n | d k r� | j j i t d d 6d d 6� | j d d d c | 7<n� | d k r� | j j i t d d 6d d 6� | j d d d c d 7<nr | t k r4| j j i t d d 6d d 6� | j | _ n5 | j d d d c | | j j t d � t B� 7<t S( Nu &u >u "u 'u =u <u `u ParseErroru typeu0 unexpected-character-in-unquoted-attribute-valueu datai����i u u invalid-codepointu �u eof-in-attribute-value-no-quotes( u "u 'u =u <u `( u &u >u "u 'u =u <u `u ( R R% R RX R RC RF R R& R R R R RI R) R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRy s, !c C` s� | j j � } | t k r* | j | _ n� | d k rC | j � n� | d k r^ | j | _ n� | t k r� | j j i t d d 6d d 6� | j j | � | j | _ n> | j j i t d d 6d d 6� | j j | � | j | _ t S( Nu >u /u ParseErroru typeu$ unexpected-EOF-after-attribute-valueu datau* unexpected-character-after-attribute-value( R R% R RX R RF RY R R R& R R, R R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR{ s" c C` s� | j j � } | d k r5 t | j d <| j � n� | t k r� | j j i t d d 6d d 6� | j j | � | j | _ n> | j j i t d d 6d d 6� | j j | � | j | _ t S( Nu >u selfClosingu ParseErroru typeu# unexpected-EOF-after-solidus-in-tagu datau) unexpected-character-after-solidus-in-tag( R R% R5 R RF R R R& R R, R R RX ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRY 4 s c C` sc | j j d � } | j d d � } | j j i t d d 6| d 6� | j j � | j | _ t S( Nu >u u �u Commentu typeu data( R RI t replaceR R& R R% R R R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRW F s c C` sB | j j � g } | d d k rv | j | j j � � | d d k r�i t d d 6d d 6| _ | j | _ t Snw| d d k r(t } xP d d d! d" d# d$ f D]6 } | j | j j � � | d | k r� t } Pq� q� W| r�i t d d 6d d 6d d 6d d 6t d 6| _ | j | _ t Sn� | d d k r�| j d k r�| j j j r�| j j j d j | j j j k r�t } xP d d d d d d g D]6 } | j | j j � � | d | k r�t } Pq�q�W| r�| j | _ t Sn | j j i t d d 6d d 6� x | r1| j j | j � � qW| j | _ t S(% Ni����u -u Commentu typeu u datau du Du ou Ou cu Cu tu Tu yu Yu pu Pu eu Eu Doctypeu nameu publicIdu systemIdu correctu [u Au ParseErroru expected-dashes-or-doctype( u du D( u ou O( u cu C( u tu T( u yu Y( u pu P( u eu E( R R% R& R R t commentStartStateR R5 R R t doctypeStateR t treet openElementst namespacet defaultNamespacet cdataSectionStateR R, R" RW ( R R0 t matchedt expected( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyRT U sR % c C` s1 | j j � } | d k r* | j | _ n| d k rn | j j i t d d 6d d 6� | j d c d 7<n� | d k r� | j j i t d d 6d d 6� | j j | j � | j | _ no | t k r| j j i t d d 6d d 6� | j j | j � | j | _ n | j d c | 7<| j | _ t S( Nu -u u ParseErroru typeu invalid-codepointu datau �u >u incorrect-commentu eof-in-comment( R R% t commentStartDashStateR R R& R R R R t commentStateR5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR} � s( c C` s5 | j j � } | d k r* | j | _ n| d k rn | j j i t d d 6d d 6� | j d c d 7<n� | d k r� | j j i t d d 6d d 6� | j j | j � | j | _ ns | t k r| j j i t d d 6d d 6� | j j | j � | j | _ n# | j d c d | 7<| j | _ t S( Nu -u u ParseErroru typeu invalid-codepointu datau -�u >u incorrect-commentu eof-in-comment( R R% t commentEndStateR R R& R R R R R� R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR� � s( c C` s� | j j � } | d k r* | j | _ n� | d k rn | j j i t d d 6d d 6� | j d c d 7<ns | t k r� | j j i t d d 6d d 6� | j j | j � | j | _ n# | j d c | | j j d � 7<t S( Nu -u u ParseErroru typeu invalid-codepointu datau �u eof-in-comment( u -u ( R R% t commentEndDashStateR R R& R R R R RI R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR� � s c C` s� | j j � } | d k r* | j | _ n� | d k rz | j j i t d d 6d d 6� | j d c d 7<| j | _ ns | t k r� | j j i t d d 6d d 6� | j j | j � | j | _ n# | j d c d | 7<| j | _ t S( Nu -u u ParseErroru typeu invalid-codepointu datau -�u eof-in-comment-end-dash( R R% R� R R R& R R R� R R R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR� � s c C` s� | j j � } | d k r= | j j | j � | j | _ nf| d k r� | j j i t d d 6d d 6� | j d c d 7<| j | _ n| d k r� | j j i t d d 6d d 6� | j | _ n� | d k r| j j i t d d 6d d 6� | j d c | 7<n� | t k r^| j j i t d d 6d d 6� | j j | j � | j | _ nE | j j i t d d 6d d 6� | j d c d | 7<| j | _ t S( Nu >u u ParseErroru typeu invalid-codepointu datau --�u !u, unexpected-bang-after-double-dash-in-commentu -u, unexpected-dash-after-double-dash-in-commentu eof-in-comment-double-dashu unexpected-char-in-commentu --( R R% R R& R R R R R� t commentEndBangStateR R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR� � s6 c C` s2 | j j � } | d k r= | j j | j � | j | _ n� | d k rk | j d c d 7<| j | _ n� | d k r� | j j i t d d 6d d 6� | j d c d 7<| j | _ ns | t k r | j j i t d d 6d d 6� | j j | j � | j | _ n# | j d c d | 7<| j | _ t S( Nu >u -u datau --!u u ParseErroru typeu invalid-codepointu --!�u eof-in-comment-end-bang-state( R R% R R& R R R R� R R� R R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR� � s( c C` s� | j j � } | t k r* | j | _ n� | t k r� | j j i t d d 6d d 6� t | j d <| j j | j � | j | _ n> | j j i t d d 6d d 6� | j j | � | j | _ t S( Nu ParseErroru typeu! expected-doctype-name-but-got-eofu datau correctu need-space-after-doctype( R R% R t beforeDoctypeNameStateR R R R& R R R R R, R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR~ s c C` s? | j j � } | t k r n | d k r{ | j j i t d d 6d d 6� t | j d <| j j | j � | j | _ n� | d k r� | j j i t d d 6d d 6� d | j d <| j | _ nv | t k r"| j j i t d d 6d d 6� t | j d <| j j | j � | j | _ n | | j d <| j | _ t S( Nu >u ParseErroru typeu+ expected-doctype-name-but-got-right-bracketu datau correctu u invalid-codepointu �u nameu! expected-doctype-name-but-got-eof( R R% R R R& R R R R R t doctypeNameStateR R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR� s. c C` ss | j j � } | t k rG | j d j t � | j d <| j | _ n(| d k r� | j d j t � | j d <| j j | j � | j | _ n� | d k r� | j j i t d d 6d d 6� | j d c d 7<| j | _ n� | t k r\| j j i t d d 6d d 6� t | j d <| j d j t � | j d <| j j | j � | j | _ n | j d c | 7<t S( Nu nameu >u u ParseErroru typeu invalid-codepointu datau �u eof-in-doctype-nameu correct( R R% R R RD R t afterDoctypeNameStateR R R& R R R� R R R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR� 6 s, c C` s� | j j � } | t k r n�| d k rL | j j | j � | j | _ n�| t k r� t | j d <| j j | � | j j i t d d 6d d 6� | j j | j � | j | _ n9| d k r)t } xB d d! d" d# d$ f D]+ } | j j � } | | k r� t } Pq� q� W| r�| j | _ t Snp | d% k r�t } xB d& d' d( d) d* f D]+ } | j j � } | | k rQt } PqQqQW| r�| j | _ t Sn | j j | � | j j i t d d 6d d 6i | d 6d 6� t | j d <| j | _ t S(+ Nu >u correctu ParseErroru typeu eof-in-doctypeu datau pu Pu uu Uu bu Bu lu Lu iu Iu cu Cu su Su yu Yu tu Tu eu Eu mu Mu* expected-space-or-right-bracket-in-doctypeu datavars( u pu P( u uu U( u bu B( u lu L( u iu I( u cu C( u su S( u yu Y( u su S( u tu T( u eu E( u mu M( R R% R R R& R R R R R R, R R5 t afterDoctypePublicKeywordStatet afterDoctypeSystemKeywordStatet bogusDoctypeState( R RJ R� R� ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR� O sT c C` s� | j j � } | t k r* | j | _ n� | d k rw | j j i t d d 6d d 6� | j j | � | j | _ ny | t k r� | j j i t d d 6d d 6� t | j d <| j j | j � | j | _ n | j j | � | j | _ t S( Nu 'u "u ParseErroru typeu unexpected-char-in-doctypeu datau eof-in-doctypeu correct( u 'u "( R R% R t" beforeDoctypePublicIdentifierStateR R R& R R, R R R R R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR� � s" c C` sg | j j � } | t k r nE| d k rF d | j d <| j | _ n | d k rn d | j d <| j | _ n� | d k r� | j j i t d d 6d d 6� t | j d <| j j | j � | j | _ n� | t k r(| j j i t d d 6d d 6� t | j d <| j j | j � | j | _ n; | j j i t d d 6d d 6� t | j d <| j | _ t S( Nu "u u publicIdu 'u >u ParseErroru typeu unexpected-end-of-doctypeu datau correctu eof-in-doctypeu unexpected-char-in-doctype( R R% R R t( doctypePublicIdentifierDoubleQuotedStateR t( doctypePublicIdentifierSingleQuotedStateR R& R R R R R� R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR� � s4 c C` s? | j j � } | d k r* | j | _ n| d k rn | j j i t d d 6d d 6� | j d c d 7<n� | d k r� | j j i t d d 6d d 6� t | j d <| j j | j � | j | _ np | t k r(| j j i t d d 6d d 6� t | j d <| j j | j � | j | _ n | j d c | 7<t S( Nu "u u ParseErroru typeu invalid-codepointu datau publicIdu �u >u unexpected-end-of-doctypeu correctu eof-in-doctype( R R% t! afterDoctypePublicIdentifierStateR R R& R R R R R R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR� � s* c C` s? | j j � } | d k r* | j | _ n| d k rn | j j i t d d 6d d 6� | j d c d 7<n� | d k r� | j j i t d d 6d d 6� t | j d <| j j | j � | j | _ np | t k r(| j j i t d d 6d d 6� t | j d <| j j | j � | j | _ n | j d c | 7<t S( Nu 'u u ParseErroru typeu invalid-codepointu datau publicIdu �u >u unexpected-end-of-doctypeu correctu eof-in-doctype( R R% R� R R R& R R R R R R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR� � s* c C` s� | j j � } | t k r* | j | _ nZ| d k rX | j j | j � | j | _ n,| d k r� | j j i t d d 6d d 6� d | j d <| j | _ n� | d k r� | j j i t d d 6d d 6� d | j d <| j | _ n� | t k rI| j j i t d d 6d d 6� t | j d <| j j | j � | j | _ n; | j j i t d d 6d d 6� t | j d <| j | _ t S( Nu >u "u ParseErroru typeu unexpected-char-in-doctypeu datau u systemIdu 'u eof-in-doctypeu correct( R R% R t- betweenDoctypePublicAndSystemIdentifiersStateR R R& R R R t( doctypeSystemIdentifierDoubleQuotedStatet( doctypeSystemIdentifierSingleQuotedStateR R R� R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR� � s6 c C` s8 | j j � } | t k r n| d k rL | j j | j � | j | _ n� | d k rt d | j d <| j | _ n� | d k r� d | j d <| j | _ n� | t k r� | j j i t d d 6d d 6� t | j d <| j j | j � | j | _ n; | j j i t d d 6d d 6� t | j d <| j | _ t S( Nu >u "u u systemIdu 'u ParseErroru typeu eof-in-doctypeu datau correctu unexpected-char-in-doctype( R R% R R R& R R R R� R� R R R R� R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR� s. c C` s� | j j � } | t k r* | j | _ n� | d k rw | j j i t d d 6d d 6� | j j | � | j | _ ny | t k r� | j j i t d d 6d d 6� t | j d <| j j | j � | j | _ n | j j | � | j | _ t S( Nu 'u "u ParseErroru typeu unexpected-char-in-doctypeu datau eof-in-doctypeu correct( u 'u "( R R% R t" beforeDoctypeSystemIdentifierStateR R R& R R, R R R R R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR� s" c C` sg | j j � } | t k r nE| d k rF d | j d <| j | _ n | d k rn d | j d <| j | _ n� | d k r� | j j i t d d 6d d 6� t | j d <| j j | j � | j | _ n� | t k r(| j j i t d d 6d d 6� t | j d <| j j | j � | j | _ n; | j j i t d d 6d d 6� t | j d <| j | _ t S( Nu "u u systemIdu 'u >u ParseErroru typeu unexpected-char-in-doctypeu datau correctu eof-in-doctype( R R% R R R� R R� R R& R R R R R� R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR� / s4 c C` s? | j j � } | d k r* | j | _ n| d k rn | j j i t d d 6d d 6� | j d c d 7<n� | d k r� | j j i t d d 6d d 6� t | j d <| j j | j � | j | _ np | t k r(| j j i t d d 6d d 6� t | j d <| j j | j � | j | _ n | j d c | 7<t S( Nu "u u ParseErroru typeu invalid-codepointu datau systemIdu �u >u unexpected-end-of-doctypeu correctu eof-in-doctype( R R% t! afterDoctypeSystemIdentifierStateR R R& R R R R R R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR� L s* c C` s? | j j � } | d k r* | j | _ n| d k rn | j j i t d d 6d d 6� | j d c d 7<n� | d k r� | j j i t d d 6d d 6� t | j d <| j j | j � | j | _ np | t k r(| j j i t d d 6d d 6� t | j d <| j j | j � | j | _ n | j d c | 7<t S( Nu 'u u ParseErroru typeu invalid-codepointu datau systemIdu �u >u unexpected-end-of-doctypeu correctu eof-in-doctype( R R% R� R R R& R R R R R R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR� d s* c C` s� | j j � } | t k r n� | d k rL | j j | j � | j | _ n� | t k r� | j j i t d d 6d d 6� t | j d <| j j | j � | j | _ n. | j j i t d d 6d d 6� | j | _ t S( Nu >u ParseErroru typeu eof-in-doctypeu datau correctu unexpected-char-in-doctype( R R% R R R& R R R R R R R� R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR� | s c C` s | j j � } | d k r= | j j | j � | j | _ n> | t k r{ | j j | � | j j | j � | j | _ n t S( Nu >( R R% R R& R R R R R, R5 ( R RJ ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR� � s c C` sN g } x� t r� | j | j j d � � | j | j j d � � | j j � } | t k r` Pq | d d d k r� | d d | d <Pq | j | � q Wd j | � } | j d � } | d k rx6 t | � D]( } | j j i t d d 6d d 6� q� W| j d d � } n | r>| j j i t d d 6| d 6� n | j | _ t S( Nu ]u >i����i����u ]]u u i u ParseErroru typeu invalid-codepointu datau �u Characters( R5 R& R RI R% R R( t countt rangeR R R| R R ( R RJ R% t nullCountRw ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR� � s. N(N t __name__t __module__t __doc__R R R$ R4 R RB RC RF R RG RN RL RP RR RS RH RU RV RM R[ R\ RO R_ R` RQ Ra Rc Rb Rd Rh Rf Re Rg Ri Rk Rj Rl Rm Ro Rn Rp RX Rq Rs Rr Rx Rz Ry R{ RY RW RT R} R� R� R� R� R� R~ R� R� R� R� R� R� R� R� R� R� R� R� R� R� R� R� ( ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyR s� HP # 6 " - 3 N( t __future__R R R t pip._vendor.sixR R* t collectionsR t constantsR R R R R R R R R R t _inputstreamR t _trieR R6 t objectR ( ( ( sC /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_tokenizer.pyt <module> s