Edit File: models.cpython-37.pyc
B �b�3 � @ s� d dl Z d dlmZ d dlmZ d dlmZ d dlmZ d dl m Z d dl m Z m Z mZmZmZmZmZ dd lmZmZ dd lmZ dd lmZmZmZ G d d � d �Z G dd� d�Z ee e f Z ee Z!G dd� d�Z"dS )� N)�Counter)�aliases)�sha256)�dumps)�sub)�Any�Dict�Iterator�List�Optional�Tuple�Union� )�NOT_PRINTABLE_PATTERN�TOO_BIG_SEQUENCE)� mess_ratio)� iana_name�is_multi_byte_encoding� unicode_rangec @ s e Zd ZdBeeeedee d�dd�Ze ed�dd�Z e ed�d d �Z e ed �d d ��Z e ed �dd��Ze ed �dd��Ze ed �dd��Zed �dd�Zed �dd�Zd dd�dd�Ze ed �dd��Ze ee d �d d ��Ze ed �d d ��Ze ed �d d!��Ze ee d �d"d#��Ze ed �d$d%��Ze ed �d&d'��Z e ed �d(d)��Z e ed �d*d+��Z e ed �d,d-��Z e ed �d.d/��Z e ed d �d0d1��Z!e ed �d2d3��Z"e ee d �d4d5��Z#e ee d �d6d7��Z$d d �d8d9�Z%d d �d:d;�Z&dCeed=�d>d?�Z'e ed �d@dA��Z(dS )D� CharsetMatchN�CoherenceMatches)�payload�guessed_encoding�mean_mess_ratio�has_sig_or_bom� languages�decoded_payloadc C sF || _ || _|| _|| _|| _d | _g | _d| _d | _d | _ || _ d S )Ng ) �_payload� _encoding�_mean_mess_ratio� _languages�_has_sig_or_bom�_unicode_ranges�_leavesZ_mean_coherence_ratio�_output_payload�_output_encoding�_string)�selfr r r r r r � r( �J/opt/alt/python37/lib/python3.7/site-packages/charset_normalizer/models.py�__init__ s zCharsetMatch.__init__)�other�returnc C s> t |t�s&td�t|j�t| j����| j|jko<| j|jkS )Nz&__eq__ cannot be invoked on {} and {}.)� isinstancer � TypeError�format�str� __class__�encoding� fingerprint)r' r+ r( r( r) �__eq__( s zCharsetMatch.__eq__c C sv t |t�st�t| j|j �}t| j|j �}|dk rj|dkrj|dkr^| j|jkr^| j|jkS | j|jkS | j|jk S )zQ Implemented to make sorted available upon CharsetMatches items. g{�G�z�?g{�G�z�?g )r- r � ValueError�abs�chaos� coherence�multi_byte_usage)r' r+ Zchaos_differenceZcoherence_differencer( r( r) �__lt__1 s zCharsetMatch.__lt__)r, c C s dt t| ��t | j� S )Ng �?)�lenr0 �raw)r' r( r( r) r9 D s z CharsetMatch.multi_byte_usagec C s t �dt� tt| �d�S )z� Check once again chaos in decoded text, except this time, with full content. Use with caution, this can be very slow. Notice: Will be removed in 3.0 z=chaos_secondary_pass is deprecated and will be removed in 3.0g �?)�warnings�warn�DeprecationWarningr r0 )r' r( r( r) �chaos_secondary_passH s z!CharsetMatch.chaos_secondary_passc C s t �dt� dS )zy Coherence ratio on the first non-latin language detected if ANY. Notice: Will be removed in 3.0 z<coherence_non_latin is deprecated and will be removed in 3.0g )r= r> r? )r' r( r( r) �coherence_non_latinU s z CharsetMatch.coherence_non_latinc C s, t �dt� ttdt| ��� �}t|�� �S )z_ Word counter instance on decoded text. Notice: Will be removed in 3.0 z2w_counter is deprecated and will be removed in 3.0� ) r= r> r? r r r0 �lowerr �split)r' Zstring_printable_onlyr( r( r) � w_countera s zCharsetMatch.w_counterc C s" | j d kr t| j| jd�| _ | j S )N�strict)r& r0 r r )r' r( r( r) �__str__o s zCharsetMatch.__str__c C s d� | j| j�S )Nz <CharsetMatch '{}' bytes({})>)r/ r2 r3 )r' r( r( r) �__repr__u s zCharsetMatch.__repr__c C s8 t |t�r|| kr"td�|j���d |_| j�|� d S )Nz;Unable to add instance <{}> as a submatch of a CharsetMatch)r- r r5 r/ r1 r& r# �append)r' r+ r( r( r) � add_submatchx s zCharsetMatch.add_submatchc C s | j S )N)r )r' r( r( r) r2 � s zCharsetMatch.encodingc C sH g }x>t �� D ]2\}}| j|kr,|�|� q| j|kr|�|� qW |S )z� Encoding name are known by many name, using this could help when searching for IBM855 when it's listed as CP855. )r �itemsr2 rI )r' Z also_known_as�u�pr( r( r) �encoding_aliases� s z CharsetMatch.encoding_aliasesc C s | j S )N)r! )r' r( r( r) �bom� s zCharsetMatch.bomc C s | j S )N)r! )r' r( r( r) �byte_order_mark� s z CharsetMatch.byte_order_markc C s dd� | j D �S )z� Return the complete list of possible languages found in decoded sequence. Usually not really useful. Returned list may be empty even if 'language' property return something != 'Unknown'. c S s g | ] }|d �qS )r r( )�.0�er( r( r) � <listcomp>� s z*CharsetMatch.languages.<locals>.<listcomp>)r )r' r( r( r) r � s zCharsetMatch.languagesc C sp | j sbd| jkrdS ddlm}m} t| j�r8|| j�n|| j�}t|�dksVd|krZdS |d S | j d d S )z� Most probable language found in decoded sequence. If none were detected or inferred, the property will return "Unknown". �asciiZEnglishr )�encoding_languages�mb_encoding_languagesz Latin Based�Unknown)r �could_be_from_charsetZcharset_normalizer.cdrU rV r r2 r; )r' rU rV r r( r( r) �language� s zCharsetMatch.languagec C s | j S )N)r )r' r( r( r) r7 � s zCharsetMatch.chaosc C s | j s dS | j d d S )Ng r r )r )r' r( r( r) r8 � s zCharsetMatch.coherencec C s t | jd dd�S )N�d � )�ndigits)�roundr7 )r' r( r( r) � percent_chaos� s zCharsetMatch.percent_chaosc C s t | jd dd�S )NrZ r[ )r\ )r] r8 )r' r( r( r) �percent_coherence� s z CharsetMatch.percent_coherencec C s | j S )z+ Original untouched bytes. )r )r' r( r( r) r< � s zCharsetMatch.rawc C s | j S )N)r# )r' r( r( r) �submatch� s zCharsetMatch.submatchc C s t | j�dkS )Nr )r; r# )r' r( r( r) � has_submatch� s zCharsetMatch.has_submatchc C s@ | j d k r| j S dd� t| �D �}ttdd� |D ���| _ | j S )Nc S s g | ] }t |��qS r( )r )rQ �charr( r( r) rS � s z*CharsetMatch.alphabets.<locals>.<listcomp>c S s h | ] }|r|�qS r( r( )rQ �rr( r( r) � <setcomp>� s z)CharsetMatch.alphabets.<locals>.<setcomp>)r" r0 �sorted�list)r' Zdetected_rangesr( r( r) � alphabets� s zCharsetMatch.alphabetsc C s | j gdd� | jD � S )z� The complete list of encoding that output the exact SAME str result and therefore could be the originating encoding. This list does include the encoding available in property 'encoding'. c S s g | ] }|j �qS r( )r2 )rQ �mr( r( r) rS � s z6CharsetMatch.could_be_from_charset.<locals>.<listcomp>)r r# )r' r( r( r) rX � s z"CharsetMatch.could_be_from_charsetc C s | S )z> Kept for BC reasons. Will be removed in 3.0. r( )r' r( r( r) �first� s zCharsetMatch.firstc C s | S )z> Kept for BC reasons. Will be removed in 3.0. r( )r' r( r( r) �best� s zCharsetMatch.best�utf_8)r2 r, c C s2 | j dks| j |kr,|| _ t| ��|d�| _| jS )z� Method to get re-encoded bytes payload using given target encoding. Default to UTF-8. Any errors will be simply ignored by the encoder NOT replaced. N�replace)r% r0 �encoder$ )r' r2 r( r( r) �output s zCharsetMatch.outputc C s t | �� ��� S )zw Retrieve the unique SHA256 computed using the transformed (re-encoded) payload. Not the original one. )r rn � hexdigest)r' r( r( r) r3 s zCharsetMatch.fingerprint)N)rk ))�__name__� __module__� __qualname__�bytesr0 �float�boolr r* �objectr4 r: �propertyr9 r@ rA r rE rG rH rJ r2 r rN rO rP r rY r7 r8 r^ r_ r< r` ra rg rX ri rj rn r3 r( r( r( r) r sb r c @ s� e Zd ZdZdee d�dd�Zee d�dd�Ze e e f ed �d d �Z e d�d d �Z ed�dd�Zedd �dd�Zed d�dd�Zed d�dd�ZdS )�CharsetMatchesz� Container with every CharsetMatch items ordered by default from most probable to the less one. Act like a list(iterable) but does not implements all related methods. N)�resultsc C s |r t |�ng | _d S )N)re �_results)r' ry r( r( r) r* s zCharsetMatches.__init__)r, c c s | j E d H d S )N)rz )r' r( r( r) �__iter__ s zCharsetMatches.__iter__)�itemr, c C sN t |t�r| j| S t |t�rFt|d�}x | jD ]}||jkr0|S q0W t�dS )z� Retrieve a single item either by its position or encoding name (alias may be used here). Raise KeyError upon invalid index or encoding not present in results. FN)r- �intrz r0 r rX �KeyError)r' r| �resultr( r( r) � __getitem__! s zCharsetMatches.__getitem__c C s t | j�S )N)r; rz )r' r( r( r) �__len__/ s zCharsetMatches.__len__c C s t | j�dkS )Nr )r; rz )r' r( r( r) �__bool__2 s zCharsetMatches.__bool__c C s~ t |t�s td�t|j����t|j�tkrbx4| j D ]*}|j |j kr4|j |j kr4|� |� dS q4W | j � |� t| j �| _ dS )z~ Insert a single match. Will be inserted accordingly to preserve sort. Can be inserted as a submatch. z-Cannot append instance '{}' to CharsetMatchesN)r- r r5 r/ r0 r1 r; r< r rz r3 r7 rJ rI re )r' r| �matchr( r( r) rI 5 s zCharsetMatches.appendr c C s | j s dS | j d S )zQ Simply return the first match. Strict equivalent to matches[0]. Nr )rz )r' r( r( r) rj I s zCharsetMatches.bestc C s | � � S )zP Redundant method, call the method best(). Kept for BC reasons. )rj )r' r( r( r) ri Q s zCharsetMatches.first)N)rp rq rr �__doc__r r r* r r{ r r} r0 r� r� ru r� rI r rj ri r( r( r( r) rx s rx c @ sj e Zd Zeee ee ee eee eeeee ed� dd�Ze e ee f d�dd��Z ed�dd�Z d S ) �CliDetectionResult) �pathr2 rN �alternative_encodingsrY rg r r7 r8 � unicode_path� is_preferredc C sF || _ | | _|| _|| _|| _|| _|| _|| _|| _| | _ | | _ d S )N) r� r� r2 rN r� rY rg r r7 r8 r� ) r' r� r2 rN r� rY rg r r7 r8 r� r� r( r( r) r* ] s zCliDetectionResult.__init__)r, c C s2 | j | j| j| j| j| j| j| j| j| j | j d� S )N) r� r2 rN r� rY rg r r7 r8 r� r� ) r� r2 rN r� rY rg r r7 r8 r� r� )r' r( r( r) �__dict__w s zCliDetectionResult.__dict__c C s t | jddd�S )NT� )� ensure_ascii�indent)r r� )r' r( r( r) �to_json� s zCliDetectionResult.to_jsonN)rp rq rr r0 r r ru rt r* rw r r r� r� r( r( r( r) r� \ s r� )#r= � collectionsr Zencodings.aliasesr �hashlibr �jsonr �rer �typingr r r r r r r Zconstantr r Zmdr �utilsr r r r rx r0 rt ZCoherenceMatchr r� r( r( r( r) �<module> s $ C