Edit File: autotracer.cpython-37.pyc
B V�d�S � @ s� d Z dZddlZddlZddlZddlmZ ddlmZm Z m Z ddl m Z m Z ddl mZ ddlmZmZmZmZmZ dd lmZ ddlZdd lmZ d d lm Z d d l m Z m Z m Z ddl!m"Z"m#Z#m$Z$ ddl%m&Z& ddl'm(Z( ddl)m*Z* ddl+m,Z,m-Z-m.Z. dZ/yddl0m1Z2 W n, e3k �rD Z4 z e4Z/dZ2W ddZ4[4X Y nX eG dd� d��Z5eG dd� d��Z6eG dd� d��Z7G dd� de�Z8e9e8d �d d �Z:G d d � d e �Z;e<d!k�r�e.� ej=d"ej>d#� e;� Z?e?� dS )$z' This module contains Autotracer class z ssa.modules� N)� defaultdict)� dataclass�field�asdict)�datetime� timedelta)� fnmatchcase)�Iterator�Optional� NamedTuple�List�Iterable)�urlparse)�setup_database� )�Common)�iter_domains_data�iter_urls_data�get_url_durations� )�status�disabled_users�is_edition_supported)� load_tunables)�autotracing_tunables_schema)�autotracing_stats_file)�is_xray_version_supported�is_kernel_version_supported� sentry_initc @ sJ e Zd ZU dZeed< eed< eed< eed< eed< eed< eed< d S ) �URLzURL data container�uri� avg_duration� max_duration�total_slow_reqs� total_reqs� belongs_to�belongs_to_userN)�__name__� __module__� __qualname__�__doc__�str�__annotations__�int� r. r. �G/opt/alt/python37/lib/python3.7/site-packages/ssa/modules/autotracer.pyr 2 s r c @ s� e Zd ZU dZdZeed< dZeed< dZeed< dZ eed< dZ eed< dZ eed< dZ eed < dZ eed < dZeed < dZeed < dZeed < dS )� RulesRejectszRejects counters containerr �non_wp� throttled�density�slowness� max_slowness�disabled�nginx� frequency� server_limit� domain_limit� no_domainN)r' r( r) r* r1 r- r, r2 r3 r4 r5 r6 r7 r8 r9 r: r; r. r. r. r/ r0 > s r0 c @ sD e Zd ZU dZeed< dZeed< dZeed< e e d�Z e ed< dS ) �Statsz AutoTracer statistics container� rules_versionr �urls_processed� urls_selected)�default_factory�rejectsN) r' r( r) r* r+ r, r> r- r? r r0 rA r. r. r. r/ r< N s r< c @ s" e Zd ZU dZeed< eed< dS )�URLSz" Representation of an URL � domain_name�uri_pathN)r' r( r) r* r+ r, r. r. r. r/ rB W s rB )�url�returnc C s\ t | �}|jrd|j� �nd}|jr2|j� |� �nd}|j�dd�}|�d�d }t||�S )z� Split URL into domain_name and uripath including query string :param url: URL of format protocol://domain/path;parameters?query#fragment :return: namedtuple URL(domain_name, uripath) �?� �/zwww.�:r )r �query�path�netloc�replace�splitrB )rE � fragments�qsr Z_no_www_netlocZ_no_port_netlocr. r. r/ � url_split_ s rR c s� e Zd ZdZd7� fdd� Z� fdd�Zeed�dd ��Zee e e d �d d ��Z e e d �dd�Zed�dd�Zd8e e ed�dd�Ze e d�dd�Zeee d�dd�Zee e d�d d �Zee e d�d d �Ze e ee d �d!d"�Ze e e d#�d$d%�Zee d�d&d'�Z d9e e ee d(�d)d*�Z ee d�d+d,�Z e e d�d-d.�Z dd�d/d0�Z!dd�d1d2�Z"ed�d3d4�Z#ed�d5d6�Z$� Z%S ):� AutoTracerz0 SSA autotracing module implementation. Nc sN t � �� t�d�| _| j�dt� t� | _t� | _ d | _ |rB|nt � | _ d S )NZ auto_tracerzAutoTracer enabled: %s) �super�__init__�logging� getLogger�logger�info� __package__�listr � tasks_list�statsr �engine)�selfr^ )� __class__r. r/ rU r s zAutoTracer.__init__c sF t � s| j�d� d S t� s,| j�d� d S t� sB| j�d� d S td krztj�d�rj| j� dt t �� n | j�d� d S tj }|d k �r:| j�d� t � �� | j�d| j� t| j�| _x|| �� D ]p}| j�d |j� | j jd 7 _y||j| jd � W q� tk �r4 } z| j� d t |�� W d d }~X Y q�X q�W | �� d S ) Nz*AutoTracer skipped: unsupported CL editionz.AutoTracer skipped: unsupported kernel versionz-AutoTracer skipped: unsupported X-Ray versionz/opt/alt/php-xrayz1AutoTracer skipped: X-ray module import error: %sz'AutoTracer skipped: X-Ray not installedzAutoTracer startedz AutoTracer loaded config: %szStarting auto task for %sr )rE Z tracing_countzFailed to start task: %s)r rX rY r r �xray_lib�osrL �exists�errorr+ �xray_import_errorZstart_autotracingrT rU �debug�configr< r= r] �urls_scheduledr r? �request_number� Exception�save_iteration_stats)r_ Z start_toolrE �e)r` r. r/ �__call__| s8 (zAutoTracer.__call__)rF c C s t dt�S )z$ Load configuration zautotracing.json)r r r. r. r. r/ � load_conf� s zAutoTracer.load_conf)rC rF c C s t dkr dS t �| �S )z; Gets domain info for the specified domain N)ra � domain_info)rC r. r. r/ �gets_domaininfo� s zAutoTracer.gets_domaininfo)� user_namerF c C s" | j r tdkrdS t�|�jS dS )z> Says if nginx is enabled for a specific user NF)Z skip_nginxra ZNginxUserCache� is_enabled)r_ rq r. r. r/ �nginx_is_enabled� s zAutoTracer.nginx_is_enabledc C s, t dd� | jD ��}| j�d|� | j| S )z>Recalculate limit per server taking into account running tasksc S s( g | ] }|d dkr|d dkr|�qS )r �running�userz *autotracing*r. )�.0�taskr. r. r/ � <listcomp>� s z<AutoTracer.per_server_limit_recalculated.<locals>.<listcomp>z&Number of running autotracing tasks %s)�lenr\ rX rf Zper_server_limit)r_ Z running_countr. r. r/ � per_server_limit_recalculated� s z(AutoTracer.per_server_limit_recalculated)� full_listrF c sD |dkr t �� }| j�d|� t�� t| jd� � � fdd�|D �S )zQ Excludes tasks older than N days from the general list of tasks NzTask list loaded %s)�daysc s8 g | ]0}|� d �dk r |� d �ndt� �� �kr|�qS )Z createtimeNr )�getr- � timestamp)rv rw )� n_days_agor. r/ rx � s z1AutoTracer.excludes_old_tasks.<locals>.<listcomp>)ra r\ rX rf r �nowr Zmin_retracing_interval)r_ r{ r. )r r/ �excludes_old_tasks� s z AutoTracer.excludes_old_tasks)� current_urlrF c C s| t |�}xn| jD ]d}t |d �}|j|jko6|j|jk}t|j|j�oRt|j|j�}|s\|r| j�d||d � dS qW dS )z� Excludes url from the list if it completely matches the current url or if domain names match and "*" follows the domain name in the list rE z2Skipped: URL %s was traced recently. Matched by %sTF)rR r\ rC rD r rX rf )r_ r� �cZ task_data�tZ direct_matchZwildcard_matchr. r. r/ �exclude_thesame_urls� s z AutoTracer.exclude_thesame_urls)�url_total_reqs�domain_total_reqsrF c C s4 | j r0t�t�||��}| j�d|� || jkS dS )z-Check that URL density passes given thresholdzCalculated density %sT)r3 �npZaminZcorrcoefrX rf Zdensity_threshold)r_ r� r� Z url_densityr. r. r/ �pass_by_density� s zAutoTracer.pass_by_density)� url_durationsrF c C s0 d| j }t�||�}| j�d||� || jkS )z� The measure of "slowness" for URL is: at least N% of its requests take more than X seconds. N% -- self.slow_duration_percentage X -- self.slow_duration_threshold �d z.Calculated %sth percentile %s for min duration)Zslow_duration_percentager� � percentilerX rf Zslow_duration_threshold)r_ r� Zreversed_percentileZreversed_percentile_valuer. r. r/ �pass_by_slowness_percentile� s z&AutoTracer.pass_by_slowness_percentilec C s* t �|| j�}| j�d| j|� || jkS )a The opposite to pass_by_slowness_percentile method. The measure of "much slowness" for URL is: at least N% of its requests must take less than X seconds. N% -- self.max_slow_duration_percentage X -- self.max_slow_duration_threshold z.Calculated %sth percentile %s for max duration)r� r� Z max_slow_duration_percentagerX rf Zmax_slow_duration_threshold)r_ r� Zpercentile_valuer. r. r/ � pass_by_max_slowness_percentile s z*AutoTracer.pass_by_max_slowness_percentile)�url_throttled_reqsr� rF c C s8 |dkr dS t |�t |� d }| j�d|� || jkS )zY Check that percent of throttled requests per URL passes given threshold NFr� z Calculated throttled percent %s)�sumrX rf Z allowed_throttling_percentage)r_ r� r� Zthrottled_percentr. r. r/ �pass_by_allowed_throttling s z%AutoTracer.pass_by_allowed_throttling)� wp_statusrF c C s | j rt|�S dS )zx Check that URLs of a particular domain should be analyzed. For now we skip non-wordpress sites T)Zonly_wp�bool)r_ r� r. r. r/ �pass_by_engine s zAutoTracer.pass_by_enginec c s �x t | j�D �]�}d}| �|j�sZ| j�d� |j}| jj j |7 _ | j j |7 _ qt t | j|j ��}�x�t| j|j t|�� ��D �]|\}}|| jkr�q�| j�d|� | j j d7 _ | �|�d�|d �s�| j�d� | jj jd7 _q�| �|d |j��s&| j�d� | jj jd7 _q�|| }| �|��sZ| j�d � | jj jd7 _q�| �|��s�| j�d � | jj jd7 _q�|dk�r�| � |j �}|dk�r�| j�d � | jj j d7 _ q�n|j }t |t t!�"|��t#|d �t#|d �t$|�|j |d �V q�W qW dS )a Select all URLs suitable for auto tracing by very basic rules: - WP site - suitable throttling - suitable density - measure of "slow" URL ORDER OF RULES MUST NOT BE CHANGED: IT AFFECTS STATISTICS COUNTERS Nz Skipped by engine: non-wordpresszProcessing URL %sr r� r� z Skipped by throttled percentzSkipped by densityz Skipped by slowness percentilez"Skipped by max slowness percentilez"Skipped by unavailable domain infoZ url_slow_reqs)r! r# r$ r" r% r& )%r r^ r� Zis_a_wordpress_domainrX rf � urls_numberr] rA r1 r> �dictr rC r r[ �keysZnon_url_fieldsr� r} r2 r� r� r3 r� r4 r� r5 rp r; ru r r- r� Zmeanr� �max) r_ � domain_data� domain_ownerZ skipped_countZdomain_url_durationsrE �dataZ durationsro r. r. r/ � urls_computationally_filtered& sb z(AutoTracer.urls_computationally_filtered)�stats_collectedrF c c s� | � � xv| �� D ]j}|j| jkrF| j�d|j� | jj jd7 _q| � |j�rv| j�d|j� | jj j d7 _ q|V qW dS )z� From selected by computed thresholds URLs take those for which: - autotracing enabled - nginx disabled ORDER OF RULES MUST NOT BE CHANGED: IT AFFECTS STATISTICS COUNTERS z'Skipped: autotracing is disabled for %sr z Skipped: nginx is enabled for %sN) �fill_in_disabled_usersr� r& r rX rf r] rA r6 rs r7 )r_ r� rE r. r. r/ r? | s zAutoTracer.urls_selectedc c s� d}t t�}| �� }| j�d|� | �� | _| �� }x�|D ]�}| �|j �r^| j j j d7 _ q:||k r�||j | jk r�|d7 }||j d7 < |V q�| j�d|j | j� | j j jd7 _q:| j�d|j |� | j j jd7 _q:W dS )a= Schedule autotracing by sorted list taking into account the limits: - no same task for 10 days - limit per server - limit per domain ORDER OF RULES MUST NOT BE CHANGED: IT AFFECTS STATISTICS COUNTERS Return resulting list of URLs scheduled for auto tracing r zSorted scheduled list %sr z#Skipped URL %s by domain limit (%s)z#Skipped URL %s by server limit (%s)N)r r- � urls_sortedrX rf r� r\ rz r� r r] rA r8 r% Zper_domain_limitr: r9 )r_ Zgeneral_tasks_counterZtasks_counter_per_domainZ sorted_urlsZper_server_smart_limitrE r. r. r/ rh � s. zAutoTracer.urls_scheduledc C s, t t| �� �dd� dd�}t |dd� dd�S )zd Sort URLs by total number of requests first and by average duration second c S s | j S )N)r! )�ur. r. r/ �<lambda>� � z(AutoTracer.urls_sorted.<locals>.<lambda>T)�key�reversec S s | j S )N)r$ )r� r. r. r/ r� � r� )�sortedr[ r? )r_ Z first_serier. r. r/ r� � s zAutoTracer.urls_sortedc C s t � | _ dS )z6 Fill internal list of disabled users N)r )r_ r. r. r/ r� � s z!AutoTracer.fill_in_disabled_usersc C sp | j dk rly,ttd��}t�t| j �|� W dQ R X W n4 tk rj } z| j�dt |�� W dd}~X Y nX dS )zI Save collected statistics for current iteration to file N�wz*Unable to save iteration stats to file: %s) r] �openr �json�dumpr �OSErrorrX �warningr+ )r_ � stats_filerl r. r. r/ rk � s z AutoTracer.save_iteration_statsc C sv yFt t��4}t�|�}t|d |d |d tf |d ��}W dQ R X W n& ttjtfk rl t| j �}Y nX t |�S )z@ Load statistics for latest iteration from file r= r> r? rA N) r� r r� �loadr< r0 r� �JSONDecodeError�KeyErrorr= r )r_ r� �_dataZ stat_datar. r. r/ �load_iteration_stats� s z AutoTracer.load_iteration_statsc C s* | � � }|�tt� d tt� �d�� |S )rH r )r Zdisabled_users_quantity)r� �updater� r ry r )r_ Z stats_loadedr. r. r/ � get_stats� s zAutoTracer.get_stats)N)N)N)&r' r( r) r* rU rm � staticmethodr� rn r+ r �objectrp r� rs r- rz r[ r� r� r� r r� r� r� r� r r r� r? rh r r� r� rk r� r� � __classcell__r. r. )r` r/ rS m s4 % W, rS �__main__zauto_tracer_standalone.log)�filename�level)@r* rZ r� rV rb � collectionsr Z dataclassesr r r r r �fnmatchr �typingr r r r r � urllib.parser Znumpyr� Zssa.dbr �commonr Zstorager r r � autotracingr r r � configurationr Zconfiguration.schemesr �internal.constantsr �internal.utilsr r r re Zxray.shared_libraryZshared_libraryra � ImportErrorrl r r0 r< rB r+ rR rS r' � basicConfig�DEBUGr� r. r. r. r/ �<module> sP