Edit File: processor.cpython-37.pyc
B V�dy � @ s� d Z ddlZddlZddlZddlZddlmZmZ ddlmZm Z m Z ddl m Z m Z ddlmZ ddlmZ dd lmZ dd lmZ d d lmZmZmZmZ d d lm Z d dl m Z m Z m Z e G dd� de��Z!dS )z- This module contains RequestProcessor class � N)�datetime� timedelta)�Thread�RLock�current_thread)�Callable�Any� )� AutoTracer)�Common)� DecisionMaker)�StatisticsSender� )� session_scope�setup_database� RequestResult�cleanup_old_data)�SSAError)� singleton� url_split�switch_schedstatsc s� e Zd ZdZdZd#� fdd� Zedd� �Zed�d d �Z dd �d d �Z dd �dd�Z e e d�dd�Zd$ed�dd�Zdd� Zeeed�dd��Zd d � Zd%d d �Zedd �d!d"�Z� ZS )&�RequestProcessorz^ SSA Request processor implementation. Only one instance is allowed to be created �d Nc sx t � �� t�d�| _| j�dt� tdd� |r6|nt� | _ t � | _ t |d�| _ t� | _t|d�| _| �� g | _d S )NZ req_processorzProcessor enabled: %sT)�enabled)�engine)�super�__init__�logging� getLogger�logger�info� __package__r r r r �_lockr �decision_makerr �senderr � auto_tracer�start_background_routine�_buffer)�selfr )� __class__� �F/opt/alt/python37/lib/python3.7/site-packages/ssa/modules/processor.pyr ) s zRequestProcessor.__init__c C s | j d S )zp Return config file value multiplied by 1000000, as we receive duration in microseconds i@B )Zrequests_duration)r( r* r* r+ �configured_duration= s z$RequestProcessor.configured_duration)�reportc C sJ y| j �|� W n4 tk rD } z| j�dt|�� W dd}~X Y nX dS )z( Call Statistics Sender zStatisticsSender failed: %sN)r$ �sendr r �error�str)r( r- �er* r* r+ � send_statsE s zRequestProcessor.send_stats)�returnc C s* t | jdd�}|�� | j�d|j� dS )zA Start dumper|DecisionMaker thread in background T)�target�daemonz[%s] Routine startedN)r �background_routine�startr r �name)r( �tr* r* r+ r& N s z)RequestProcessor.start_background_routinec C s� x�t �� }|jdkr�|jdkr�| j�dt� j|� | �| j � | j�dt� j|� | �| j � | j�dt� j|� | �| j �}| j�dt� j|� t | j � | �| j|� tdd� | �d� q| �|j� qW d S ) z� Dumps collected stats to file once an hour. Runs DecisionMaker once a day Cleanup storage after DecisionMaker run r z2[%s] Routine thread launching buffer flushing (%s)z-[%s] Routine thread launching AutoTracer (%s)z0[%s] Routine thread launching DecisionMaker (%s)z*[%s] Routine thread launching cleanup (%s)T)r �< N)r �now�minute�hourr r r r8 � _safe_exec�flush_with_lockr% r# r r r2 r � _simple_sleep�_sleep_till_next_hour)r( Ztickr- r* r* r+ r6 V s0 z#RequestProcessor.background_routine)�actionr3 c G sR y||� S t k rL t�� \}}}| jjdt|�||dt�� id� Y nX dS )zACall requested Callable with given args and capture any exceptionz %s failed with exception %s, %sZorig_traceback)�extraN)� Exception�sys�exc_infor � exceptionr0 � traceback� format_exc)r( rB �args�et�ev�_r* r* r+ r> y s zRequestProcessor._safe_exec� )�to_sleepc C s" | j �dt� j|� t�|� dS )zP Log and sleep given number of seconds or 15 minutes by default z%[%s] Routine thread sleeping for (%s)N)r r r r8 �time�sleep)r( rO r* r* r+ r@ � s z RequestProcessor._simple_sleepc C s* t dd�t |d� �� }| �t|�� dS )zF Sleep the number of minutes remaining till next hour r )�hours)�minutesN)r � total_secondsr@ �int)r( Z start_minuteZ sleep_forr* r* r+ rA � s z&RequestProcessor._sleep_till_next_hour)� timestampr3 c C s t �| �jS )zN Takes an hour of a day, to which the given timestamp belongs )r � fromtimestampr= )rV r* r* r+ �get_interval_for� s z!RequestProcessor.get_interval_forc C s4 | j � | jd d � }g | _W d Q R X | �|� d S )N)r" r' � flush_buffer)r( �objectsr* r* r+ r? � s z RequestProcessor.flush_with_lockc C s: |dkr| j }|sdS t| j��}|�|� W dQ R X dS )z6 Save in-memory buffer into database. N)r' r r Zbulk_save_objects)r( rZ �dbr* r* r+ rY � s z RequestProcessor.flush_buffer)�datar3 c C s� |� d�}| �|�r&| j�d|� dS t|�\}}| j�dt� j� g }| j�n | j� t |||d |d |d | j k|d |d |d |d d � � t | j�| j kr�| jdd� }g | _W dQ R X | �|� | j�d t� j� dS ) z, Process given request data �urlz %s ignoredNz-[%s] Acquires lock to handle request countersrV �duration�hitting_limits�throttled_time�io_throttled_time� wordpress) �domain�pathrV r^ Zis_slow_requestr_ r` ra rb z-[%s] Released lock to handle request counters)�getZ is_ignoredr �debugr r r8 r" r' �appendr r, �len� BUFFER_SIZErY )r( r\ r] rc �uriZobjects_per_threadr* r* r+ �handle� s4 zRequestProcessor.handle)N)rN )N)�__name__� __module__� __qualname__�__doc__ri r �propertyr, �dictr2 r& r6 r r r> rU r@ rA � staticmethodrX r? rY rk � __classcell__r* r* )r) r+ r s # r )"ro r rE rP rH r r � threadingr r r �typingr r Z autotracerr �commonr r# r Z stat_senderr r[ r r r r �internal.exceptionsr �internal.utilsr r r r r* r* r* r+ �<module> s