Edit File: decision_maker.cpython-37.pyc
B V�dC+ � @ s� d Z dZddlZddlZddlZddlmZ ddlZddl m Z ddl m Z ddl mZ dd lmZmZmZ d d lmZ d d lmZ d d lmZ d dlmZmZ G dd� de�Z e dkr�e� ej dej d� e � Z e � dS )z* This module contains DecisionMaker class z ssa.modules� N)�isfile)�is_cl_solo_edition)�setup_database� )�Common)�iter_domains_data�iter_urls_data�get_url_durations� )� load_tunables)�ssa_tunables_schema)� report_path)�previous_day_date� sentry_initc s" e Zd ZdZd/� fdd� Zdd� Zeed�dd ��Ze ed�d d ��Z e e d�d d ��Z e e d�dd��Zedd� �Ze d�dd�Zeed�dd�Zeed�dd�Zeed�dd �Zeeeeed �d d ��Zd d!� Ze d"�d#d$�Ze d�d%d&�Zeed'�d(d)�Zd*d+� Z eeed,�d-d.�Z � Z S )0� DecisionMakerz, SSA Decision maker implementation. Nc s8 t � �� t�d�| _| j�dt� |r,|nt� | _d S )NZdecision_makerzDecisionMaker enabled: %s) �super�__init__�logging� getLogger�logger�info� __package__r �engine)�selfr )� __class__� �K/opt/alt/python37/lib/python3.7/site-packages/ssa/modules/decision_maker.pyr ' s zDecisionMaker.__init__c C sZ | j �d� | j �d| j� | �� | _| j �d| j� | �� }| �|� | j �d|� |S )NzDecisionMaker startedz DecisionMaker loaded config: %sz!DecisionMaker loaded tunables: %szDecisionMaker report: %s)r r �debug�config�load_external_conf�external_tunables�data_processing�add_json_report)r �reportr r r �__call__. s zDecisionMaker.__call__)�returnc C s t j�t| �S )zE Full path to given filename in DM reports directory )�os�path�joinr )�namer r r � _report_file9 s zDecisionMaker._report_filec C s | � d�S )zU Full path to current DM report: report.json in DM reports directory z report.json)r* )r r r r �current_report_file@ s z!DecisionMaker.current_report_filec C s t t� g d�S )z& Returns empty report )�date�domains)�dictr )r r r r � _empty_reportG s zDecisionMaker._empty_reportc C s dhS )N� correlationr )r r r r �solo_filtered_optionsN s z#DecisionMaker.solo_filtered_optionsc C s t dt�S )z"Load external configuration valueszssa.json)r r r r r r r R s z DecisionMaker.load_external_confc C s� | j }�xnt| j�D �]^}t� }d}tt| j|j��}� x�t| j|jt|�� ��D ]�\}}| � |�rt| j � d|� qR|| j krR| � |�dtdgd ��|d �s�qR| �|d |j�}|�|�} | dkr�| j �dt|�� qR| �|d �rR| �|�rRt�| �} t|d �} || 7 }|�t|| t| �t|d ��d �� qRW |r| j|| j d d �} |d �t|jt | �|t|j�| d�� qW |d �r�| j|d | j dd �|d < |S )z� Going through the list of domains, for each domain we go through the list of urls. During data processing, we will form the resulting dictionary. r z %s ignored�url_throttled_reqs� �url_total_reqsNz Unable to get durations for %s� url_slow_reqsz.2f)r) �reqs_numZaverage_durationr0 r6 )� list_to_sort� leave_top�key_for_sortingr- )r) Z slow_urls� slow_reqs� total_reqs�urlsr: ) r/ r r �listr. r � domain_namer �keys� is_ignoredr r �non_url_fields�is_throttling_suitable�get�get_correlation�domain_total_reqs�error�str�request_number_exceeded�correlation_conditions�np�mean�sum�append�int�float�report_sorting� urls_number�len�domains_number) r r# � domain_data� urls_dataZdomain_slow_reqs�domain_url_durationsZdomain_data_keyZdomain_data_value�correlation_value� durationsZ average_duration_calculationZsum_url_slow_reqs� sorted_urlsr r r r! W s` z DecisionMaker.data_processing)r5 r% c s* | j pd� � �fdd�tdt��� �D �S )z� Based on the 'url_slow_reqs' list, a new list will be formed, where the elements of the original list will be iteratively summed by the number of elements equal to 'time' r3 c s g | ]}t �|� | � ��qS r )rL )�.0�i)�timer5 r r � <listcomp>� s z@DecisionMaker.list_handling_considering_time.<locals>.<listcomp>r )r\ �rangerR )r r5 r )r\ r5 r � list_handling_considering_time� s z,DecisionMaker.list_handling_considering_time)�url_slow_reqs_by_timer% c C s x|D ]}|| j krdS qW dS )z_ This functions will check if any of elements is greater than "request_number" TF)�request_number)r r` r[ r r r �$compare_elements_with_request_number� s z2DecisionMaker.compare_elements_with_request_number)r4 rE c C s | j s dS t�t�||��S )z} Calculates the correlation coefficient using the "url_total_reqs" and the "domain_total_reqs" lists r )r0 rJ �amin�corrcoef)r r4 rE r r r rD � s z DecisionMaker.get_correlation)r7 r8 r9 r% c s. | j � fdd�dd� |r&| d|� S | S dS )a Will sort the domain list by "slow_reqs", the goal is to leave only "domains_number" of uppers, also per each domain will sort urls by "reqs_num", the goal is to leave only "urls_number" of uppers. leave_top == 0 allows to keep the full list c s | � S )Nr )�dict_)r9 r r �<lambda>� � z.DecisionMaker.report_sorting.<locals>.<lambda>T)�key�reverseN)�sort)r7 r8 r9 r )r9 r rP � s z DecisionMaker.report_sortingc C s� | j }t|�r�t|��D}yt�|�}W n tjk r@ d}Y nX |�dd��dd�}W dQ R X d|� d�}| �|�}t � ||� dS ) z# Rename old report �unknownr, z dd.mm.yyyy�.�_NZreport__z.json) r+ r �open�json�load�JSONDecodeErrorrC �replacer* r& �rename)r Z old_reportZ json_data�dZdate_from_reportZnew_report_nameZ new_reportr r r �rename_old_report� s z DecisionMaker.rename_old_report)r# c C s: | � � t| jddd��}tj||ddd� W dQ R X dS )z# Makes json report �wzutf-8)�encodingF� )� ensure_ascii�indentN)ru rn r+ ro �dump)r r# �fr r r r" � s z DecisionMaker.add_json_reportc sj t dd�r� fdd�}nd}y(t� j��}tj||d�}W dQ R X W n ttjfk rd � j}Y nX |S )zT Return contents of current report or empty report in case of error T)�skip_jwt_checkc s � fdd�| � � D �S )z� Filter decoded object by fields, given in solo_filtered_options property, e.g. 'correlation' c s i | ]\}}|� j kr||�qS r )r1 )rZ �k�v)r r r � <dictcomp>� s zJDecisionMaker.get_json_report.<locals>._filtering_hook.<locals>.<dictcomp>)�items)�value)r r r �_filtering_hook� s z6DecisionMaker.get_json_report.<locals>._filtering_hookN)� object_hook)r rn r+ ro rp �OSErrorrq r/ )r r� r# Z report_dictr )r r �get_json_report� s z DecisionMaker.get_json_report)rW r% c C s | j s dS || jkS )a� If correlation flag is enabled - we'll compare correlation_coefficient from configuration with calculated correlation coefficient. If the calculated value exceeds the configuration value - we return True otherwise False. At the same time if correlation flag is disabled - we'll also return "True" since in this case the correlation coefficient is not checked and its value is specified as zero in final report. T)r0 �correlation_coefficient)r rW r r r rI � s z$DecisionMaker.correlation_conditionsc C s | � |�}| �|�S )z� At least one element from the received list (url_slow_reqs_by_time) must be greater than request_number )r_ rb )r r5 r` r r r rH � s z%DecisionMaker.request_number_exceeded)r2 r4 r% c C s4 t |�t |� d }| j�d|� || j�dd�kS )zY Check that percent of throttled requests per URL passes given threshold �d z Calculated throttled percent %s� allowed_throttling_percentager )rL r r r rC )r r2 r4 �throttled_percentr r r rB s z$DecisionMaker.is_throttling_suitable)N) �__name__� __module__� __qualname__�__doc__r r$ � staticmethodrG r* �propertyr+ r. r/ �setr1 r r! r= r_ �boolrb rD rN rP ru r" r� rI rH rB � __classcell__r r )r r r " s4 < r �__main__z decision_maker_standalone.log)�filename�level)!r� r ro r r& �os.pathr �numpyrJ �clcommon.lib.cleditionr �ssa.dbr �commonr �storager r r � configurationr �configuration.schemesr �internal.constantsr �internal.utilsr r r r� � basicConfig�INFO�dmr r r r �<module> s, p