Edit File: stat_sender.cpython-37.pyc
B V�d� � @ sd d Z dZddlZddlZddlZddlmZ ddlmZ ddl m Z m Z ddl m Z ddlmZ dd lmZmZ dd lmZ dd lmZ dd lmZ d dlmZ d dlm Z ddl m Z ddl m Z ddl!m"Z"m#Z# ddl$m%Z%m&Z&m'Z'm(Z( G dd� de�Z)e*dk�r`e&� ej+dej,d� ye)� �-� W n4 e k �r^ Z. ze/e.� e0d ��W ddZ.[.X Y nX dS )zd This module contains SSA classes for sending e-mails with report and sending reports to ClickHouse z ssa.modules� N)� EmailMessage)�partial)�Optional�Any)�get_admin_email)�is_cl_solo_edition)�Session�Response)� HTTPAdapter)�RequestException)�Retry� )�Common)� DecisionMaker� )� stat_server)�SSAError)�Mailer�render_report_table)� read_sys_id� sentry_init� duration_cast� format_datec s� e Zd ZdZ� fdd�Zdedd�dd�Zeee d�d d �Z ee d �d d �Z ee d �dd�Z ee d�dd�Zeeed�dd��Zeed�dd�Zee d�dd�Z� ZS ) �StatisticsSenderz4 Send report to ClickHouse and over e-mails c s� t � �� t�d�| _| j�dt� t� | _dt � d�| _ t � | _ t dtdg�tddd g�dd �}t|d �}t� | _| j�d|� t| jjd d �| j_d S )N� stat_senderz StatisticsSender enabled: %szhttps://z /api/clos-ssa� �POSTi� i� i� )�total�method_whitelist�status_forcelist�backoff_factor)� max_retries� )�timeout)�super�__init__�logging� getLogger�logger�info� __package__r �sys_idr � ch_endpointr � mail_senderr � frozensetr r �session�mountr �request)�selfZ retry_conf�adapter)� __class__� �H/opt/alt/python37/lib/python3.7/site-packages/ssa/modules/stat_sender.pyr% * s zStatisticsSender.__init__N)�report�returnc C s0 |dkrt � �� }| jr"| �|� | �|� dS )zi Send given report over e-mail If no report given, get it from DecisionMaker API N)r �get_json_report� summary_notification_enabled� email_report�clickhouse_report)r2 r7 r5 r5 r6 �send; s zStatisticsSender.send)� report_viewr8 c C s� | � � }|r�|�d�r�t|�\}}| jj|dt|d �|d�}|jt�|�dd|d � d�d� |j|d d|d � d �d� | j� |� |S d S ) z4 Create and send e-mail with report �domainsZ ssa_report�date)Z recipient�templater@ �html�jsonZreport_z.json)�subtype�filenamerB z.htmlN) �get_mail_recipient�getr r- �_messager �add_attachmentrC �dumpsZ_send)r2 r> Zmail_toZ report_table�mail�msgr5 r5 r6 r; G s z StatisticsSender.email_report)r8 c C sH y| � � pt� S tk rB } z| j�dt|�� W dd}~X Y nX dS )z� Retrieve a recipient's e-mail: 1. get address from a wmt-api utility 2. if command failed or address is empty, get address of server admin z get_admin_email failed with: %sN)�wmt_api_report_emailr � Exceptionr( �error�str)r2 �er5 r5 r6 rF ` s z#StatisticsSender.get_mail_recipientc C s� t dd�rdnd}y&tjd|� �dgdddd�j�� }W n> tjtttfk rv } z| j � dt |�� d S d }~X Y nX yt � |��d ��d �S t jk r� } z| j � d t |�� W d d }~X Y n" tk r� | j � d |� Y nX d S )zD Retrieve a recipient's e-mail address from WMT API T)�skip_jwt_checkz wmt-api-solozwmt-apiz&/usr/share/web-monitoring-tool/wmtbin/z --config-get)�check�text�capture_outputzwmt-api utility failed: %sN�configZ report_emailz!wmt-api returned invalid json: %sz(wmt-api returned unexpected response: %s)r � subprocess�run�stdout�strip�CalledProcessError�AttributeError�OSError� ValueErrorr( rO rP rC �loadsrG �JSONDecodeError)r2 �_utilZ api_responserQ r5 r5 r6 rM l s( z%StatisticsSender.wmt_api_report_emailc C s� |� d�r�| j�d| j� y"| jj| j| �| �|��d�}W nN tk r� } z0| jj d|d| jid� t dt |�� ��|�W dd}~X Y nX | � |�S | j�d � d S dS ) z+ Send report to ClickHouse r? zSending POST request to %s)rC zPOST failed with %s�endpoint)�extraz'Failed to POST data to SSA API server: Nz*Report is empty, not sending to ClickHouseF) rG r( r) r, r/ �post�_ch_pack�clickhouse_formatr rO r rP �_process_response)r2 r> �resprQ r5 r5 r6 r< � s $ z"StatisticsSender.clickhouse_report)�valuer8 c C s t | d�S )z< Pack given value into data field of a dict )�data)�dict)ri r5 r5 r6 re � s zStatisticsSender._ch_pack)�original_reportr8 c C s` t � }xT|�d�D ]F}|�| j|�d�|�d�|�d�|�d�dd� |�d�D �d �� qW |S ) zd Format local report for sending to ClickHouse (required structures differ) r? �name� slow_urls� slow_reqs� total_reqsc S s< g | ]4}|� d �|� d�t|� d��t|� dd��d��qS )rm �reqs_num�average_duration� correlationr )�urlZcount_requests� avg_durationrs )rG r �float)�.0�ur5 r5 r6 � <listcomp>� s z6StatisticsSender.clickhouse_format.<locals>.<listcomp>�urls)� system_id�domainZcount_slow_urlsZcount_slow_requestsZtotal_requests�details)�listrG �appendr+ )r2 rl Z ch_reportr| r5 r5 r6 rf � s z"StatisticsSender.clickhouse_format)�responser8 c C s� |j s(| jjd|j|jd|jid� dS | j�d|j|j|j� |�� }|d dkrr| jjd|d d |id� dS | j�d � d S ) z� Check received response :param response: a requests.Response object :return: True in case of success, False otherwise z&Unable to connect to server with %s:%sZ resp_text)rc Fz [%s:%s] Response received %s�status�okz Received response with status %sr� z Sent to ClickHouse successfullyT) r� r( rO � status_code�reasonrT r) rt rC )r2 r� �resultr5 r5 r6 rg � s z"StatisticsSender._process_response)N)�__name__� __module__� __qualname__�__doc__r% rk r= r r r; rP rF rM �boolr< � staticmethodr re r~ rf r rg � __classcell__r5 r5 )r4 r6 r % s r �__main__zstat_sender_standalone.log)rE �level)1r� r* rC r& rW � email.messager � functoolsr �typingr r �clcommon.cpapir �clcommon.lib.cleditionr �requestsr r Zrequests.adaptersr Zrequests.exceptionsr Z$requests.packages.urllib3.util.retryr �commonr �decision_makerr �internal.constantsr �internal.exceptionsr Zinternal.mailerr r �internal.utilsr r r r r r� � basicConfig�INFOr= �exc�print� SystemExitr5 r5 r5 r6 �<module> s< 0