[jira] [Commented] (AIRFLOW-7110) Refactor the way logs are written to/read from Elasticsearch

["Patrick Gaudreau (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-7110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17080023#comment-17080023 ] 

Patrick Gaudreau commented on AIRFLOW-7110:
-------------------------------------------

We ran into a similar issue as well when using Airflow deployed on an Azure AKS with the following configuration:
{noformat}
    AIRFLOW__CORE__REMOTE_LOGGING: true    
    AIRFLOW__ELASTICSEARCH__WRITE_STDOUT: true
{noformat}
Our logs are properly shipped to elasticsearch (because we write to stdout and are using bkpr that sends everything from stdout to elasticsearch) but the Airflow UI can't get the logs back because there is no 'log_id' field created.

 

One possible solution to our problem would be if the 'log_id' field name was configurable.  We could then add an extra kubernetes label called 'kubernetes.labels.log_id' to every log using for example a Pod Mutation Hook.

> Refactor the way logs are written to/read from Elasticsearch
> ------------------------------------------------------------
>
>                 Key: AIRFLOW-7110
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-7110
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: configuration, logging, operators, ui
>    Affects Versions: 1.10.9
>            Reporter: Noël BARDELOT
>            Priority: Major
>
> See:
> * in the configuration,
>     - under section 'core' option named 'remote_logging'
>     - under section 'elasticsearch' options named 'host', 'log_id_template', 'end_of_log_mark', 'frontend', 'write_stdout' and 'json_format'
> * in the documentation,
>   - https://airflow.apache.org/docs/stable/howto/write-logs.html
> * in the Airflow code,
>   - BaseOperator in airflow/models/baseoperator.py
>   - ElasticsearchTaskHandler in airflow/utils/log/es_task_handler.py
>   - the detail of a task's view in airflow/www/views.py
> Current behaviour:
> If one sets the options as stated in the documentation in order to write logs from the task to the worker stdout logs in JSON format, and upload those logs in Elasticsearch (using ELK or an equivalent stack), the field 'log_id' needed by the handler and the view to work is not written by default to the logs.
> One needs to adapt the log collection from the worker's container stdout logs in order to provide this 'log_id' field, which agregates other usual fields (see 'log_id_template' which is used by the handler to request logs from Elasticsearch to be shown in the view).
> An example for fluentd:
> {code}
>     # Also add log_id to airflow.task logs
>     <filter airflow.task.**>
>       @type record_transformer
>       enable_ruby
>       <record>
>         log_id ${record["dag_id"]}_${record["task_id"]}_${record["execution_date"]}_${record["try_number"]}
>         offset ${time = Time.now; time.to_i * (10 ** 9) + time.nsec}
>       </record>
>     </filter>
> {code}
> What we expect:
> It should be the job of the Operator to log the needed 'log_id' field (Airflow reads it and needs it, so Airflow should write it itself...). That should be done in the BaseOperator, so that it's available to all inheriting subclasses. This field could be added only if the 'remote_logging' is set to True for example in order to avoid writing more logs than necessary.
> Plus, there is no benefit to make the 'log_id_template' configurable. As is, there is no way to make anything else than the current value work, and nobody should ever modify it (if someone modified that value, the view wouldn't work anyway).
> Plus, the 'frontend' configuration option should not presume that 'https://' is the protoocol to use in order to make a link to the external log viewer (Kibana or other...). The 'frontend' option should contain the whole URL templated by 'log_id'.
> Plus, all this behaviour should be far more documented in the 'write-logs' documentation.
> Plus, a DAG example should be added to Airflow's already existing example, in order to show how a subclass of BaseOperator should use the logging mechanism to write a log.
> Plus, Elasticsearch's 'host' option under Airflow's configuration is wrongly named and documented. It is not a host, but a server with host and port.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)