Re: [I] OpenSearch - log_id uses logical_date despite log_id_template default with run_id and map_index [airflow]

["devgonvarun (via GitHub)" <gi...@apache.org>]

devgonvarun commented on issue #53926:
URL: https://github.com/apache/airflow/issues/53926#issuecomment-3149749676

   Hi @eladkal and @Owen-CH-Leung  did you get a chance to look into this?
   
   Since the OpensearchTaskHandler doesn't yet support write_to_es like the ElasticsearchTaskHandler, I have to use a log shipper (e.g., Fluentd) to send task logs to OpenSearch. I extract the worker pod logs (kubernetes executor in my case) and construct a log_id from the Kubernetes pod labels. This log_id is used by the OpensearchTaskHandler to find hits in OpenSearch and fetch the correct corresponding logs for display in the airflow UI. Presently the log_id uses logical_date but the Kubernetes pod labels don't include the logical_date anymore, hence this issue. Fortunately, both run_id and map_index are available in the kubernetes pod labels, so if the log_id in OpensearchTaskHandler works as expected using the default log_id_template {dag_id}-{task_id}-{run_id}-{map_index}-{try_number} then everything should work fine. Once the log_id is correctly constructed to match the format expected by the OpensearchTaskHandler, the log retrieval should work.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org