You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/04/01 20:56:09 UTC
[GitHub] [airflow] codenamestif commented on issue #15144: Support of Timestamp for Cloudwatch-based remote logging
codenamestif commented on issue #15144:
URL: https://github.com/apache/airflow/issues/15144#issuecomment-812167706
Here is an example of the current behaviour:
```
*** Reading remote log from Cloudwatch log_group: my_log_group log_stream: item_sub_category_2_demand_dag_v1/item_sub_category_2_demand/2021-04-01T15_42_13.963961+00_00/1.log.
Dependencies all met for <TaskInstance: item_sub_category_2_demand_dag_v1.item_sub_category_2_demand 2021-04-01T15:42:13.963961+00:00 [queued]>
Dependencies all met for <TaskInstance: item_sub_category_2_demand_dag_v1.item_sub_category_2_demand 2021-04-01T15:42:13.963961+00:00 [queued]>
--------------------------------------------------------------------------------
Starting attempt 1 of 4
--------------------------------------------------------------------------------
Executing <Task(ECSOperator): item_sub_category_2_demand> on 2021-04-01T15:42:13.963961+00:00
Started process 89 to run task
Running <TaskInstance: item_sub_category_2_demand_dag_v1.item_sub_category_2_demand 2021-04-01T15:42:13.963961+00:00 [running]> on host xxx-xxx
Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=item_sub_category_2_demand_dag_v1
AIRFLOW_CTX_TASK_ID=item_sub_category_2_demand
AIRFLOW_CTX_EXECUTION_DATE=2021-04-01T15:42:13.963961+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-04-01T15:42:13.963961+00:00
```
After I adjusted the handler I have got the next one:
```
*** Reading remote log from Cloudwatch log_group: my_log_group log_stream: item_sub_category_2_demand_dag_v1/item_sub_category_2_demand/2021-04-01T15_42_13.963961+00_00/1.log.
[2021-04-01T17:42:15.351000+02:00] - Dependencies all met for <TaskInstance: item_sub_category_2_demand_dag_v1.item_sub_category_2_demand 2021-04-01T15:42:13.963961+00:00 [queued]>
[2021-04-01T17:42:15.411000+02:00] - Dependencies all met for <TaskInstance: item_sub_category_2_demand_dag_v1.item_sub_category_2_demand 2021-04-01T15:42:13.963961+00:00 [queued]>
[2021-04-01T17:42:15.411000+02:00] -
--------------------------------------------------------------------------------
[2021-04-01T17:42:15.411000+02:00] - Starting attempt 1 of 4
[2021-04-01T17:42:15.411000+02:00] -
--------------------------------------------------------------------------------
[2021-04-01T17:42:15.432000+02:00] - Executing <Task(ECSOperator): item_sub_category_2_demand> on 2021-04-01T15:42:13.963961+00:00
[2021-04-01T17:42:15.435000+02:00] - Started process 89 to run task
[2021-04-01T17:42:15.613000+02:00] - Running <TaskInstance: item_sub_category_2_demand_dag_v1.item_sub_category_2_demand 2021-04-01T15:42:13.963961+00:00 [running]> on host xxx-xxx
[2021-04-01T17:42:15.718000+02:00] - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=item_sub_category_2_demand_dag_v1
AIRFLOW_CTX_TASK_ID=item_sub_category_2_demand
AIRFLOW_CTX_EXECUTION_DATE=2021-04-01T15:42:13.963961+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-04-01T15:42:13.963961+00:00
```
I also made a formatting of the timestamp based on UI timezone. If timezone formatting is a good idea, then there is also an issue with `ECSOperator`, that fetches the logs from the Cloudwatch and prints them in UTC. But this is another thing that can be improved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org