You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/04/01 19:39:58 UTC
[GitHub] [airflow] codenamestif opened a new issue #15144: Support of Timestamp for Cloudwatch-based remote logging
codenamestif opened a new issue #15144:
URL: https://github.com/apache/airflow/issues/15144
**Description**
`CloudwatchTaskHandler` doesn't parse log event timestamp. As result if Airflow uses `Cloudwatch` for remote logging, logs in Airflow UI miss timestamp. They just contain concatenated messages.
**Use case / motivation**
It's more convenient to have a timestamp next to the log message to better understand what and when happened.
**Are you willing to submit a PR?**
I can do that, but that would be my first pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #15144: Support of Timestamp for Cloudwatch-based remote logging
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #15144:
URL: https://github.com/apache/airflow/issues/15144#issuecomment-841106690
fixed by https://github.com/apache/airflow/pull/15173
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jhtimmins commented on issue #15144: Support of Timestamp for Cloudwatch-based remote logging
Posted by GitBox <gi...@apache.org>.
jhtimmins commented on issue #15144:
URL: https://github.com/apache/airflow/issues/15144#issuecomment-812141160
@codenamestif could you include log output and/or images of what you see?
If you're interested in taking on this PR, it sounds like a solid first one. I'm happy to answer any questions you have about getting started.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jhtimmins commented on issue #15144: Support of Timestamp for Cloudwatch-based remote logging
Posted by GitBox <gi...@apache.org>.
jhtimmins commented on issue #15144:
URL: https://github.com/apache/airflow/issues/15144#issuecomment-812258584
@codenamestif Thanks for including the output.
Would you like to submit a PR for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] codenamestif commented on issue #15144: Support of Timestamp for Cloudwatch-based remote logging
Posted by GitBox <gi...@apache.org>.
codenamestif commented on issue #15144:
URL: https://github.com/apache/airflow/issues/15144#issuecomment-812167706
Here is an example of the current behaviour:
```
*** Reading remote log from Cloudwatch log_group: my_log_group log_stream: item_sub_category_2_demand_dag_v1/item_sub_category_2_demand/2021-04-01T15_42_13.963961+00_00/1.log.
Dependencies all met for <TaskInstance: item_sub_category_2_demand_dag_v1.item_sub_category_2_demand 2021-04-01T15:42:13.963961+00:00 [queued]>
Dependencies all met for <TaskInstance: item_sub_category_2_demand_dag_v1.item_sub_category_2_demand 2021-04-01T15:42:13.963961+00:00 [queued]>
--------------------------------------------------------------------------------
Starting attempt 1 of 4
--------------------------------------------------------------------------------
Executing <Task(ECSOperator): item_sub_category_2_demand> on 2021-04-01T15:42:13.963961+00:00
Started process 89 to run task
Running <TaskInstance: item_sub_category_2_demand_dag_v1.item_sub_category_2_demand 2021-04-01T15:42:13.963961+00:00 [running]> on host xxx-xxx
Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=item_sub_category_2_demand_dag_v1
AIRFLOW_CTX_TASK_ID=item_sub_category_2_demand
AIRFLOW_CTX_EXECUTION_DATE=2021-04-01T15:42:13.963961+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-04-01T15:42:13.963961+00:00
```
After I adjusted the handler I have got the next one:
```
*** Reading remote log from Cloudwatch log_group: my_log_group log_stream: item_sub_category_2_demand_dag_v1/item_sub_category_2_demand/2021-04-01T15_42_13.963961+00_00/1.log.
[2021-04-01T17:42:15.351000+02:00] - Dependencies all met for <TaskInstance: item_sub_category_2_demand_dag_v1.item_sub_category_2_demand 2021-04-01T15:42:13.963961+00:00 [queued]>
[2021-04-01T17:42:15.411000+02:00] - Dependencies all met for <TaskInstance: item_sub_category_2_demand_dag_v1.item_sub_category_2_demand 2021-04-01T15:42:13.963961+00:00 [queued]>
[2021-04-01T17:42:15.411000+02:00] -
--------------------------------------------------------------------------------
[2021-04-01T17:42:15.411000+02:00] - Starting attempt 1 of 4
[2021-04-01T17:42:15.411000+02:00] -
--------------------------------------------------------------------------------
[2021-04-01T17:42:15.432000+02:00] - Executing <Task(ECSOperator): item_sub_category_2_demand> on 2021-04-01T15:42:13.963961+00:00
[2021-04-01T17:42:15.435000+02:00] - Started process 89 to run task
[2021-04-01T17:42:15.613000+02:00] - Running <TaskInstance: item_sub_category_2_demand_dag_v1.item_sub_category_2_demand 2021-04-01T15:42:13.963961+00:00 [running]> on host xxx-xxx
[2021-04-01T17:42:15.718000+02:00] - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=item_sub_category_2_demand_dag_v1
AIRFLOW_CTX_TASK_ID=item_sub_category_2_demand
AIRFLOW_CTX_EXECUTION_DATE=2021-04-01T15:42:13.963961+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-04-01T15:42:13.963961+00:00
```
I also made a formatting of the timestamp based on UI timezone. If timezone formatting is a good idea, then there is also an issue with `ECSOperator`, that fetches the logs from the Cloudwatch and prints them in UTC. But this is another thing that can be improved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #15144: Support of Timestamp for Cloudwatch-based remote logging
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #15144:
URL: https://github.com/apache/airflow/issues/15144#issuecomment-812128682
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal closed issue #15144: Support of Timestamp for Cloudwatch-based remote logging
Posted by GitBox <gi...@apache.org>.
eladkal closed issue #15144:
URL: https://github.com/apache/airflow/issues/15144
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] codenamestif commented on issue #15144: Support of Timestamp for Cloudwatch-based remote logging
Posted by GitBox <gi...@apache.org>.
codenamestif commented on issue #15144:
URL: https://github.com/apache/airflow/issues/15144#issuecomment-812843561
@jhtimmins i prepared a draft pr since i have a question about implementation details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] codenamestif commented on issue #15144: Support of Timestamp for Cloudwatch-based remote logging
Posted by GitBox <gi...@apache.org>.
codenamestif commented on issue #15144:
URL: https://github.com/apache/airflow/issues/15144#issuecomment-812369475
@jhtimmins yes, i will prepare a pr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org