You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/07/11 07:46:15 UTC
[GitHub] [airflow] kuikeelc opened a new issue, #24958: log_filename_template writes different try_number, then when it reads the log
kuikeelc opened a new issue, #24958:
URL: https://github.com/apache/airflow/issues/24958
### Apache Airflow version
2.3.3 (latest released)
### What happened
When you use `remote_logging = True `and push the logs to GCS. It writes the log exactly as defined in `log_filename_template`. However, when reading the log back, it adds 1 and Airflow gives the error that it can't find the logfile.
Our current setting is:
`log_filename_template = dag_id={{ "{{" }} ti.dag_id {{ "}}" }}/run_id={{ "{{" }} ti.run_id {{ "}}" }}/task_id={{ "{{" }} ti.task_id {{ "}}" }}/{%% if ti.map_index >= 0 %%}map_index={{ "{{" }} ti.map_index {{ "}}" }}/{%% endif %%}attempt={{ "{{" }} ti.try_number {{ "}}" }}.log`
Every task in try_number 1 will be read as 2.log, which is not correct.
### What you think should happen instead
_No response_
### How to reproduce
Use v2.3.3 icm with GCS and remote_logging = True
### Operating System
Linux
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kuikeelc commented on issue #24958: log_filename_template writes different try_number, then when it reads the log
Posted by GitBox <gi...@apache.org>.
kuikeelc commented on issue #24958:
URL: https://github.com/apache/airflow/issues/24958#issuecomment-1270454045
@bbovenzi Did you get a chance to have a look at this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kuikeelc commented on issue #24958: log_filename_template writes different try_number, then when it reads the log
Posted by GitBox <gi...@apache.org>.
kuikeelc commented on issue #24958:
URL: https://github.com/apache/airflow/issues/24958#issuecomment-1209284519
Hi,
yes. It is both in the Grid as well as the Graph GUI when I press "Log".
This is understandable, since they both lead to the same page as far as I
can see.
It seems that the incorrect attempt_nr is being sent along. It seems it
tries to retrieve the next attempt (= last attempt + 1) instead of the
actual attempt_number.
Op ma 8 aug. 2022 om 20:39 schreef Brent Bovenzi ***@***.***>:
> @kuikeelc <https://github.com/kuikeelc> is this issue in both the grid
> view and the /log page?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/airflow/issues/24958#issuecomment-1208473795>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AA2FWCDQEOHBMRAXQAU574TVYFH63ANCNFSM53GSBDBA>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bbovenzi commented on issue #24958: log_filename_template writes different try_number, then when it reads the log
Posted by GitBox <gi...@apache.org>.
bbovenzi commented on issue #24958:
URL: https://github.com/apache/airflow/issues/24958#issuecomment-1208473795
@kuikeelc is this issue in both the grid view and the `/log` page?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kuikeelc commented on issue #24958: log_filename_template writes different try_number, then when it reads the log
Posted by GitBox <gi...@apache.org>.
kuikeelc commented on issue #24958:
URL: https://github.com/apache/airflow/issues/24958#issuecomment-1209432251
If it helps. This is the error message I get when requesting the logfile:
*** Unable to read remote log from gs://[HIDDEN
LOCATION]/dag_id=osb__unit_test/run_id=scheduled__2022-08-07T22:15:00+00:00/task_id=from_aws_s3_multiple_csv_no_dedup_test/attempt=2.log
As you can see, it tries to request attempt=2.log. But that file does not
exist. The task ran once successfully and the maximum attempt is therefore
1. So it would work if the attempt number is used when request the log from
Google Cloud Storage
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #24958: log_filename_template writes different try_number, then when it reads the log
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #24958:
URL: https://github.com/apache/airflow/issues/24958#issuecomment-1180073813
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24958: log_filename_template writes different try_number, then when it reads the log
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24958:
URL: https://github.com/apache/airflow/issues/24958#issuecomment-1181719734
@bbovenzi -> isn't that already on your radar ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] log_filename_template writes different try_number, then when it reads the log [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #24958: log_filename_template writes different try_number, then when it reads the log
URL: https://github.com/apache/airflow/issues/24958
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24958: log_filename_template writes different try_number, then when it reads the log
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24958:
URL: https://github.com/apache/airflow/issues/24958#issuecomment-1180228295
@uranusjr @dstandish @jedcunningham -> I think it should be related to recent fixes with log templates?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] log_filename_template writes different try_number, then when it reads the log [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #24958:
URL: https://github.com/apache/airflow/issues/24958#issuecomment-1797922704
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kuikeelc commented on issue #24958: log_filename_template writes different try_number, then when it reads the log
Posted by GitBox <gi...@apache.org>.
kuikeelc commented on issue #24958:
URL: https://github.com/apache/airflow/issues/24958#issuecomment-1181698434
It seems to be a GUI bug. The files are uploaded to Google Cloud Storage with the correct attempt numbers. In the GUI (Task Instance > Log > Select an Attempt, e.g. 1,2,3,etc.) however it always seems to select the max(attempt)+1 for all attempts (1,2,3,4,etc.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] log_filename_template writes different try_number, then when it reads the log [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #24958:
URL: https://github.com/apache/airflow/issues/24958#issuecomment-1751628330
This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org