You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/02/19 06:45:34 UTC
[GitHub] [airflow] heeroyuy925 opened a new issue #14312: Remote log on azure blob storage display issue on microsoft provider airflow/providers/microsoft/azure/log/wasb_task_handler.py
heeroyuy925 opened a new issue #14312:
URL: https://github.com/apache/airflow/issues/14312
<!--
IMPORTANT!!!
PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!
PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!
Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the context.
-->
**Apache Airflow version**: 2.1.0.dev0
**Environment**: docker-compose
**pip package I installed**:
azure-storage-blob 12.7.1
**What happened**:
In my spike environment, we are testing to store remote log to Azure blob storage.
When we triggered the example dags, we find the log is displayed in one line which is hard to read.
For example:
*** Reading remote log from platform-boundary-airflowlogs/example_bash_operator/also_run_this/2021-02-07T03:42:07.725535+00:00/1.log.
b'[2021-02-07 03:42:14,900] {taskinstance.py:1739} DEBUG - Updating task params ({\'example_key\': \'example_value\'}) with DagRun.conf ({\'example_key\': \'example_value\'})\n[2021-02-07 03:42:14,901] {executor_loader.py:82} DEBUG - Loading core executor: LocalExecutor\n[2021-02-07 03:42:14,936] {__init__.py:51} DEBUG - Loading core task runner: StandardTaskRunner\n[2021-02-07 03:42:14,957] {taskinstance.py:1739} DEBUG -----------------------'
<!-- (please include exact error messages if you can) -->
**What you expected to happen**:
The log on UI can be displayed as follow:
Reading remote log from platform-boundary-airflowlogs/example_bash_operator/runme_0/2021-02-18T09:07:31.667381+00:00/1.log.
[2021-02-18 09:07:33,878] {taskinstance.py:851} INFO - Dependencies all met for <TaskInstance: example_bash_operator.runme_0 2021-02-18T09:07:31.667381+00:00 [queued]>
[2021-02-18 09:07:33,950] {taskinstance.py:851} INFO - Dependencies all met for <TaskInstance: example_bash_operator.runme_0 2021-02-18T09:07:31.667381+00:00 [queued]>
[2021-02-18 09:07:33,954] {taskinstance.py:1042} INFO -
[2021-02-18 09:07:33,956] {taskinstance.py:1043} INFO - Starting attempt 1 of 1
[2021-02-18 09:07:33,958] {taskinstance.py:1044} INFO -
[2021-02-18 09:07:33,994] {taskinstance.py:1063} INFO - Executing <Task(BashOperator): runme_0> on 2021-02-
--- other logs ---
<!-- What do you think went wrong? -->
In the wasb_task_handler.py, the remote log is tried to be read as string. But the log is too long so python read it as bytes which will be showed started with " b' " on UI.
**How to reproduce it**:
1.Set your log to the remote log store to Azure blob storage
2.Trigger the example dags
3.Check the log of example dags job on Web UI
<!---
As minimally and precisely as possible. Keep in mind we do not have access to your cluster or dags.
If you are using kubernetes, please attempt to recreate the issue using minikube or kind.
## Install minikube/kind
- Minikube https://minikube.sigs.k8s.io/docs/start/
- Kind https://kind.sigs.k8s.io/docs/user/quick-start/
If this is a UI bug, please provide a screenshot of the bug or a link to a youtube video of the bug in action
You can include images using the .md style of
To record a screencast, mac users can use QuickTime and then create an unlisted youtube video with the resulting .mov file.
--->
**Anything else we need to know**:
I am trying to fix airflow/providers/microsoft/azure/log/wasb_task_handler.py by decode the remote string.
<!--
How often does this problem occur? Once? Every time etc?
Any relevant logs to include? Put them here in side a detail tag:
<details><summary>x.log</summary> lots of stuff </details>
-->
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #14312: Remote log on azure blob storage display issue on microsoft provider airflow/providers/microsoft/azure/log/wasb_task_handler.py
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #14312:
URL: https://github.com/apache/airflow/issues/14312#issuecomment-786272265
Closed via https://github.com/apache/airflow/pull/14313
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #14312: Remote log on azure blob storage display issue on microsoft provider airflow/providers/microsoft/azure/log/wasb_task_handler.py
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #14312:
URL: https://github.com/apache/airflow/issues/14312#issuecomment-786292924
Yes. You'll have to wait for new provider release
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy closed issue #14312: Remote log on azure blob storage display issue on microsoft provider airflow/providers/microsoft/azure/log/wasb_task_handler.py
Posted by GitBox <gi...@apache.org>.
ephraimbuddy closed issue #14312:
URL: https://github.com/apache/airflow/issues/14312
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ldacey commented on issue #14312: Remote log on azure blob storage display issue on microsoft provider airflow/providers/microsoft/azure/log/wasb_task_handler.py
Posted by GitBox <gi...@apache.org>.
ldacey commented on issue #14312:
URL: https://github.com/apache/airflow/issues/14312#issuecomment-782889314
Ah, good there is any issue already. Yep, my airflow logs are displayed as a single line which makes them unreadable on the webserver. If I download the log from the UI, or if I open the log directly in the blob container then the formatting is fine.
`b"[2021-02-21 16:16:03,590]`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #14312: Remote log on azure blob storage display issue on microsoft provider airflow/providers/microsoft/azure/log/wasb_task_handler.py
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #14312:
URL: https://github.com/apache/airflow/issues/14312#issuecomment-781868138
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ldacey commented on issue #14312: Remote log on azure blob storage display issue on microsoft provider airflow/providers/microsoft/azure/log/wasb_task_handler.py
Posted by GitBox <gi...@apache.org>.
ldacey commented on issue #14312:
URL: https://github.com/apache/airflow/issues/14312#issuecomment-786289707
Nice, thanks. For future reference with changes that happen in provider packages, would I wait for pypi to have this change reflected here? https://pypi.org/project/apache-airflow-providers-microsoft-azure/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] flolas commented on issue #14312: Remote log on azure blob storage display issue on microsoft provider airflow/providers/microsoft/azure/log/wasb_task_handler.py
Posted by GitBox <gi...@apache.org>.
flolas commented on issue #14312:
URL: https://github.com/apache/airflow/issues/14312#issuecomment-782869187
We are facing same issue... after upgrading airflow 2.0.0 to 2.0.1 and providers 1.0.0 to 1.0.1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] flolas edited a comment on issue #14312: Remote log on azure blob storage display issue on microsoft provider airflow/providers/microsoft/azure/log/wasb_task_handler.py
Posted by GitBox <gi...@apache.org>.
flolas edited a comment on issue #14312:
URL: https://github.com/apache/airflow/issues/14312#issuecomment-782869187
We are facing same issue... after upgrading airflow 2.0.0 to 2.0.1 and providers 1.0.0 to 1.0.1. Maybe happened after #12188
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org