You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ldacey (via GitHub)" <gi...@apache.org> on 2023/09/01 17:58:56 UTC

[GitHub] [airflow] ldacey opened a new issue, #34022: Unable to read or write logs on Azure Blob after 2.7.0 update

ldacey opened a new issue, #34022:
URL: https://github.com/apache/airflow/issues/34022

   ### Apache Airflow version
   
   2.7.0
   
   ### What happened
   
   After updating to 2.7.0 my worker is unable to read or write logs to Azure Blob. My previous configuration has not been changed since May 2022, so there must have been a recent change which broke it.
   
   My container is "airflow-logs" and then I save production logs in "wasb-airflow-logs" folder/prefix. There are some other 'folders' for different environments.
   
   docker stack variables:
   ```python
   AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS: log_config.LOGGING_CONFIG
   AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: wasb-airflow-logs
   ```
   
   log_config.py
   ```python
   from copy import deepcopy
   from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG
   
   REMOTE_BASE_LOG_FOLDER = "wasb://airflow-logs@account.blob.core.windows.net"
   LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)
   ```
   
   After 2.7.0 upgrade I noticed that my worker was spamming the following error message. It seems like Airflow suddenly things "wasb-airflow-logs" is the container instead of the prefix? 
   
   ```
   [2023-09-01 14:39:29,785: ERROR/ForkPoolWorker-11] Could not write logs to wasb-airflow-logs/dag_id=
   azure.core.exceptions.ResourceNotFoundError: The specified container does not exist.
   ```
   
   And my tasks all started with this error:
   
   ```
   *** tried listing blobs with prefix=wasb-airflow-logs/dag_id=...attempt=1.log and container=airflow-logs
   *** could not list blobs The requested URI does not represent any resource on the server.
   ```
   
   For now, I had to revert my changes to use `AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER` instead which allows the worker to run at least, but it cannot read/write logs.
   
   
   ### What you think should happen instead
   
   _No response_
   
   ### How to reproduce
   
   Any change I made to the REMOTE_BASE_LOG_FOLDER did not work and resulted in an error:
   
   ```
   ImportError: Unable to load custom logging from log_config.LOGGING_CONFIG due to Incorrect remote log configuration. Please check the configuration of option 'host' in section 'elasticsearch' if you are using Elasticsearch. In the other case, 'remote_base_log_folder' option in the 'logging' section.
   ```
   
   Here are the things I tried:
   
   ```python
   REMOTE_BASE_LOG_FOLDER = "wasb-airflow-logs"
   REMOTE_BASE_LOG_FOLDER = "wasb://airflow-logs@account.blob.core.windows.net"
   REMOTE_BASE_LOG_FOLDER = "wasb://airflow-logs@account.blob.core.windows.net/wasb-airflow-logs/"
   LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)
   ```
   
   ### Operating System
   
   Ubuntu 22.04
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Other Docker-based deployment
   
   ### Deployment details
   
   docker stack deploy
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ephraimbuddy commented on issue #34022: Unable to read or write logs on Azure Blob after 2.7.0 update

Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #34022:
URL: https://github.com/apache/airflow/issues/34022#issuecomment-1705490342

   @ldacey, I would like to know what your configurations were before the upgrade. 
   1. What was your `apache-airflow-providers-microsoft-azure` version previously?
   2. What version were you upgrading to
   3. How did you provide your connection previously? was it through env var or you entered it in the UI? If with environment variable, what was the format e.g was it `wasb://username:accountkey@account....`. 
   
   I doubt if you were able to specify the remote_base_log_folder previously as `wasb://airflow-logs@account.blob.core.windows.net` as you mentioned above. That's almost a full connection uri which I think should not be what remote_base_log_folder should be(for azure) but I'm interested to learn how you made your connection previously. 
   
   There's a new version of the provider: 6.3.0, maybe you should try that too.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] pankajkoti commented on issue #34022: Unable to read or write logs on Azure Blob after 2.7.0 update

Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on issue #34022:
URL: https://github.com/apache/airflow/issues/34022#issuecomment-1703261263

   hi @ldacey 
   
   Can you check what is the version of apache-airflow-providers-microsoft-azure in your deployment. I don't think it would be directly related to a change in Airflow 2.7.0, but could rather be with the provider.
   
   A new version of the provider was just released today https://pypi.org/project/apache-airflow-providers-microsoft-azure/6.3.0/ 
   
   Can you try installing 6.3.0 version of the provider and see if the issue persists?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] pankajkoti commented on issue #34022: Unable to read or write logs on Azure Blob after 2.7.0 update

Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on issue #34022:
URL: https://github.com/apache/airflow/issues/34022#issuecomment-1704004495

   you can keep the constraints as is and just install the newer azure provider at the end of your image build process


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ldacey closed issue #34022: Unable to read or write logs on Azure Blob after 2.7.0 update

Posted by "ldacey (via GitHub)" <gi...@apache.org>.
ldacey closed issue #34022: Unable to read or write logs on Azure Blob after 2.7.0 update
URL: https://github.com/apache/airflow/issues/34022


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #34022: Unable to read or write logs on Azure Blob after 2.7.0 update

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #34022:
URL: https://github.com/apache/airflow/issues/34022#issuecomment-1704024639

   > Weird thing is that the scheduler died immediately (must be due to some other package since I remove constraints completely for this build). Ill try to revert back and add the 6.3.0 version
   
   This is a known Celery bug that has been introduced few days and they already deleted 5.3.2 and 5.3.3 releases that cause the problem - we've reported it to celery team and added protection for it for 2.7.1 but also celery team removed the buggy versions.
   
   As @pankajkoti explained: the **RIGHT** (and intended) way of installing different version of packages than that in the constraints is to install airflow first with constraints and THEN separately adding/removing/updating packages without the constraints. 
   
   See this part of our documentation that explains it and gives examples: https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#upgrading-and-installing-dependencies-including-providers
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ldacey commented on issue #34022: Unable to read or write logs on Azure Blob after 2.7.0 update

Posted by "ldacey (via GitHub)" <gi...@apache.org>.
ldacey commented on issue #34022:
URL: https://github.com/apache/airflow/issues/34022#issuecomment-1703398855

   This is the version which my image was using:
   
   Downloading apache_airflow_providers_microsoft_azure-6.2.4-py3-none-any.whl.metadata (7.6 kB)
   
   I removed constraints to try to rebuild and the latest version was added:
   Downloading apache_airflow_providers_microsoft_azure-6.3.0-py3-none-any.whl.metadata 
   
   Weird thing is that the scheduler died immediately (must be due to some other package since I remove constraints completely for this build). Ill try to revert back and add the 6.3.0 version 
   
   ```
   sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column celery_taskmeta.children does not exist
   LINE 1: ...ry_taskmeta.date_done, celery_taskmeta.traceback, celery_tas...
                                                                ^
   [SQL: SELECT celery_taskmeta.id, celery_taskmeta.task_id, celery_taskmeta.status, celery_taskmeta.result, celery_taskmeta.date_done, celery_taskmeta.traceback, celery_taskmeta.children 
   FROM celery_taskmeta 
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ldacey commented on issue #34022: Unable to read or write logs on Azure Blob after 2.7.0 update

Posted by "ldacey (via GitHub)" <gi...@apache.org>.
ldacey commented on issue #34022:
URL: https://github.com/apache/airflow/issues/34022#issuecomment-1705454732

   After pinning celery, I have no issues with the workers. I still cannot read or write logs on Azure though for some reason.
   
   This prefix and container are correct. After my update to 2.7.0 there are no remote logs though. Logs from task (reading remote logs):
   > *** tried listing blobs with prefix=wasb-airflow-logs/dag_id=zendesk/run_id=scheduled__2023-09-04T06:51:00+00:00/task_id=support.transform.custom-fields/attempt=1.log and container=airflow-logs
   > *** could not list blobs The requested URI does not represent any resource on the server.
   
   Logs from worker are below. Is the worker trying to write to a container called "wasb-airflow-logs" instead of "airflow-logs"? I am not sure why this is different from the logs being read which is correct (container is "airflow-logs" and prefix is "wasb-airflow-logs". As mentioned, this configuration has worked for a long time with no changes.
   
   > [2023-09-04 15:26:26,796: ERROR/ForkPoolWorker-3] Could not write logs to wasb-airflow-logs/dag_id=example/run_id=scheduled__2023-09-01T14:10:00+00:00/task_id=offered.extract/attempt=2.log
   > Traceback (most recent call last):
   > azure.core.exceptions.ResourceNotFoundError: The specified container does not 
   
   
   
   log_config.py (I tested a few REMOTE_BASE_LOG_FOLDER values in this file as well, as you can see from my original issue).
   ```python
   from copy import deepcopy
   from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG
   
   LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)
   ```
   
   env vars:
   ```python
     AIRFLOW__LOGGING__REMOTE_LOGGING: "true"
     AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: azure_blob
     AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: wasb-airflow-logs
     ```
     
     
   
   pip freeze results:
   ```
   apache-airflow==2.7.0
   apache-airflow-providers-celery==3.3.3
   apache-airflow-providers-common-sql==1.7.1
   apache-airflow-providers-docker==3.7.4
   apache-airflow-providers-ftp==3.5.1
   apache-airflow-providers-google==10.7.0
   apache-airflow-providers-http==4.5.1
   apache-airflow-providers-imap==3.3.1
   apache-airflow-providers-microsoft-azure==6.3.0
   apache-airflow-providers-mysql==5.3.0
   apache-airflow-providers-odbc==4.0.0
   apache-airflow-providers-postgres==5.6.0
   apache-airflow-providers-redis==3.3.1
   apache-airflow-providers-salesforce==5.4.2
   apache-airflow-providers-sftp==4.6.0
   apache-airflow-providers-sqlite==3.4.3
   apache-airflow-providers-ssh==3.7.2
   
   adal==1.2.7
   adlfs==2023.8.0
   azure-batch==14.0.0
   azure-common==1.1.28
   azure-core==1.29.3
   azure-cosmos==4.5.0
   azure-datalake-store==0.0.53
   azure-identity==1.14.0
   azure-keyvault-secrets==4.7.0
   azure-kusto-data==4.2.0
   azure-mgmt-containerinstance==8.0.0
   azure-mgmt-core==1.4.0
   azure-mgmt-cosmosdb==9.2.0
   azure-mgmt-datafactory==1.1.0
   azure-mgmt-datalake-nspkg==3.0.1
   azure-mgmt-datalake-store==0.5.0
   azure-mgmt-nspkg==3.0.2
   azure-mgmt-resource==23.0.1
   azure-nspkg==3.0.2
   azure-servicebus==7.11.1
   azure-storage-blob==12.17.0
   azure-storage-common==2.1.0
   azure-storage-file==2.1.0
   azure-storage-file-datalake==12.12.0
   azure-synapse-spark==0.7.0
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ldacey commented on issue #34022: Unable to read or write logs on Azure Blob after 2.7.0 update

Posted by "ldacey (via GitHub)" <gi...@apache.org>.
ldacey commented on issue #34022:
URL: https://github.com/apache/airflow/issues/34022#issuecomment-1705536872

   It looks like 9 months ago (Nov 2022) I added changed my config to use deepcopy.
   
   ```python
   from copy import deepcopy
   from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG
   
   REMOTE_BASE_LOG_FOLDER = "wasb://container@account.blob.core.windows.net"
   LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)
   ```
   
   I also had a env var for REMOTE_BASE_LOG_FOLDER set as well. I assume that this variable was overwriting the variable in my log_config.py.
   
   <img width="1163" alt="image" src="https://github.com/apache/airflow/assets/12517855/5b2b64f6-a300-4bb3-930e-3f10801d864e">
   
   ___
   > I doubt if you were able to specify the remote_base_log_folder previously as wasb://airflow-logs@account.blob.core.windows.net as you mentioned above. 
   
   
   Isn't that what the official docs show? https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/logging/index.html
   
   <img width="857" alt="image" src="https://github.com/apache/airflow/assets/12517855/9ec2e245-62e9-4a74-8973-9f6fafe1cba9">
   
   
   Regarding the previous versions, my last image build was 4 months ago and Airflow 2.6.0 and the 6.0.0 provider for Azure were installed.
   ```
   Downloading apache_airflow-2.6.0-py3-none-any.whl (12.1 MB)
   Downloading apache_airflow_providers_microsoft_azure-6.0.0-py3-none-any.whl (118 kB)
   ```
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ldacey commented on issue #34022: Unable to read or write logs on Azure Blob after 2.7.0 update

Posted by "ldacey (via GitHub)" <gi...@apache.org>.
ldacey commented on issue #34022:
URL: https://github.com/apache/airflow/issues/34022#issuecomment-1707209690

   Hm, I have been using the same conn_id since 2018 for Azure Blob (HTTP conn type):
   ```yaml
     host: https://account_name.blob.core.windows.net/etl01?restype=container&comp=list
     login: account_name
     password: account_key
     port: null
     schema: ''
     ```
   
   I made a new conn_id with the airflow-logs container in the host URL using the "Azure Blob Storage" connection type template in the UI:
   https://account_name.blob.core.windows.net/airflow-logs?restype=container&comp=list
   
   This worked, but it made a folder "airflow-logs/wasb-airflow-logs" inside of my airflow-logs container.
   
   I changed the URL to "https://account_name.blob.core.windows.net?restype=container&comp=list" and I am good to go now. Logs are being saved and written successfully.
   
   Closing this. It seems like there was some change but it works and this makes sense (the etl01 container in my original host URL must have been ignored since that is not the logging container, and it was literally a test container I made when I first setup Azure Blob years ago).
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ephraimbuddy commented on issue #34022: Unable to read or write logs on Azure Blob after 2.7.0 update

Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #34022:
URL: https://github.com/apache/airflow/issues/34022#issuecomment-1705546056

   > Isn't that what the official docs show? https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/logging/index.html
   
   Surprising...
   Well, regardless of your previous settings, it should work but to troubleshoot why it's not working(I'm assuming it's a connection problem), can you share a little info about your connection? Does the host URL start with https? if you don't use that, what route do you use to connect? SAS token or shared_access_key in extra?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ephraimbuddy commented on issue #34022: Unable to read or write logs on Azure Blob after 2.7.0 update

Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #34022:
URL: https://github.com/apache/airflow/issues/34022#issuecomment-1705549135

   > I also had a env var for REMOTE_BASE_LOG_FOLDER set as well. I assume that this variable was overwriting the variable in my log_config.py.
   
   I don't think the one in config is necessary. The env var will always override it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org