You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ldacey (via GitHub)" <gi...@apache.org> on 2023/09/01 17:58:56 UTC
[GitHub] [airflow] ldacey opened a new issue, #34022: Unable to read or write logs on Azure Blob after 2.7.0 update
ldacey opened a new issue, #34022:
URL: https://github.com/apache/airflow/issues/34022
### Apache Airflow version
2.7.0
### What happened
After updating to 2.7.0 my worker is unable to read or write logs to Azure Blob. My previous configuration has not been changed since May 2022, so there must have been a recent change which broke it.
My container is "airflow-logs" and then I save production logs in "wasb-airflow-logs" folder/prefix. There are some other 'folders' for different environments.
docker stack variables:
```python
AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS: log_config.LOGGING_CONFIG
AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: wasb-airflow-logs
```
log_config.py
```python
from copy import deepcopy
from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG
REMOTE_BASE_LOG_FOLDER = "wasb://airflow-logs@account.blob.core.windows.net"
LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)
```
After 2.7.0 upgrade I noticed that my worker was spamming the following error message. It seems like Airflow suddenly things "wasb-airflow-logs" is the container instead of the prefix?
```
[2023-09-01 14:39:29,785: ERROR/ForkPoolWorker-11] Could not write logs to wasb-airflow-logs/dag_id=
azure.core.exceptions.ResourceNotFoundError: The specified container does not exist.
```
And my tasks all started with this error:
```
*** tried listing blobs with prefix=wasb-airflow-logs/dag_id=...attempt=1.log and container=airflow-logs
*** could not list blobs The requested URI does not represent any resource on the server.
```
For now, I had to revert my changes to use `AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER` instead which allows the worker to run at least, but it cannot read/write logs.
### What you think should happen instead
_No response_
### How to reproduce
Any change I made to the REMOTE_BASE_LOG_FOLDER did not work and resulted in an error:
```
ImportError: Unable to load custom logging from log_config.LOGGING_CONFIG due to Incorrect remote log configuration. Please check the configuration of option 'host' in section 'elasticsearch' if you are using Elasticsearch. In the other case, 'remote_base_log_folder' option in the 'logging' section.
```
Here are the things I tried:
```python
REMOTE_BASE_LOG_FOLDER = "wasb-airflow-logs"
REMOTE_BASE_LOG_FOLDER = "wasb://airflow-logs@account.blob.core.windows.net"
REMOTE_BASE_LOG_FOLDER = "wasb://airflow-logs@account.blob.core.windows.net/wasb-airflow-logs/"
LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)
```
### Operating System
Ubuntu 22.04
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other Docker-based deployment
### Deployment details
docker stack deploy
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #34022: Unable to read or write logs on Azure Blob after 2.7.0 update
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #34022:
URL: https://github.com/apache/airflow/issues/34022#issuecomment-1705490342
@ldacey, I would like to know what your configurations were before the upgrade.
1. What was your `apache-airflow-providers-microsoft-azure` version previously?
2. What version were you upgrading to
3. How did you provide your connection previously? was it through env var or you entered it in the UI? If with environment variable, what was the format e.g was it `wasb://username:accountkey@account....`.
I doubt if you were able to specify the remote_base_log_folder previously as `wasb://airflow-logs@account.blob.core.windows.net` as you mentioned above. That's almost a full connection uri which I think should not be what remote_base_log_folder should be(for azure) but I'm interested to learn how you made your connection previously.
There's a new version of the provider: 6.3.0, maybe you should try that too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajkoti commented on issue #34022: Unable to read or write logs on Azure Blob after 2.7.0 update
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on issue #34022:
URL: https://github.com/apache/airflow/issues/34022#issuecomment-1703261263
hi @ldacey
Can you check what is the version of apache-airflow-providers-microsoft-azure in your deployment. I don't think it would be directly related to a change in Airflow 2.7.0, but could rather be with the provider.
A new version of the provider was just released today https://pypi.org/project/apache-airflow-providers-microsoft-azure/6.3.0/
Can you try installing 6.3.0 version of the provider and see if the issue persists?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajkoti commented on issue #34022: Unable to read or write logs on Azure Blob after 2.7.0 update
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on issue #34022:
URL: https://github.com/apache/airflow/issues/34022#issuecomment-1704004495
you can keep the constraints as is and just install the newer azure provider at the end of your image build process
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ldacey closed issue #34022: Unable to read or write logs on Azure Blob after 2.7.0 update
Posted by "ldacey (via GitHub)" <gi...@apache.org>.
ldacey closed issue #34022: Unable to read or write logs on Azure Blob after 2.7.0 update
URL: https://github.com/apache/airflow/issues/34022
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #34022: Unable to read or write logs on Azure Blob after 2.7.0 update
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #34022:
URL: https://github.com/apache/airflow/issues/34022#issuecomment-1704024639
> Weird thing is that the scheduler died immediately (must be due to some other package since I remove constraints completely for this build). Ill try to revert back and add the 6.3.0 version
This is a known Celery bug that has been introduced few days and they already deleted 5.3.2 and 5.3.3 releases that cause the problem - we've reported it to celery team and added protection for it for 2.7.1 but also celery team removed the buggy versions.
As @pankajkoti explained: the **RIGHT** (and intended) way of installing different version of packages than that in the constraints is to install airflow first with constraints and THEN separately adding/removing/updating packages without the constraints.
See this part of our documentation that explains it and gives examples: https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#upgrading-and-installing-dependencies-including-providers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ldacey commented on issue #34022: Unable to read or write logs on Azure Blob after 2.7.0 update
Posted by "ldacey (via GitHub)" <gi...@apache.org>.
ldacey commented on issue #34022:
URL: https://github.com/apache/airflow/issues/34022#issuecomment-1703398855
This is the version which my image was using:
Downloading apache_airflow_providers_microsoft_azure-6.2.4-py3-none-any.whl.metadata (7.6 kB)
I removed constraints to try to rebuild and the latest version was added:
Downloading apache_airflow_providers_microsoft_azure-6.3.0-py3-none-any.whl.metadata
Weird thing is that the scheduler died immediately (must be due to some other package since I remove constraints completely for this build). Ill try to revert back and add the 6.3.0 version
```
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column celery_taskmeta.children does not exist
LINE 1: ...ry_taskmeta.date_done, celery_taskmeta.traceback, celery_tas...
^
[SQL: SELECT celery_taskmeta.id, celery_taskmeta.task_id, celery_taskmeta.status, celery_taskmeta.result, celery_taskmeta.date_done, celery_taskmeta.traceback, celery_taskmeta.children
FROM celery_taskmeta
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ldacey commented on issue #34022: Unable to read or write logs on Azure Blob after 2.7.0 update
Posted by "ldacey (via GitHub)" <gi...@apache.org>.
ldacey commented on issue #34022:
URL: https://github.com/apache/airflow/issues/34022#issuecomment-1705454732
After pinning celery, I have no issues with the workers. I still cannot read or write logs on Azure though for some reason.
This prefix and container are correct. After my update to 2.7.0 there are no remote logs though. Logs from task (reading remote logs):
> *** tried listing blobs with prefix=wasb-airflow-logs/dag_id=zendesk/run_id=scheduled__2023-09-04T06:51:00+00:00/task_id=support.transform.custom-fields/attempt=1.log and container=airflow-logs
> *** could not list blobs The requested URI does not represent any resource on the server.
Logs from worker are below. Is the worker trying to write to a container called "wasb-airflow-logs" instead of "airflow-logs"? I am not sure why this is different from the logs being read which is correct (container is "airflow-logs" and prefix is "wasb-airflow-logs". As mentioned, this configuration has worked for a long time with no changes.
> [2023-09-04 15:26:26,796: ERROR/ForkPoolWorker-3] Could not write logs to wasb-airflow-logs/dag_id=example/run_id=scheduled__2023-09-01T14:10:00+00:00/task_id=offered.extract/attempt=2.log
> Traceback (most recent call last):
> azure.core.exceptions.ResourceNotFoundError: The specified container does not
log_config.py (I tested a few REMOTE_BASE_LOG_FOLDER values in this file as well, as you can see from my original issue).
```python
from copy import deepcopy
from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG
LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)
```
env vars:
```python
AIRFLOW__LOGGING__REMOTE_LOGGING: "true"
AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: azure_blob
AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: wasb-airflow-logs
```
pip freeze results:
```
apache-airflow==2.7.0
apache-airflow-providers-celery==3.3.3
apache-airflow-providers-common-sql==1.7.1
apache-airflow-providers-docker==3.7.4
apache-airflow-providers-ftp==3.5.1
apache-airflow-providers-google==10.7.0
apache-airflow-providers-http==4.5.1
apache-airflow-providers-imap==3.3.1
apache-airflow-providers-microsoft-azure==6.3.0
apache-airflow-providers-mysql==5.3.0
apache-airflow-providers-odbc==4.0.0
apache-airflow-providers-postgres==5.6.0
apache-airflow-providers-redis==3.3.1
apache-airflow-providers-salesforce==5.4.2
apache-airflow-providers-sftp==4.6.0
apache-airflow-providers-sqlite==3.4.3
apache-airflow-providers-ssh==3.7.2
adal==1.2.7
adlfs==2023.8.0
azure-batch==14.0.0
azure-common==1.1.28
azure-core==1.29.3
azure-cosmos==4.5.0
azure-datalake-store==0.0.53
azure-identity==1.14.0
azure-keyvault-secrets==4.7.0
azure-kusto-data==4.2.0
azure-mgmt-containerinstance==8.0.0
azure-mgmt-core==1.4.0
azure-mgmt-cosmosdb==9.2.0
azure-mgmt-datafactory==1.1.0
azure-mgmt-datalake-nspkg==3.0.1
azure-mgmt-datalake-store==0.5.0
azure-mgmt-nspkg==3.0.2
azure-mgmt-resource==23.0.1
azure-nspkg==3.0.2
azure-servicebus==7.11.1
azure-storage-blob==12.17.0
azure-storage-common==2.1.0
azure-storage-file==2.1.0
azure-storage-file-datalake==12.12.0
azure-synapse-spark==0.7.0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ldacey commented on issue #34022: Unable to read or write logs on Azure Blob after 2.7.0 update
Posted by "ldacey (via GitHub)" <gi...@apache.org>.
ldacey commented on issue #34022:
URL: https://github.com/apache/airflow/issues/34022#issuecomment-1705536872
It looks like 9 months ago (Nov 2022) I added changed my config to use deepcopy.
```python
from copy import deepcopy
from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG
REMOTE_BASE_LOG_FOLDER = "wasb://container@account.blob.core.windows.net"
LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)
```
I also had a env var for REMOTE_BASE_LOG_FOLDER set as well. I assume that this variable was overwriting the variable in my log_config.py.
<img width="1163" alt="image" src="https://github.com/apache/airflow/assets/12517855/5b2b64f6-a300-4bb3-930e-3f10801d864e">
___
> I doubt if you were able to specify the remote_base_log_folder previously as wasb://airflow-logs@account.blob.core.windows.net as you mentioned above.
Isn't that what the official docs show? https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/logging/index.html
<img width="857" alt="image" src="https://github.com/apache/airflow/assets/12517855/9ec2e245-62e9-4a74-8973-9f6fafe1cba9">
Regarding the previous versions, my last image build was 4 months ago and Airflow 2.6.0 and the 6.0.0 provider for Azure were installed.
```
Downloading apache_airflow-2.6.0-py3-none-any.whl (12.1 MB)
Downloading apache_airflow_providers_microsoft_azure-6.0.0-py3-none-any.whl (118 kB)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ldacey commented on issue #34022: Unable to read or write logs on Azure Blob after 2.7.0 update
Posted by "ldacey (via GitHub)" <gi...@apache.org>.
ldacey commented on issue #34022:
URL: https://github.com/apache/airflow/issues/34022#issuecomment-1707209690
Hm, I have been using the same conn_id since 2018 for Azure Blob (HTTP conn type):
```yaml
host: https://account_name.blob.core.windows.net/etl01?restype=container&comp=list
login: account_name
password: account_key
port: null
schema: ''
```
I made a new conn_id with the airflow-logs container in the host URL using the "Azure Blob Storage" connection type template in the UI:
https://account_name.blob.core.windows.net/airflow-logs?restype=container&comp=list
This worked, but it made a folder "airflow-logs/wasb-airflow-logs" inside of my airflow-logs container.
I changed the URL to "https://account_name.blob.core.windows.net?restype=container&comp=list" and I am good to go now. Logs are being saved and written successfully.
Closing this. It seems like there was some change but it works and this makes sense (the etl01 container in my original host URL must have been ignored since that is not the logging container, and it was literally a test container I made when I first setup Azure Blob years ago).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #34022: Unable to read or write logs on Azure Blob after 2.7.0 update
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #34022:
URL: https://github.com/apache/airflow/issues/34022#issuecomment-1705546056
> Isn't that what the official docs show? https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/logging/index.html
Surprising...
Well, regardless of your previous settings, it should work but to troubleshoot why it's not working(I'm assuming it's a connection problem), can you share a little info about your connection? Does the host URL start with https? if you don't use that, what route do you use to connect? SAS token or shared_access_key in extra?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #34022: Unable to read or write logs on Azure Blob after 2.7.0 update
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #34022:
URL: https://github.com/apache/airflow/issues/34022#issuecomment-1705549135
> I also had a env var for REMOTE_BASE_LOG_FOLDER set as well. I assume that this variable was overwriting the variable in my log_config.py.
I don't think the one in config is necessary. The env var will always override it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org