Re: [D] Remote Logging to Azure Blob Storage Still Broke in Airflow 3.0.4 [airflow]

["pawgajda-drs (via GitHub)" <gi...@apache.org>]

GitHub user pawgajda-drs added a comment to the discussion: Remote Logging to Azure Blob Storage Still Broke in Airflow 3.0.4

I will paste my comment from https://github.com/apache/airflow/discussions/37461, maybe it will help some of you with setting this up:

If anyone's stumbles upon this issue when trying to make it work, here are my notes.

1. First of all, when running Helm Chart version 1.18.0, you're on Airflow 3.0.2. Override your images to 3.0.6, otherwise it just won't work at all:
```
images:
  airflow:
    repository: apache/airflow
    tag: "3.0.6"
  pod_template:
    repository: apache/airflow
    tag: "3.0.6"
```

2. Configure WASB/Azure Blob Storage Conenction via Airflow UI or via Environment Variable.
More information here:  https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#storing-connections-in-environment-variables

If you want to configure it via Environment Variable, I recommend this format:
```
AIRFLOW_CONN_WASB_LOGGING='{"conn_type": "wasb", "host": "__STORAGE_ACCOUNT_HOST__", "login": "__STORAGE_ACCOUNT_NAME__", "password": "__STORAGE_ACCOUNT_SHARED_KEY__"}'
```
bare minimum version:
```
AIRFLOW_CONN_WASB_LOGGING='{"conn_type": "wasb", "login": "__STORAGE_ACCOUNT_NAME__", "password": "__STORAGE_ACCOUNT_SHARED_KEY__"}'
```
I recommend setting it up as `Secret` then pass it to Helm Chart like this:
```
extraEnvFrom: |
  - secretRef:
      name: airflow-remote-logging-secret
```

Finally, the configuration itself:
```
env:
  # Enable Remote Logging
  - name: "AIRFLOW__LOGGING__REMOTE_LOGGING"
    value: "True"
  # Delete Local Logs when Remote Logging is Enabled
  - name: "AIRFLOW__LOGGING__DELETE_LOCAL_LOGS"
    value: "True"
  # Directory inside Blob Container
  # needs to start with 'wasb' or 'wasb/' or 'wasb://'
  - name: "AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER"
    value: "wasb/logs"
  # Blob Container Name
  - name: "AIRFLOW__AZURE_REMOTE_LOGGING__REMOTE_WASB_LOG_CONTAINER"
    value: "airflow-logs"
  # Auth for Blob Storage
  # The naming convention is AIRFLOW_CONN_{CONN_ID} (lowercase value here)
  - name: "AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID"
    value: "wasb_logging"
```

A few notes:
Regarding `AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER` value, documentation here: https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/logging/index.html says to use
```
wasb://path/to/logs
```
format, but for version 3.0.6 do `wasb/path/to/logs`, otherwise your path will look like `wasb:/noname/path/to/logs`. 

Regarding `AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID` set it to lowercase name of your variable or the name of connection you created via UI, that's it.

Thanks to everyone who participated in this discussion and other discussion related to this topic:
https://github.com/apache/airflow/issues/54192
https://github.com/apache/airflow/issues/54413

GitHub link: https://github.com/apache/airflow/discussions/54489#discussioncomment-14508536

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org