Re: [D] Airflow 2.10.5 - DAGs not re-parsed when overwritten in AKS deployment with PVC storage [airflow]

["edgBR (via GitHub)" <gi...@apache.org>]

GitHub user edgBR edited a discussion: Airflow 2.10.5 - DAGs not re-parsed when overwritten in AKS deployment with PVC storage

Hello community,

We have deployed Airflow on AKS (1.32.6) using ArgoCD, following most production recommendations:

- External metadata DB: Azure PostgreSQL.
- External Redis: Azure Redis.
- Kubernetes Workload Identity for authentication.
- Logs: Azure Storage account with PV + PVC.
- DAGs: Azure Storage account with PV + PVC.
- To executors: KubernetesExecutor and CeleryExecutor.

### Deployment approach for the dags

We rely exclusively on the KubernetesPodOperator. Our CI/CD pipeline:

1. Builds Docker images (tagged in a shared registry).
2. Runs DAG parsing unit tests.
3. Syncs DAGs to the target environment.
4. Updates Airflow variables to reference Docker image tags in DEV.
5. Promotes tags (dev → pre-prod) copy the dags to PRE (triggers the dag via API) and updates Airflow variables accordingly.
6. If successfull run, we deploy to PRO.

This workflow allows us to build/deploy multiple times per hour. We avoid the traditional "3-branch setup" because it slowed us down. And because the time is use to build once, deploy to multiple places.

Although we are aware of git-sync being the recommended approach, we opted instead for syncing DAGs through azcopy, which runs as part of the ArgoCD deployment. This job scans storage accounts periodically and copies updates into the `/opt/airflow/dags` folder. We dont know if we need this job or not, or the PVC should be syncing the dags automatically.

`The issue`

- **First deployment of a DAG works fine**.
- **When overwriting an existing DAG file, the changes are not picked up.**
    - The updated file appears in the storage account.
     - However, Airflow does not re-parse the DAG.
- If we **rename the file** and redeploy, the DAG is parsed correctly and the updates appear.

Looking into the metadata tables, it seems **Airflow may be hashing only the DAG file name, not its content**. This raises the question:

- Is Airflow skipping reparsing because it detects the file "hash" as unchanged?
- Or is this behavior caused by something in our PVC/azcopy setup?

Current PVC/PV setup

PersistentVolume:
```yaml

# abbreviated for clarity
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-airflow-dags
  annotations:
    pv.kubernetes.io/provisioned-by: blob.csi.azure.com
spec:
  capacity:
    storage: 20Gi
  accessModes: [ReadWriteMany]
  storageClassName: azureblob-fuse-premium
  mountOptions:
    - -o allow_other
    - --file-cache-timeout-in-seconds=60
    - --use-attr-cache=true
    - --cancel-list-on-mount-seconds=10
    - -o attr_timeout=60
    - -o entry_timeout=60
    - -o negative_timeout=60
    - --virtual-directory=true
    - --log-level=LOG_WARNING
    - --cache-size-mb=3000
    - --use-adls=true    
  csi:
    driver: blob.csi.azure.com
    volumeHandle: <unique-handle>
    volumeAttributes:
      protocol: fuse2
      storageAccount: <account>
      containerName: airflow-dags
      clientID: <workload-identity-client-id>
      resourceGroup: <rg>
      subscriptionId: <sub>

```

PersistentVolumeClaim:

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-airflow-dags
spec:
  storageClassName: azureblob-fuse-premium
  accessModes: [ReadWriteMany]
  resources:
    requests:
      storage: 10Gi
  volumeName: pv-airflow-dags
```

### AzCopy sync job

We use azcopy sync to mirror DAGs from Blob Storage into the Airflow DAGs folder. Example command:

```bash
#!/bin/sh

set -xe  # Enable debugging

echo "azcopy --version:"
azcopy --version
echo "az --version:"
az --version

echo "Contants ${AIRFLOW_DAGS_FOLDER}:"
ls -la "${AIRFLOW_DAGS_FOLDER}"

# echo "date: $(date)"
# echo "date + 2 min: $(date -u -d '2 minutes' '+%Y-%m-%dT%H:%MZ')"
# # echo "date + 2 hours: $(date -u -d '2 hours' '+%Y-%m-%dT%H:%MZ')"

# # Generate the SAS token
# SAS_TOKEN=$(az storage container generate-sas \
#   --account-name $AZURE_STORAGE_ACCOUNT \
#   --account-key $AZURE_STORAGE_KEY \
#   --name $AZURE_STORAGE_CONTAINER \
#   --permissions lr \
#   --expiry $(date -u -d '2 minutes' '+%Y-%m-%dT%H:%MZ') \
#   --output tsv)

# # Check if SAS token was generated
# if [ -z "$SAS_TOKEN" ]; then
#   echo "Failed to generate SAS token"
#   exit 1
# fi

echo "AZURE_FEDERATED_TOKEN_FILE: ${AZURE_FEDERATED_TOKEN_FILE}"
echo "AZURE_CLIENT_ID: ${AZURE_CLIENT_ID}"
echo "AZCOPY_TENANT_ID: ${AZCOPY_TENANT_ID}"
echo "AZURE_STORAGE_ACCOUNT: ${AZURE_STORAGE_ACCOUNT}"
echo "AZURE_STORAGE_CONTAINER: ${AZURE_STORAGE_CONTAINER}"
echo "AIRFLOW_DAGS_FOLDER: ${AIRFLOW_DAGS_FOLDER}"
echo "AIRFLOW_DAGS_PATH: ${AIRFLOW_DAGS_PATH}"
echo "AZCOPY_AUTO_LOGIN_TYPE: ${AZCOPY_AUTO_LOGIN_TYPE}"

# echo "Cleaning up old DAGs..."
# rm -rf ${AIRFLOW_DAGS_FOLDER}*
# rm -rf ${AIRFLOW_DAGS_FOLDER}.*

set +x # Disable debugging for sensitive information
# As azcopy dosn't support Workload Identity login - https://github.com/Azure/azure-storage-azcopy/issues/2545#issuecomment-1967036204
# we need to use az login with the federated token
az login --federated-token "$(cat ${AZURE_FEDERATED_TOKEN_FILE})" --service-principal -u ${AZURE_CLIENT_ID} -t ${AZCOPY_TENANT_ID}
set -x # Enable debugging again

azcopy sync "https://${AZURE_STORAGE_ACCOUNT}.blob.core.windows.net/${AZURE_STORAGE_CONTAINER}" "${AIRFLOW_DAGS_FOLDER}" --recursive=true --delete-destination=true


echo "Contants `${AIRFLOW_DAGS_FOLDER}${AIRFLOW_DAGS_PATH}` path after sync:"
ls -la "${AIRFLOW_DAGS_FOLDER}${AIRFLOW_DAGS_PATH}"

echo "Contants `${AIRFLOW_DAGS_FOLDER}${AIRFLOW_DAGS_PATH}..` path after sync:"
ls -la "${AIRFLOW_DAGS_FOLDER}${AIRFLOW_DAGS_PATH}.."

# echo "Test check file `${AIRFLOW_DAGS_FOLDER}${AIRFLOW_DAGS_PATH}data_flow/data_flow_pipeline.py` content after sync:"
# cat "${AIRFLOW_DAGS_FOLDER}${AIRFLOW_DAGS_PATH}data_flow/data_flow_pipeline.py"

# Run the sync every 30 seconds
# while true; do
#     azcopy login --identity --identity-client-id "${AZCOPY_MSI_CLIENT_ID}"
#     azcopy sync "https://${AZURE_STORAGE_ACCOUNT}.blob.core.windows.net/${AZURE_STORAGE_CONTAINER}/" "${AIRFLOW_DAGS_FOLDER}"
#     sleep 30
# done
```

### Question

1. Is this an expected Airflow behavior (hashing filenames only)?
2. If so, how can we ensure DAG updates are re-parsed without renaming files?
3. Or is this likely an issue with our PVC + azcopy sync setup?


We’ve been stuck on this for over a month and would appreciate any guidance.


GitHub link: https://github.com/apache/airflow/discussions/54937

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org