[D] Airflow using NFS(PVC) - jobs running in namespace different from airflow install namespace [airflow]

["karanalang (via GitHub)" <gi...@apache.org>]

GitHub user karanalang created a discussion: Airflow using NFS(PVC) - jobs running in namespace different from airflow install namespace

I've Airflow installed in namespace - airflow, and i'm using NFS to store the dags & pyspark code. I want to run the airflow jobs in different namespace - airflow-spark-apps .. however, i'm unable to do this since the PVC associated with the ns - airflow-spark-apps is not accessible to airflow deployed in ns - airflow.

Here is the helm chart deployment :

```
helm upgrade --install airflow apache-airflow/airflow \
    --namespace airflow \
    --set dags.persistence.enabled=true \
    --set dags.persistence.existingClaim=airflow-dags-pvc \
    --set dags.persistence.subPath="airflow-dags" \
    --set global.persistence.existingClaimNamespace=airflow-spark-apps \    
    --set dags.gitSync.enabled=false \
    --set images.airflow.repository=[artifacts.versa-networks.com](http://artifacts.versa-networks.com/):8443/airflow-image \
    --set images.airflow.tag=0.0.1 \
    --set scheduler.resources.requests.memory="1024Mi" \
    --set scheduler.resources.requests.cpu="500m" \
    --set scheduler.resources.limits.memory="2048Mi" \
    --set scheduler.resources.limits.cpu="1000m" \
    --set webserver.resources.requests.memory="512Mi" \
    --set webserver.resources.requests.cpu="250m" \
    --set webserver.resources.limits.memory="1024Mi" \
    --set webserver.resources.limits.cpu="500m" \
    --set workers.resources.requests.memory="1024Mi" \
    --set workers.resources.requests.cpu="500m" \
    --set workers.resources.limits.memory="4096Mi" \
    --set workers.resources.limits.cpu="2000m" \
    --version 1.9.0 \
    --set serviceAccount.create=false \
    --set serviceAccount.name=airflow \
    --set config.kubernetes_executor.multi_namespace_mode=True \
    --set config.kubernetes_executor.multi_namespace_mode_namespace_list="airflow-spark-apps" \
    --debug
```

PVC & PV created in namespace - airflow-spark-apps:

```
apiVersion: v1
kind: PersistentVolume
metadata:
  name: airflow-dags-pv
  namespace: storage
  labels:
    app: airflow-dags
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: nfs-service.storage.svc.cluster.local
    path: "/exports/airflow-dags"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: airflow-dags-pvc
  namespace: airflow-spark-apps
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ""
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      app: airflow-dags
```

Airflow worker is not starting up, error :

```

Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  67s (x33 over 161m)  default-scheduler  0/4 nodes are available: persistentvolumeclaim "airflow-dags-pvc" not found. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling..
```

How do I fix this ? Can I have the airflow jobs running in namespace - airflow-spark-apps, while the airflow install is in ns - airflow ?

tia!


GitHub link: https://github.com/apache/airflow/discussions/44439

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Airflow using NFS(PVC) - jobs running in namespace different from airflow install namespace [airflow]

["jscheffl (via GitHub)" <gi...@apache.org>]

GitHub user jscheffl added a comment to the discussion: Airflow using NFS(PVC) - jobs running in namespace different from airflow install namespace

Have you considered to have a PV and PVC in each namespace (PV need to be per namespace so you need to have two different not to have a name colliding and each NS to have a PVC) and each of the definitions point to the same NFS backend server+export? Then you can "share" but still have no clash.

(Long time ago that we had a on-prem NFS setup but I hardly remember that we also mounted NFS volumes across namespaces, just needed more PV+PVC setups per namespace pointint to the same backend)

GitHub link: https://github.com/apache/airflow/discussions/44439#discussioncomment-11410668

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org