You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Kaxil Naik (JIRA)" <ji...@apache.org> on 2018/12/30 00:44:00 UTC
[jira] [Updated] (AIRFLOW-2755) k8s workers think DAGs are always
in `/tmp/dags`
[ https://issues.apache.org/jira/browse/AIRFLOW-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kaxil Naik updated AIRFLOW-2755:
--------------------------------
Fix Version/s: (was: 2.0.0)
1.10.2
> k8s workers think DAGs are always in `/tmp/dags`
> ------------------------------------------------
>
> Key: AIRFLOW-2755
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2755
> Project: Apache Airflow
> Issue Type: Bug
> Components: configuration, worker
> Reporter: Aldo
> Assignee: Aldo
> Priority: Minor
> Fix For: 1.10.2
>
>
> We have Airflow configured to use the `KubernetesExecutor` and run tasks in newly created pods.
> I tried to use the `PythonOperator` to import the python callable from a python module located in the DAGs directory as [that should be possible|https://github.com/apache/incubator-airflow/blob/c7a472ed6b0d8a4720f57ba1140c8cf665757167/airflow/__init__.py#L42]. Airflow complained that the module was not found.
> After a fair amount of digging we found that the issue was that the workers have the `AIRFLOW__CORE__DAGS_FOLDER` environment variable set to `/tmp/dags` as [you can see from the code|https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/kubernetes/worker_configuration.py#L84].
> Unset that environment variable from within the task's pod and running the task manually worked as expected. I think that this path should be configurable (I'll give it a try to add a `kubernetes.worker_dags_folder` configuration).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)