Airflow Worker - DAGs folder is pointing to Scheduler machines DAG bag

[devang pandey <de...@gmail.com>]

Hi All,

I am pretty new to Airflow and we are planning to use it at organisation
level for automations (mainly data-extraction pipelines).

I have setup an airflow cluster with one server and 2 VDI machines. Using
Airflow 1.10.9 , postgres , celery 4.4.0 and redis as message broker.

Thing is I am successfully able to run example dags on my cluster but when
trying to run my own dag - its gets scheduled and picked up by worker
machine but then fails with an error message "DAG ID could not found".

On taking a closer look I observed my worker machine while picking up the
task is passing a - sd parameter which is pointing to dag folder of my
scheduler machine / server .

*Things I can confirm:*

*1- DAG is present on all machines , master and worker machines inside
AIRFLOW_HOME/dags*
*2- Airflow worker config file points to local correct dag location.*

I tried multiple things but could not understand the issue. Please if you
provide suggestions.


Thank you,
Devang

Re: Airflow Worker - DAGs folder is pointing to Scheduler machines DAG bag

[devang pandey <de...@gmail.com>]

Thanks for quick response, will try to keep the path identical . Thanks.

On Thu, May 7, 2020 at 3:14 PM Ash Berlin-Taylor <as...@apache.org> wrote:

> Here's the issue I was thinking of
> https://issues.apache.org/jira/browse/AIRFLOW-5177
>
> On May 7 2020, at 10:43 am, Ash Berlin-Taylor <as...@apache.org> wrote:
>
> Right now the location of the dags folder needs to be identical on both
> the scheduler and all the workers. Sorry.
>
> This is a bug, and it *shouldn't* need to be the case. (Apache Jira is
> having a hiccough right now so I can't find the issue at the moment.)
>
> -ash
>
> On May 7 2020, at 8:53 am, devang pandey <de...@gmail.com> wrote:
>
> Hi All,
>
> I am pretty new to Airflow and we are planning to use it at organisation
> level for automations (mainly data-extraction pipelines).
>
> I have setup an airflow cluster with one server and 2 VDI machines. Using
> Airflow 1.10.9 , postgres , celery 4.4.0 and redis as message broker.
>
> Thing is I am successfully able to run example dags on my cluster but when
> trying to run my own dag - its gets scheduled and picked up by worker
> machine but then fails with an error message "DAG ID could not found".
>
> On taking a closer look I observed my worker machine while picking up the
> task is passing a - sd parameter which is pointing to dag folder of my
> scheduler machine / server .
>
> *Things I can confirm:*
>
> *1- DAG is present on all machines , master and worker machines inside
> AIRFLOW_HOME/dags*
> *2- Airflow worker config file points to local correct dag location.*
>
> I tried multiple things but could not understand the issue. Please if you
> provide suggestions.
>
>
> Thank you,
> Devang
>
>

Re: Airflow Worker - DAGs folder is pointing to Scheduler machines DAG bag

[Ash Berlin-Taylor <as...@apache.org>]

Here's the issue I was thinking of https://issues.apache.org/jira/browse/AIRFLOW-5177

On May 7 2020, at 10:43 am, Ash Berlin-Taylor <as...@apache.org> wrote:
> Right now the location of the dags folder needs to be identical on both the scheduler and all the workers. Sorry.
>
> This is a bug, and it shouldn't need to be the case. (Apache Jira is having a hiccough right now so I can't find the issue at the moment.)
> -ash
> On May 7 2020, at 8:53 am, devang pandey <de...@gmail.com> wrote:
> > Hi All,
> >
> > I am pretty new to Airflow and we are planning to use it at organisation level for automations (mainly data-extraction pipelines).
> >
> > I have setup an airflow cluster with one server and 2 VDI machines. Using Airflow 1.10.9 , postgres , celery 4.4.0 and redis as message broker.
> >
> > Thing is I am successfully able to run example dags on my cluster but when trying to run my own dag - its gets scheduled and picked up by worker machine but then fails with an error message "DAG ID could not found".
> >
> > On taking a closer look I observed my worker machine while picking up the task is passing a - sd parameter which is pointing to dag folder of my scheduler machine / server .
> >
> > Things I can confirm:
> >
> > 1- DAG is present on all machines , master and worker machines inside AIRFLOW_HOME/dags
> > 2- Airflow worker config file points to local correct dag location.
> >
> > I tried multiple things but could not understand the issue. Please if you provide suggestions.
> >
> >
> > Thank you,
> > Devang
> >
>

Re: Airflow Worker - DAGs folder is pointing to Scheduler machines DAG bag

[Ash Berlin-Taylor <as...@apache.org>]

Right now the location of the dags folder needs to be identical on both the scheduler and all the workers. Sorry.

This is a bug, and it shouldn't need to be the case. (Apache Jira is having a hiccough right now so I can't find the issue at the moment.)
-ash
On May 7 2020, at 8:53 am, devang pandey <de...@gmail.com> wrote:
> Hi All,
>
> I am pretty new to Airflow and we are planning to use it at organisation level for automations (mainly data-extraction pipelines).
>
> I have setup an airflow cluster with one server and 2 VDI machines. Using Airflow 1.10.9 , postgres , celery 4.4.0 and redis as message broker.
>
> Thing is I am successfully able to run example dags on my cluster but when trying to run my own dag - its gets scheduled and picked up by worker machine but then fails with an error message "DAG ID could not found".
>
> On taking a closer look I observed my worker machine while picking up the task is passing a - sd parameter which is pointing to dag folder of my scheduler machine / server .
>
> Things I can confirm:
>
> 1- DAG is present on all machines , master and worker machines inside AIRFLOW_HOME/dags
> 2- Airflow worker config file points to local correct dag location.
>
> I tried multiple things but could not understand the issue. Please if you provide suggestions.
>
>
> Thank you,
> Devang
>