You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by "Niebler Thomas (DC-IH/SDL1)" <Th...@boschrexroth.de.INVALID> on 2020/05/04 06:41:55 UTC

Potential issue with serialized DAGs in decoupled Webserver and Scheduler

Hi all,

I have a probably rather special use case scenario:
Using Airflow 1.10.10, I would like to physically decouple the Webserver and the Scheduler for some secure access reasons. According to https://airflow.apache.org/docs/stable/dag-serialization.html, this should be a piece of cake with Airflow 1.10.10, since DAGs are stored in the Metadata database and the webserver does not need to access the DAG files anymore. The metadata database is of course reachable by both Airflow instances:

Docker Image Instance: Airflow Webserver <----> Docker Image Instance: Metadata database <----> Physical Machine: Airflow Scheduler

However, every time I start a DAG manually, it crashes with a rather lengthy error message:

  File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 1255, in trigger
    external_trigger=True
  File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/airflow/models/dag.py", line 1818, in create_dagrun
    return self.get_dag().create_dagrun(run_id=run_id,
AttributeError: 'NoneType' object has no attribute 'create_dagrun'

This basically boils down to self.get_dag() not having set the Boolean flag store_serialized_flags to True (or whatever value the config is set to), but always using False (the default value).
This then leads to Airflow attempting to read the DAG file, ignoring the DAG database entry and returning None, which obviously has no attribute create_dagrun.

I’ve got several questions here now:

  1.  Is my scenario even possible or am I overlooking something rather obvious?
  2.  Is the crashing DAG behavior intended like that? It rather seems like a bug to me.
  3.  Is it worth fixing this issue (if it is one) for Airflow 1.10.x, considering that Airflow 2.0.0 does not even contain the corresponding classes anymore and takes a different path?

Mit freundlichen Grüßen / Best regards

Dr. Thomas Niebler
Data Scientist
Sales Data Lab, Analytics DC-IH/SDL1

Tel. +49 9352 18-2392
Fax +49 9352 18-0
Thomas.Niebler@boschrexroth.de<ma...@boschrexroth.de>
www.boschrexroth.com

Bosch Rexroth AG
Partensteiner Straße 23
97816 Lohr am Main
GERMANY

[BOSCH REXROTH]<http://www.boschrexroth.com/>



Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart HRB 23192
Vorstand: Rolf Najork (Vorsitzender), Dr. Markus Forschner, Dr. Heiner Lang, Reinhard Schäfer, Dr. Marc Wucherer
Vorsitzender des Aufsichtsrats: Christoph Kübel
​

Re: Potential issue with serialized DAGs in decoupled Webserver and Scheduler

Posted by Kaxil Naik <ka...@gmail.com>.
Hi Anita,

Yes, we fixed all the bugs that were found or reported during 1.10.10 rc
voting.

There were some bugs in the DagCode that were found in 1.10.10rc's (have
tagged you in few of the PR with fixes) but
fixed before 1.10.10 was released.

Some of Astronomer's (https://www.astronomer.io/cloud/) customers use Dag
Serialization in production (with Airflow >=1.10.5) and haven't reported
any critical/major issues
yet even with 1.10.10.

But after looking at the changes I do not think there are many major issues
> left that have not been fixed in Airflow 1.10.10 already.


I looked at the PR and I am not sure it fixes anything. "conf.getboolean"
is an expensive operation and moving the store_serialized_dags
config from settings.py  to all places just means that conf.getboolean is
called every time and if it is run inside a loop, that would
increase the time taken by the process running the loop.

store_serialized_dags should be set to false everywhere except UI, experimental
> API and dag import


That is the case now too, I have left comments on the PR.


Regards,
Kaxil

On Tue, May 5, 2020 at 4:58 PM Anita Fronczak <ak...@gmail.com> wrote:

> Hello again,
>
> So I have created a PR: https://github.com/apache/airflow/pull/8718
>
> Some context. Cloud Composer supports store_serialized_dags and
> store_dag_code for Airflow 1.10.3 and 1.10.6 (but without rendering
> templates) to support VPC SC (
> https://cloud.google.com/composer/docs/release-notes#March_20_2020). Some
> changes were made that did not make into open source Apache Airflow on time
> to make it work.
> But after looking at the changes I do not think there are many major
> issues left that have not been fixed in Airflow 1.10.10 already.
> The PR is a version that merges into master not the one from Airflow
> 1.10.6.
>
> Anita
>
> pon., 4 maj 2020 o 12:09 Niebler Thomas (DC-IH/SDL1)
> <Th...@boschrexroth.de.invalid> napisał(a):
>
>> Hi Kaxil,
>>
>> Awesome, thank you 😊 I should have checked the closed bugs more
>> thoroughly :-X
>>
>> Mit freundlichen Grüßen / Best regards
>>
>> Dr. Thomas Niebler
>> Data Scientist
>> Sales Data Lab, Analytics DC-IH/SDL1
>>
>> Tel. +49 9352 18-2392
>> Fax +49 9352 18-0
>> Thomas.Niebler@boschrexroth.de<ma...@boschrexroth.de>
>> www.boschrexroth.com
>>
>> Bosch Rexroth AG
>> Partensteiner Straße 23
>> 97816 Lohr am Main
>> GERMANY
>>
>> [BOSCH REXROTH]<http://www.boschrexroth.com/>
>>
>>
>>
>> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart HRB 23192
>> Vorstand: Rolf Najork (Vorsitzender), Dr. Markus Forschner, Dr. Heiner
>> Lang, Reinhard Schäfer, Dr. Marc Wucherer
>> Vorsitzender des Aufsichtsrats: Christoph Kübel
>> ​
>> Von: Kaxil Naik <ka...@gmail.com>
>> Gesendet: Monday, May 4, 2020 11:38 AM
>> An: dev@airflow.apache.org
>> Cc: Niebler Thomas (DC-IH/SDL1) <Th...@boschrexroth.de>
>> Betreff: Re: Potential issue with serialized DAGs in decoupled Webserver
>> and Scheduler
>>
>> And there was a wider issue, the trigger method didn't actually used the
>> DagBag already created for the Webserver but instead created a new one
>> which wasn't necessary. And the existing DagBag in views.py already passed
>> all the flags correctly :)
>>
>> Regards,
>> Kaxil
>>
>> On Mon, May 4, 2020, 10:30 Kaxil Naik <kaxilnaik@gmail.com<mailto:
>> kaxilnaik@gmail.com>> wrote:
>> Hi Niebler,
>>
>> Like Ash mentioned this is fixed for 1.10.11 already.
>>
>> This was first reported in following Github issue:
>>
>> - https://github.com/apache/airflow/issues/8247
>>
>> And PR that fixes it for v1-10-test:
>> https://github.com/apache/airflow/pull/8411
>>
>> And https://github.com/apache/airflow/pull/8501 also fixes it in Master.
>>
>> Regards,
>> Kaxil
>>
>> On Mon, May 4, 2020, 09:21 Ash Berlin-Taylor <ash@apache.org<mailto:
>> ash@apache.org>> wrote:
>> I think this was fixed by this PR - aiming for 1.10.11
>>
>> https://github.com/apache/airflow/pull/8501
>>
>> On 4 May 2020 07:41:55 BST, "Niebler Thomas (DC-IH/SDL1)"
>> <Thomas.Niebler@boschrexroth.de.INVALID<mailto:
>> Thomas.Niebler@boschrexroth.de.INVALID>> wrote:
>> >Hi all,
>> >
>> >I have a probably rather special use case scenario:
>> >Using Airflow 1.10.10, I would like to physically decouple the
>> >Webserver and the Scheduler for some secure access reasons. According
>> >to https://airflow.apache.org/docs/stable/dag-serialization.html, this
>> >should be a piece of cake with Airflow 1.10.10, since DAGs are stored
>> >in the Metadata database and the webserver does not need to access the
>> >DAG files anymore. The metadata database is of course reachable by both
>> >Airflow instances:
>> >
>> >Docker Image Instance: Airflow Webserver <----> Docker Image Instance:
>> >Metadata database <----> Physical Machine: Airflow Scheduler
>> >
>> >However, every time I start a DAG manually, it crashes with a rather
>> >lengthy error message:
>> >
>> >File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py",
>> >line 1255, in trigger
>> >    external_trigger=True
>> >File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line
>> >74, in wrapper
>> >    return func(*args, **kwargs)
>> >File "/usr/local/lib/python3.7/site-packages/airflow/models/dag.py",
>> >line 1818, in create_dagrun
>> >    return self.get_dag().create_dagrun(run_id=run_id,
>> >AttributeError: 'NoneType' object has no attribute 'create_dagrun'
>> >
>> >This basically boils down to self.get_dag() not having set the Boolean
>> >flag store_serialized_flags to True (or whatever value the config is
>> >set to), but always using False (the default value).
>> >This then leads to Airflow attempting to read the DAG file, ignoring
>> >the DAG database entry and returning None, which obviously has no
>> >attribute create_dagrun.
>> >
>> >I’ve got several questions here now:
>> >
>> >1.  Is my scenario even possible or am I overlooking something rather
>> >obvious?
>> >2.  Is the crashing DAG behavior intended like that? It rather seems
>> >like a bug to me.
>> >3.  Is it worth fixing this issue (if it is one) for Airflow 1.10.x,
>> >considering that Airflow 2.0.0 does not even contain the corresponding
>> >classes anymore and takes a different path?
>> >
>> >Mit freundlichen Grüßen / Best regards
>> >
>> >Dr. Thomas Niebler
>> >Data Scientist
>> >Sales Data Lab, Analytics DC-IH/SDL1
>> >
>> >Tel. +49 9352 18-2392
>> >Fax +49 9352 18-0
>> >Thomas.Niebler@boschrexroth.de<mailto:Thomas.Niebler@boschrexroth.de
>> ><mailto:Thomas.Niebler@boschrexroth.de<mailto:
>> Thomas.Niebler@boschrexroth.de>>
>> >www.boschrexroth.com<http://www.boschrexroth.com>
>> >
>> >Bosch Rexroth AG
>> >Partensteiner Straße 23
>> >97816 Lohr am Main
>> >GERMANY
>> >
>> >[BOSCH REXROTH]<http://www.boschrexroth.com/>
>> >
>> >
>> >
>> >Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart HRB 23192
>> >Vorstand: Rolf Najork (Vorsitzender), Dr. Markus Forschner, Dr. Heiner
>> >Lang, Reinhard Schäfer, Dr. Marc Wucherer
>> >Vorsitzender des Aufsichtsrats: Christoph Kübel
>> >​
>>
>

Re: Potential issue with serialized DAGs in decoupled Webserver and Scheduler

Posted by Anita Fronczak <ak...@gmail.com>.
Hello again,

So I have created a PR: https://github.com/apache/airflow/pull/8718

Some context. Cloud Composer supports store_serialized_dags and
store_dag_code for Airflow 1.10.3 and 1.10.6 (but without rendering
templates) to support VPC SC (
https://cloud.google.com/composer/docs/release-notes#March_20_2020). Some
changes were made that did not make into open source Apache Airflow on time
to make it work.
But after looking at the changes I do not think there are many major issues
left that have not been fixed in Airflow 1.10.10 already.
The PR is a version that merges into master not the one from Airflow 1.10.6.

Anita

pon., 4 maj 2020 o 12:09 Niebler Thomas (DC-IH/SDL1)
<Th...@boschrexroth.de.invalid> napisał(a):

> Hi Kaxil,
>
> Awesome, thank you 😊 I should have checked the closed bugs more
> thoroughly :-X
>
> Mit freundlichen Grüßen / Best regards
>
> Dr. Thomas Niebler
> Data Scientist
> Sales Data Lab, Analytics DC-IH/SDL1
>
> Tel. +49 9352 18-2392
> Fax +49 9352 18-0
> Thomas.Niebler@boschrexroth.de<ma...@boschrexroth.de>
> www.boschrexroth.com
>
> Bosch Rexroth AG
> Partensteiner Straße 23
> 97816 Lohr am Main
> GERMANY
>
> [BOSCH REXROTH]<http://www.boschrexroth.com/>
>
>
>
> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart HRB 23192
> Vorstand: Rolf Najork (Vorsitzender), Dr. Markus Forschner, Dr. Heiner
> Lang, Reinhard Schäfer, Dr. Marc Wucherer
> Vorsitzender des Aufsichtsrats: Christoph Kübel
> ​
> Von: Kaxil Naik <ka...@gmail.com>
> Gesendet: Monday, May 4, 2020 11:38 AM
> An: dev@airflow.apache.org
> Cc: Niebler Thomas (DC-IH/SDL1) <Th...@boschrexroth.de>
> Betreff: Re: Potential issue with serialized DAGs in decoupled Webserver
> and Scheduler
>
> And there was a wider issue, the trigger method didn't actually used the
> DagBag already created for the Webserver but instead created a new one
> which wasn't necessary. And the existing DagBag in views.py already passed
> all the flags correctly :)
>
> Regards,
> Kaxil
>
> On Mon, May 4, 2020, 10:30 Kaxil Naik <kaxilnaik@gmail.com<mailto:
> kaxilnaik@gmail.com>> wrote:
> Hi Niebler,
>
> Like Ash mentioned this is fixed for 1.10.11 already.
>
> This was first reported in following Github issue:
>
> - https://github.com/apache/airflow/issues/8247
>
> And PR that fixes it for v1-10-test:
> https://github.com/apache/airflow/pull/8411
>
> And https://github.com/apache/airflow/pull/8501 also fixes it in Master.
>
> Regards,
> Kaxil
>
> On Mon, May 4, 2020, 09:21 Ash Berlin-Taylor <ash@apache.org<mailto:
> ash@apache.org>> wrote:
> I think this was fixed by this PR - aiming for 1.10.11
>
> https://github.com/apache/airflow/pull/8501
>
> On 4 May 2020 07:41:55 BST, "Niebler Thomas (DC-IH/SDL1)"
> <Thomas.Niebler@boschrexroth.de.INVALID<mailto:
> Thomas.Niebler@boschrexroth.de.INVALID>> wrote:
> >Hi all,
> >
> >I have a probably rather special use case scenario:
> >Using Airflow 1.10.10, I would like to physically decouple the
> >Webserver and the Scheduler for some secure access reasons. According
> >to https://airflow.apache.org/docs/stable/dag-serialization.html, this
> >should be a piece of cake with Airflow 1.10.10, since DAGs are stored
> >in the Metadata database and the webserver does not need to access the
> >DAG files anymore. The metadata database is of course reachable by both
> >Airflow instances:
> >
> >Docker Image Instance: Airflow Webserver <----> Docker Image Instance:
> >Metadata database <----> Physical Machine: Airflow Scheduler
> >
> >However, every time I start a DAG manually, it crashes with a rather
> >lengthy error message:
> >
> >File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py",
> >line 1255, in trigger
> >    external_trigger=True
> >File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line
> >74, in wrapper
> >    return func(*args, **kwargs)
> >File "/usr/local/lib/python3.7/site-packages/airflow/models/dag.py",
> >line 1818, in create_dagrun
> >    return self.get_dag().create_dagrun(run_id=run_id,
> >AttributeError: 'NoneType' object has no attribute 'create_dagrun'
> >
> >This basically boils down to self.get_dag() not having set the Boolean
> >flag store_serialized_flags to True (or whatever value the config is
> >set to), but always using False (the default value).
> >This then leads to Airflow attempting to read the DAG file, ignoring
> >the DAG database entry and returning None, which obviously has no
> >attribute create_dagrun.
> >
> >I’ve got several questions here now:
> >
> >1.  Is my scenario even possible or am I overlooking something rather
> >obvious?
> >2.  Is the crashing DAG behavior intended like that? It rather seems
> >like a bug to me.
> >3.  Is it worth fixing this issue (if it is one) for Airflow 1.10.x,
> >considering that Airflow 2.0.0 does not even contain the corresponding
> >classes anymore and takes a different path?
> >
> >Mit freundlichen Grüßen / Best regards
> >
> >Dr. Thomas Niebler
> >Data Scientist
> >Sales Data Lab, Analytics DC-IH/SDL1
> >
> >Tel. +49 9352 18-2392
> >Fax +49 9352 18-0
> >Thomas.Niebler@boschrexroth.de<mailto:Thomas.Niebler@boschrexroth.de
> ><mailto:Thomas.Niebler@boschrexroth.de<mailto:
> Thomas.Niebler@boschrexroth.de>>
> >www.boschrexroth.com<http://www.boschrexroth.com>
> >
> >Bosch Rexroth AG
> >Partensteiner Straße 23
> >97816 Lohr am Main
> >GERMANY
> >
> >[BOSCH REXROTH]<http://www.boschrexroth.com/>
> >
> >
> >
> >Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart HRB 23192
> >Vorstand: Rolf Najork (Vorsitzender), Dr. Markus Forschner, Dr. Heiner
> >Lang, Reinhard Schäfer, Dr. Marc Wucherer
> >Vorsitzender des Aufsichtsrats: Christoph Kübel
> >​
>

AW: Potential issue with serialized DAGs in decoupled Webserver and Scheduler

Posted by "Niebler Thomas (DC-IH/SDL1)" <Th...@boschrexroth.de.INVALID>.
Hi Kaxil,

Awesome, thank you 😊 I should have checked the closed bugs more thoroughly :-X

Mit freundlichen Grüßen / Best regards

Dr. Thomas Niebler
Data Scientist
Sales Data Lab, Analytics DC-IH/SDL1

Tel. +49 9352 18-2392
Fax +49 9352 18-0
Thomas.Niebler@boschrexroth.de<ma...@boschrexroth.de>
www.boschrexroth.com

Bosch Rexroth AG
Partensteiner Straße 23
97816 Lohr am Main
GERMANY

[BOSCH REXROTH]<http://www.boschrexroth.com/>



Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart HRB 23192
Vorstand: Rolf Najork (Vorsitzender), Dr. Markus Forschner, Dr. Heiner Lang, Reinhard Schäfer, Dr. Marc Wucherer
Vorsitzender des Aufsichtsrats: Christoph Kübel
​
Von: Kaxil Naik <ka...@gmail.com>
Gesendet: Monday, May 4, 2020 11:38 AM
An: dev@airflow.apache.org
Cc: Niebler Thomas (DC-IH/SDL1) <Th...@boschrexroth.de>
Betreff: Re: Potential issue with serialized DAGs in decoupled Webserver and Scheduler

And there was a wider issue, the trigger method didn't actually used the DagBag already created for the Webserver but instead created a new one which wasn't necessary. And the existing DagBag in views.py already passed all the flags correctly :)

Regards,
Kaxil

On Mon, May 4, 2020, 10:30 Kaxil Naik <ka...@gmail.com>> wrote:
Hi Niebler,

Like Ash mentioned this is fixed for 1.10.11 already.

This was first reported in following Github issue:

- https://github.com/apache/airflow/issues/8247

And PR that fixes it for v1-10-test: https://github.com/apache/airflow/pull/8411

And https://github.com/apache/airflow/pull/8501 also fixes it in Master.

Regards,
Kaxil

On Mon, May 4, 2020, 09:21 Ash Berlin-Taylor <as...@apache.org>> wrote:
I think this was fixed by this PR - aiming for 1.10.11

https://github.com/apache/airflow/pull/8501

On 4 May 2020 07:41:55 BST, "Niebler Thomas (DC-IH/SDL1)" <Th...@boschrexroth.de.INVALID>> wrote:
>Hi all,
>
>I have a probably rather special use case scenario:
>Using Airflow 1.10.10, I would like to physically decouple the
>Webserver and the Scheduler for some secure access reasons. According
>to https://airflow.apache.org/docs/stable/dag-serialization.html, this
>should be a piece of cake with Airflow 1.10.10, since DAGs are stored
>in the Metadata database and the webserver does not need to access the
>DAG files anymore. The metadata database is of course reachable by both
>Airflow instances:
>
>Docker Image Instance: Airflow Webserver <----> Docker Image Instance:
>Metadata database <----> Physical Machine: Airflow Scheduler
>
>However, every time I start a DAG manually, it crashes with a rather
>lengthy error message:
>
>File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py",
>line 1255, in trigger
>    external_trigger=True
>File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line
>74, in wrapper
>    return func(*args, **kwargs)
>File "/usr/local/lib/python3.7/site-packages/airflow/models/dag.py",
>line 1818, in create_dagrun
>    return self.get_dag().create_dagrun(run_id=run_id,
>AttributeError: 'NoneType' object has no attribute 'create_dagrun'
>
>This basically boils down to self.get_dag() not having set the Boolean
>flag store_serialized_flags to True (or whatever value the config is
>set to), but always using False (the default value).
>This then leads to Airflow attempting to read the DAG file, ignoring
>the DAG database entry and returning None, which obviously has no
>attribute create_dagrun.
>
>I’ve got several questions here now:
>
>1.  Is my scenario even possible or am I overlooking something rather
>obvious?
>2.  Is the crashing DAG behavior intended like that? It rather seems
>like a bug to me.
>3.  Is it worth fixing this issue (if it is one) for Airflow 1.10.x,
>considering that Airflow 2.0.0 does not even contain the corresponding
>classes anymore and takes a different path?
>
>Mit freundlichen Grüßen / Best regards
>
>Dr. Thomas Niebler
>Data Scientist
>Sales Data Lab, Analytics DC-IH/SDL1
>
>Tel. +49 9352 18-2392
>Fax +49 9352 18-0
>Thomas.Niebler@boschrexroth.de<ma...@boschrexroth.de>>
>www.boschrexroth.com<http://www.boschrexroth.com>
>
>Bosch Rexroth AG
>Partensteiner Straße 23
>97816 Lohr am Main
>GERMANY
>
>[BOSCH REXROTH]<http://www.boschrexroth.com/>
>
>
>
>Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart HRB 23192
>Vorstand: Rolf Najork (Vorsitzender), Dr. Markus Forschner, Dr. Heiner
>Lang, Reinhard Schäfer, Dr. Marc Wucherer
>Vorsitzender des Aufsichtsrats: Christoph Kübel
>​

Re: Potential issue with serialized DAGs in decoupled Webserver and Scheduler

Posted by Kaxil Naik <ka...@gmail.com>.
And there was a wider issue, the trigger method didn't actually used the
DagBag already created for the Webserver but instead created a new one
which wasn't necessary. And the existing DagBag in views.py already passed
all the flags correctly :)

Regards,
Kaxil

On Mon, May 4, 2020, 10:30 Kaxil Naik <ka...@gmail.com> wrote:

> Hi Niebler,
>
> Like Ash mentioned this is fixed for 1.10.11 already.
>
> This was first reported in following Github issue:
>
> - https://github.com/apache/airflow/issues/8247
>
> And PR that fixes it for v1-10-test:
> https://github.com/apache/airflow/pull/8411
>
> And https://github.com/apache/airflow/pull/8501 also fixes it in Master.
>
> Regards,
> Kaxil
>
>
> On Mon, May 4, 2020, 09:21 Ash Berlin-Taylor <as...@apache.org> wrote:
>
>> I think this was fixed by this PR - aiming for 1.10.11
>>
>> https://github.com/apache/airflow/pull/8501
>>
>> On 4 May 2020 07:41:55 BST, "Niebler Thomas (DC-IH/SDL1)"
>> <Th...@boschrexroth.de.INVALID> wrote:
>> >Hi all,
>> >
>> >I have a probably rather special use case scenario:
>> >Using Airflow 1.10.10, I would like to physically decouple the
>> >Webserver and the Scheduler for some secure access reasons. According
>> >to https://airflow.apache.org/docs/stable/dag-serialization.html, this
>> >should be a piece of cake with Airflow 1.10.10, since DAGs are stored
>> >in the Metadata database and the webserver does not need to access the
>> >DAG files anymore. The metadata database is of course reachable by both
>> >Airflow instances:
>> >
>> >Docker Image Instance: Airflow Webserver <----> Docker Image Instance:
>> >Metadata database <----> Physical Machine: Airflow Scheduler
>> >
>> >However, every time I start a DAG manually, it crashes with a rather
>> >lengthy error message:
>> >
>> >File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py",
>> >line 1255, in trigger
>> >    external_trigger=True
>> >File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line
>> >74, in wrapper
>> >    return func(*args, **kwargs)
>> >File "/usr/local/lib/python3.7/site-packages/airflow/models/dag.py",
>> >line 1818, in create_dagrun
>> >    return self.get_dag().create_dagrun(run_id=run_id,
>> >AttributeError: 'NoneType' object has no attribute 'create_dagrun'
>> >
>> >This basically boils down to self.get_dag() not having set the Boolean
>> >flag store_serialized_flags to True (or whatever value the config is
>> >set to), but always using False (the default value).
>> >This then leads to Airflow attempting to read the DAG file, ignoring
>> >the DAG database entry and returning None, which obviously has no
>> >attribute create_dagrun.
>> >
>> >I’ve got several questions here now:
>> >
>> >1.  Is my scenario even possible or am I overlooking something rather
>> >obvious?
>> >2.  Is the crashing DAG behavior intended like that? It rather seems
>> >like a bug to me.
>> >3.  Is it worth fixing this issue (if it is one) for Airflow 1.10.x,
>> >considering that Airflow 2.0.0 does not even contain the corresponding
>> >classes anymore and takes a different path?
>> >
>> >Mit freundlichen Grüßen / Best regards
>> >
>> >Dr. Thomas Niebler
>> >Data Scientist
>> >Sales Data Lab, Analytics DC-IH/SDL1
>> >
>> >Tel. +49 9352 18-2392
>> >Fax +49 9352 18-0
>> >Thomas.Niebler@boschrexroth.de<ma...@boschrexroth.de>
>> >www.boschrexroth.com
>> >
>> >Bosch Rexroth AG
>> >Partensteiner Straße 23
>> >97816 Lohr am Main
>> >GERMANY
>> >
>> >[BOSCH REXROTH]<http://www.boschrexroth.com/>
>> >
>> >
>> >
>> >Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart HRB 23192
>> >Vorstand: Rolf Najork (Vorsitzender), Dr. Markus Forschner, Dr. Heiner
>> >Lang, Reinhard Schäfer, Dr. Marc Wucherer
>> >Vorsitzender des Aufsichtsrats: Christoph Kübel
>> >​
>>
>>

Re: Potential issue with serialized DAGs in decoupled Webserver and Scheduler

Posted by Kaxil Naik <ka...@gmail.com>.
Hi Niebler,

Like Ash mentioned this is fixed for 1.10.11 already.

This was first reported in following Github issue:

- https://github.com/apache/airflow/issues/8247

And PR that fixes it for v1-10-test:
https://github.com/apache/airflow/pull/8411

And https://github.com/apache/airflow/pull/8501 also fixes it in Master.

Regards,
Kaxil


On Mon, May 4, 2020, 09:21 Ash Berlin-Taylor <as...@apache.org> wrote:

> I think this was fixed by this PR - aiming for 1.10.11
>
> https://github.com/apache/airflow/pull/8501
>
> On 4 May 2020 07:41:55 BST, "Niebler Thomas (DC-IH/SDL1)"
> <Th...@boschrexroth.de.INVALID> wrote:
> >Hi all,
> >
> >I have a probably rather special use case scenario:
> >Using Airflow 1.10.10, I would like to physically decouple the
> >Webserver and the Scheduler for some secure access reasons. According
> >to https://airflow.apache.org/docs/stable/dag-serialization.html, this
> >should be a piece of cake with Airflow 1.10.10, since DAGs are stored
> >in the Metadata database and the webserver does not need to access the
> >DAG files anymore. The metadata database is of course reachable by both
> >Airflow instances:
> >
> >Docker Image Instance: Airflow Webserver <----> Docker Image Instance:
> >Metadata database <----> Physical Machine: Airflow Scheduler
> >
> >However, every time I start a DAG manually, it crashes with a rather
> >lengthy error message:
> >
> >File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py",
> >line 1255, in trigger
> >    external_trigger=True
> >File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line
> >74, in wrapper
> >    return func(*args, **kwargs)
> >File "/usr/local/lib/python3.7/site-packages/airflow/models/dag.py",
> >line 1818, in create_dagrun
> >    return self.get_dag().create_dagrun(run_id=run_id,
> >AttributeError: 'NoneType' object has no attribute 'create_dagrun'
> >
> >This basically boils down to self.get_dag() not having set the Boolean
> >flag store_serialized_flags to True (or whatever value the config is
> >set to), but always using False (the default value).
> >This then leads to Airflow attempting to read the DAG file, ignoring
> >the DAG database entry and returning None, which obviously has no
> >attribute create_dagrun.
> >
> >I’ve got several questions here now:
> >
> >1.  Is my scenario even possible or am I overlooking something rather
> >obvious?
> >2.  Is the crashing DAG behavior intended like that? It rather seems
> >like a bug to me.
> >3.  Is it worth fixing this issue (if it is one) for Airflow 1.10.x,
> >considering that Airflow 2.0.0 does not even contain the corresponding
> >classes anymore and takes a different path?
> >
> >Mit freundlichen Grüßen / Best regards
> >
> >Dr. Thomas Niebler
> >Data Scientist
> >Sales Data Lab, Analytics DC-IH/SDL1
> >
> >Tel. +49 9352 18-2392
> >Fax +49 9352 18-0
> >Thomas.Niebler@boschrexroth.de<ma...@boschrexroth.de>
> >www.boschrexroth.com
> >
> >Bosch Rexroth AG
> >Partensteiner Straße 23
> >97816 Lohr am Main
> >GERMANY
> >
> >[BOSCH REXROTH]<http://www.boschrexroth.com/>
> >
> >
> >
> >Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart HRB 23192
> >Vorstand: Rolf Najork (Vorsitzender), Dr. Markus Forschner, Dr. Heiner
> >Lang, Reinhard Schäfer, Dr. Marc Wucherer
> >Vorsitzender des Aufsichtsrats: Christoph Kübel
> >​
>
>

Re: Potential issue with serialized DAGs in decoupled Webserver and Scheduler

Posted by Ash Berlin-Taylor <as...@apache.org>.
I think this was fixed by this PR - aiming for 1.10.11

https://github.com/apache/airflow/pull/8501

On 4 May 2020 07:41:55 BST, "Niebler Thomas (DC-IH/SDL1)" <Th...@boschrexroth.de.INVALID> wrote:
>Hi all,
>
>I have a probably rather special use case scenario:
>Using Airflow 1.10.10, I would like to physically decouple the
>Webserver and the Scheduler for some secure access reasons. According
>to https://airflow.apache.org/docs/stable/dag-serialization.html, this
>should be a piece of cake with Airflow 1.10.10, since DAGs are stored
>in the Metadata database and the webserver does not need to access the
>DAG files anymore. The metadata database is of course reachable by both
>Airflow instances:
>
>Docker Image Instance: Airflow Webserver <----> Docker Image Instance:
>Metadata database <----> Physical Machine: Airflow Scheduler
>
>However, every time I start a DAG manually, it crashes with a rather
>lengthy error message:
>
>File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py",
>line 1255, in trigger
>    external_trigger=True
>File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line
>74, in wrapper
>    return func(*args, **kwargs)
>File "/usr/local/lib/python3.7/site-packages/airflow/models/dag.py",
>line 1818, in create_dagrun
>    return self.get_dag().create_dagrun(run_id=run_id,
>AttributeError: 'NoneType' object has no attribute 'create_dagrun'
>
>This basically boils down to self.get_dag() not having set the Boolean
>flag store_serialized_flags to True (or whatever value the config is
>set to), but always using False (the default value).
>This then leads to Airflow attempting to read the DAG file, ignoring
>the DAG database entry and returning None, which obviously has no
>attribute create_dagrun.
>
>I’ve got several questions here now:
>
>1.  Is my scenario even possible or am I overlooking something rather
>obvious?
>2.  Is the crashing DAG behavior intended like that? It rather seems
>like a bug to me.
>3.  Is it worth fixing this issue (if it is one) for Airflow 1.10.x,
>considering that Airflow 2.0.0 does not even contain the corresponding
>classes anymore and takes a different path?
>
>Mit freundlichen Grüßen / Best regards
>
>Dr. Thomas Niebler
>Data Scientist
>Sales Data Lab, Analytics DC-IH/SDL1
>
>Tel. +49 9352 18-2392
>Fax +49 9352 18-0
>Thomas.Niebler@boschrexroth.de<ma...@boschrexroth.de>
>www.boschrexroth.com
>
>Bosch Rexroth AG
>Partensteiner Straße 23
>97816 Lohr am Main
>GERMANY
>
>[BOSCH REXROTH]<http://www.boschrexroth.com/>
>
>
>
>Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart HRB 23192
>Vorstand: Rolf Najork (Vorsitzender), Dr. Markus Forschner, Dr. Heiner
>Lang, Reinhard Schäfer, Dr. Marc Wucherer
>Vorsitzender des Aufsichtsrats: Christoph Kübel
>​

AW: Potential issue with serialized DAGs in decoupled Webserver and Scheduler

Posted by "Niebler Thomas (DC-IH/SDL1)" <Th...@boschrexroth.de.INVALID>.
Hi Anita,

Thank you, that spares me some time as I hadn't tested my fixes yet... :-X

Mit freundlichen Grüßen / Best regards

Dr. Thomas Niebler
Data Scientist<br/>Sales Data Lab, Analytics DC-IH/SDL1

Tel. +49 9352 18-2392
Fax +49 9352 18-0
Thomas.Niebler@boschrexroth.de
www.boschrexroth.com

Bosch Rexroth AG
Partensteiner Straße 23
97816 Lohr am Main
GERMANY

Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart HRB 23192
Vorstand: Rolf Najork (Vorsitzender), Dr. Markus Forschner, Dr. Heiner Lang, Reinhard Schäfer, Dr. Marc Wucherer
Vorsitzender des Aufsichtsrats: Christoph Kübel

-----Ursprüngliche Nachricht-----
Von: Anita Fronczak <ak...@gmail.com> 
Gesendet: Monday, May 4, 2020 9:15 AM
An: dev@airflow.apache.org
Betreff: Re: Potential issue with serialized DAGs in decoupled Webserver and Scheduler

Hello,

Your scenario is possible. I have a ready PR that sets store_serialized_dags in all places to appropriate value (web server, CLI, experimental API) and then dag serialization works without access to DAG files (tested). I shall push the PR today.

Anita

pon., 4 maj 2020, 08:42 użytkownik Niebler Thomas (DC-IH/SDL1) <Th...@boschrexroth.de.invalid> napisał:

> Hi all,
>
> I have a probably rather special use case scenario:
> Using Airflow 1.10.10, I would like to physically decouple the 
> Webserver and the Scheduler for some secure access reasons. According 
> to https://airflow.apache.org/docs/stable/dag-serialization.html, this 
> should be a piece of cake with Airflow 1.10.10, since DAGs are stored 
> in the Metadata database and the webserver does not need to access the 
> DAG files anymore. The metadata database is of course reachable by 
> both Airflow
> instances:
>
> Docker Image Instance: Airflow Webserver <----> Docker Image Instance:
> Metadata database <----> Physical Machine: Airflow Scheduler
>
> However, every time I start a DAG manually, it crashes with a rather 
> lengthy error message:
>
>   File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", 
> line 1255, in trigger
>     external_trigger=True
>   File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", 
> line 74, in wrapper
>     return func(*args, **kwargs)
>   File "/usr/local/lib/python3.7/site-packages/airflow/models/dag.py",
> line 1818, in create_dagrun
>     return self.get_dag().create_dagrun(run_id=run_id,
> AttributeError: 'NoneType' object has no attribute 'create_dagrun'
>
> This basically boils down to self.get_dag() not having set the Boolean 
> flag store_serialized_flags to True (or whatever value the config is 
> set to), but always using False (the default value).
> This then leads to Airflow attempting to read the DAG file, ignoring 
> the DAG database entry and returning None, which obviously has no 
> attribute create_dagrun.
>
> I’ve got several questions here now:
>
>   1.  Is my scenario even possible or am I overlooking something 
> rather obvious?
>   2.  Is the crashing DAG behavior intended like that? It rather seems 
> like a bug to me.
>   3.  Is it worth fixing this issue (if it is one) for Airflow 1.10.x, 
> considering that Airflow 2.0.0 does not even contain the corresponding 
> classes anymore and takes a different path?
>
> Mit freundlichen Grüßen / Best regards
>
> Dr. Thomas Niebler
> Data Scientist
> Sales Data Lab, Analytics DC-IH/SDL1
>
> Tel. +49 9352 18-2392
> Fax +49 9352 18-0
> Thomas.Niebler@boschrexroth.de<ma...@boschrexroth.de>
> www.boschrexroth.com
>
> Bosch Rexroth AG
> Partensteiner Straße 23
> 97816 Lohr am Main
> GERMANY
>
> [BOSCH REXROTH]<http://www.boschrexroth.com/>
>
>
>
> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart HRB 23192
> Vorstand: Rolf Najork (Vorsitzender), Dr. Markus Forschner, Dr. Heiner 
> Lang, Reinhard Schäfer, Dr. Marc Wucherer Vorsitzender des 
> Aufsichtsrats: Christoph Kübel ​
>

Re: Potential issue with serialized DAGs in decoupled Webserver and Scheduler

Posted by Anita Fronczak <ak...@gmail.com>.
Hello,

Your scenario is possible. I have a ready PR that sets
store_serialized_dags in all places to appropriate value (web server, CLI,
experimental API) and then dag serialization works without access to DAG
files (tested). I shall push the PR today.

Anita

pon., 4 maj 2020, 08:42 użytkownik Niebler Thomas (DC-IH/SDL1)
<Th...@boschrexroth.de.invalid> napisał:

> Hi all,
>
> I have a probably rather special use case scenario:
> Using Airflow 1.10.10, I would like to physically decouple the Webserver
> and the Scheduler for some secure access reasons. According to
> https://airflow.apache.org/docs/stable/dag-serialization.html, this
> should be a piece of cake with Airflow 1.10.10, since DAGs are stored in
> the Metadata database and the webserver does not need to access the DAG
> files anymore. The metadata database is of course reachable by both Airflow
> instances:
>
> Docker Image Instance: Airflow Webserver <----> Docker Image Instance:
> Metadata database <----> Physical Machine: Airflow Scheduler
>
> However, every time I start a DAG manually, it crashes with a rather
> lengthy error message:
>
>   File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line
> 1255, in trigger
>     external_trigger=True
>   File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line
> 74, in wrapper
>     return func(*args, **kwargs)
>   File "/usr/local/lib/python3.7/site-packages/airflow/models/dag.py",
> line 1818, in create_dagrun
>     return self.get_dag().create_dagrun(run_id=run_id,
> AttributeError: 'NoneType' object has no attribute 'create_dagrun'
>
> This basically boils down to self.get_dag() not having set the Boolean
> flag store_serialized_flags to True (or whatever value the config is set
> to), but always using False (the default value).
> This then leads to Airflow attempting to read the DAG file, ignoring the
> DAG database entry and returning None, which obviously has no attribute
> create_dagrun.
>
> I’ve got several questions here now:
>
>   1.  Is my scenario even possible or am I overlooking something rather
> obvious?
>   2.  Is the crashing DAG behavior intended like that? It rather seems
> like a bug to me.
>   3.  Is it worth fixing this issue (if it is one) for Airflow 1.10.x,
> considering that Airflow 2.0.0 does not even contain the corresponding
> classes anymore and takes a different path?
>
> Mit freundlichen Grüßen / Best regards
>
> Dr. Thomas Niebler
> Data Scientist
> Sales Data Lab, Analytics DC-IH/SDL1
>
> Tel. +49 9352 18-2392
> Fax +49 9352 18-0
> Thomas.Niebler@boschrexroth.de<ma...@boschrexroth.de>
> www.boschrexroth.com
>
> Bosch Rexroth AG
> Partensteiner Straße 23
> 97816 Lohr am Main
> GERMANY
>
> [BOSCH REXROTH]<http://www.boschrexroth.com/>
>
>
>
> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart HRB 23192
> Vorstand: Rolf Najork (Vorsitzender), Dr. Markus Forschner, Dr. Heiner
> Lang, Reinhard Schäfer, Dr. Marc Wucherer
> Vorsitzender des Aufsichtsrats: Christoph Kübel
> ​
>