You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jeremiah Lowin <jl...@apache.org> on 2016/08/03 18:08:37 UTC

A question for the Airflow community

Dear all --

Thanks for being part of the Airflow community! We are all deeply
appreciative of your involvement in this project, your willingness to ask
questions, point out problems, and (especially) build solutions.

I'd like to ask you all a question in turn: what do you know now that you
wish you knew when you first deployed Airflow? In other words, what made
you do this: https://www.youtube.com/watch?v=gqQ99s4Ywnw

This is intentionally very open-ended and I hope many of you will respond,
even if it's just a few words on a pain point you encountered that wasn't
quite big enough to deserve a JIRA issue (but please note if you don't open
an issue, it's not getting fixed ;) Here's the link if you need it:
https://issues.apache.org/jira/browse/AIRFLOW/).

We will use your feedback to figure out not only what could be better
communicated, but also the best format for doing so. We want to make it
easier to mold Airflow to every user's unique environment. That might
involve improving the documentation; making better use of the wiki to store
knowledge; or creating a blog to present more narrative-driven examples,
use cases, and news.

Thanks so much!
-- Jeremiah (on behalf of the Airflow team)

Re: A question for the Airflow community

Posted by Andrew Phillips <an...@apache.org>.
> 2. There is a feature 'XCom' which allows you to use the Airflow 
> database
> as a key-value store. If you wish DAG and task instances to see 
> particular
> data items, you can store them via 'XCom'. I have not done this.

Thanks, Lance! An XCom is indeed what we are using now, as also 
discussed in [2]. We were hesitant to use XComs, though, because of the 
following statement in the docs, which to us made it seem like XComs 
should really be avoid if at all possible:

"If it absolutely can\u2019t be avoided, Airflow does have a feature for
operator cross-communication called XCom that is described elsewhere in
this document."

Some clarification of "how bad" XComs are, under what circumstances to 
use them, what to put in them etc. would be helpful, I think.

Regards

ap

[2] http://markmail.org/message/5jwz7c64ndnhzqyq

Re: A question for the Airflow community

Posted by Lance Norskog <la...@gmail.com>.
2. There is a feature 'XCom' which allows you to use the Airflow database
as a key-value store. If you wish DAG and task instances to see particular
data items, you can store them via 'XCom'. I have not done this.



On Fri, Aug 5, 2016 at 7:25 PM, Andrew Phillips <an...@apache.org> wrote:

> I'd like to ask you all a question in turn: what do you
>> know now that you wish you knew when you first deployed
>> Airflow?
>>
>
> Mainly questions around suggestions/best practice for addressing
> reasonably common (?) integration challenges:
>
> 1. Talking to Airflow from remote/external systems (e.g. on a cloud CI/CD
> service) where the Airflow CLI is not available and/or it is not desirable
> to expose access to the DB directly.
>
> An HTTP API, as already on the roadmap, as far as I know, would be an
> obvious candidate here, I guess, but how are people doing this currently?
> (Cf. [1])
>
> 2. Passing parameters to DAG runs beyond the run ID that remain stable
> across the DAG run. Cf. [2]
>
> A description somewhere of what happens when a DAG run actually takes
> place (when and where is the DAG object instantiated, when are task
> instances instantiated, when does template resolution happen, etc.) would
> have helped us to avoid running into gotchas based on mistaken assumptions.
>
> For example, we had initially assumed that the DAG object would be created
> *once*, at the beginning of the DAG run, and that any parameters passed to
> it would be "frozen" at that time.
>
> As far as we can tell, the DAG object is actually instantiated repeatedly,
> so e.g. the value passed as the params argument is not "frozen" at the
> beginning of the DAG run, and later task instances may see different values
> from earlier task instances.
>
> Thanks for kicking off this initiative!
>
> Regards
>
> ap
>
> [1] http://markmail.org/message/3yowkhufo7hegzoo
> [2] http://markmail.org/message/5jwz7c64ndnhzqyq
>



-- 
Lance Norskog
lance.norskog@gmail.com
Redwood City, CA

Re: Restarting the scheduler regularly - still current advice?

Posted by Bolke de Bruin <bd...@gmail.com>.
Pools are an outstanding issue, which I am working on but won’t be part of the Apache release. The Apache release will initially just be 1.7.1.3 + Licenses - Highcharts + D3.

- B.

> Op 9 aug. 2016, om 21:44 heeft Lance Norskog <la...@gmail.com> het volgende geschreven:
> 
> We are on 1.6.2 and would love to upgrade to a modern version. We were
> holding out for the first Apache release.
> 
> Also, we have cases where the various concurrent task limits are ignored
> and we have 50 tasks scheduled at once. A DAG like this:
> 
> dag = DAG(
>    dag_id='xxx', x
>    schedule_interval="0 "+str(hour)+" * * *",
>    max_active_runs=1,
>    concurrency=1
>    )
> 
> and where all of the tasks use the same Pool. max_active_runs= is ignored,
> concurrency= is ignored, the Pool is ignored.
> 
> 
> On Tue, Aug 9, 2016 at 12:08 PM, Bolke de Bruin <bd...@gmail.com> wrote:
> 
>> I disagree. Num_runs should NOT be used anymore and I would really like to
>> know ‘stuck’ schedulers on release or on master, preferably with celery
>> executor (LocalExecutor can sometimes look stuck but isn’t). Restarting
>> should only be required for clearing up database connections as we are not
>> very good at that yet.
>> 
>> - Bolke
>> 
>>> Op 9 aug. 2016, om 20:30 heeft Lance Norskog <la...@gmail.com>
>> het volgende geschreven:
>>> 
>>> Yes, it is still current advice.
>>> 
>>> My experience is that after running for (let's say) days, the app
>> develops
>>> memory corruption. I've seen three different ways that memory corruption
>>> shows up. The scheduler failure is just one of these three symptoms.
>>> 
>>> The other two symptoms are
>>> 1) the main page of the UI shows a different list of running DAGs than is
>>> what is really configured,
>>> 2) a task contains some configuration data that should be in a
>> neighboring
>>> task, and fails.
>>> 
>>> Frankly, I would configure all 5 daemons to restart periodically, not
>> just
>>> the scheduler daemon.
>>> 
>>> 
>>> On Tue, Aug 9, 2016 at 8:50 AM, Andrew Phillips <an...@apache.org>
>> wrote:
>>> 
>>>> Hi all
>>>> 
>>>> I just wanted to check to what extent the advice in [1] and [2], namely
>> to
>>>> restart the scheduler "every once in a while", is still considered
>> accurate?
>>>> 
>>>> "Restart your scheduler process to get a clean environment every once
>> in a
>>>> while. Use --num_runs N scheduler CLI option to make it stop after N
>> runs
>>>> and have some supervisor ensuring it is always running. See issue 698"
>>>> 
>>>> "The scheduler should be restarted frequently
>>>> 
>>>> In our experience, a long running scheduler process, at least with the
>>>> CeleryExecutor, ends up not scheduling some tasks. We still don’t know
>> the
>>>> exact cause, unfortunately.
>>>> 
>>>> Fortunately, airflow has a built-in workaround in the form of the —
>>>> num_runs flag. It specifies a number of iterations for the scheduler to
>> run
>>>> of its loop before it quits. We’re running it with 10 iterations, Airbnb
>>>> runs it with 5. Note that this will cause problems when using the
>>>> LocalExecutor."
>>>> 
>>>> Both documents are pretty now, so I assume this is considered still
>>>> relevant. Could you give some guidance on what kind of frequency is
>>>> recommended here, or is that very dependent on the particular
>> installation?
>>>> 
>>>> Also, which of the current JIRA issues (if any) is the new version of
>>>> "issue 698" as mentioned in the first quote? There seem to be quite a
>> few
>>>> issues relating to the scheduler getting stuck [3] - which one(s)
>> should we
>>>> follow and/or add information to to best track progress on this topic?
>>>> 
>>>> Thanks!
>>>> 
>>>> ap
>>>> 
>>>> 
>>>> [1] https://cwiki.apache.org/confluence/display/AIRFLOW/Common+Pitfalls
>>>> [2] https://medium.com/handy-tech/airflow-tips-tricks-and-pitfal
>>>> ls-9ba53fba14eb#.ahcprdr9r
>>>> [3] https://issues.apache.org/jira/browse/AIRFLOW-39?jql=project
>>>> %20%3D%20AIRFLOW%20AND%20text%20~%20%22scheduler%22
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Lance Norskog
>>> lance.norskog@gmail.com
>>> Redwood City, CA
>> 
>> 
> 
> 
> -- 
> Lance Norskog
> lance.norskog@gmail.com
> Redwood City, CA

Re: Restarting the scheduler regularly - still current advice?

Posted by Lance Norskog <la...@gmail.com>.
We are on 1.6.2 and would love to upgrade to a modern version. We were
holding out for the first Apache release.

Also, we have cases where the various concurrent task limits are ignored
and we have 50 tasks scheduled at once. A DAG like this:

dag = DAG(
    dag_id='xxx', x
    schedule_interval="0 "+str(hour)+" * * *",
    max_active_runs=1,
    concurrency=1
    )

and where all of the tasks use the same Pool. max_active_runs= is ignored,
concurrency= is ignored, the Pool is ignored.


On Tue, Aug 9, 2016 at 12:08 PM, Bolke de Bruin <bd...@gmail.com> wrote:

> I disagree. Num_runs should NOT be used anymore and I would really like to
> know ‘stuck’ schedulers on release or on master, preferably with celery
> executor (LocalExecutor can sometimes look stuck but isn’t). Restarting
> should only be required for clearing up database connections as we are not
> very good at that yet.
>
> - Bolke
>
> > Op 9 aug. 2016, om 20:30 heeft Lance Norskog <la...@gmail.com>
> het volgende geschreven:
> >
> > Yes, it is still current advice.
> >
> > My experience is that after running for (let's say) days, the app
> develops
> > memory corruption. I've seen three different ways that memory corruption
> > shows up. The scheduler failure is just one of these three symptoms.
> >
> > The other two symptoms are
> > 1) the main page of the UI shows a different list of running DAGs than is
> > what is really configured,
> > 2) a task contains some configuration data that should be in a
> neighboring
> > task, and fails.
> >
> > Frankly, I would configure all 5 daemons to restart periodically, not
> just
> > the scheduler daemon.
> >
> >
> > On Tue, Aug 9, 2016 at 8:50 AM, Andrew Phillips <an...@apache.org>
> wrote:
> >
> >> Hi all
> >>
> >> I just wanted to check to what extent the advice in [1] and [2], namely
> to
> >> restart the scheduler "every once in a while", is still considered
> accurate?
> >>
> >> "Restart your scheduler process to get a clean environment every once
> in a
> >> while. Use --num_runs N scheduler CLI option to make it stop after N
> runs
> >> and have some supervisor ensuring it is always running. See issue 698"
> >>
> >> "The scheduler should be restarted frequently
> >>
> >> In our experience, a long running scheduler process, at least with the
> >> CeleryExecutor, ends up not scheduling some tasks. We still don’t know
> the
> >> exact cause, unfortunately.
> >>
> >> Fortunately, airflow has a built-in workaround in the form of the —
> >> num_runs flag. It specifies a number of iterations for the scheduler to
> run
> >> of its loop before it quits. We’re running it with 10 iterations, Airbnb
> >> runs it with 5. Note that this will cause problems when using the
> >> LocalExecutor."
> >>
> >> Both documents are pretty now, so I assume this is considered still
> >> relevant. Could you give some guidance on what kind of frequency is
> >> recommended here, or is that very dependent on the particular
> installation?
> >>
> >> Also, which of the current JIRA issues (if any) is the new version of
> >> "issue 698" as mentioned in the first quote? There seem to be quite a
> few
> >> issues relating to the scheduler getting stuck [3] - which one(s)
> should we
> >> follow and/or add information to to best track progress on this topic?
> >>
> >> Thanks!
> >>
> >> ap
> >>
> >>
> >> [1] https://cwiki.apache.org/confluence/display/AIRFLOW/Common+Pitfalls
> >> [2] https://medium.com/handy-tech/airflow-tips-tricks-and-pitfal
> >> ls-9ba53fba14eb#.ahcprdr9r
> >> [3] https://issues.apache.org/jira/browse/AIRFLOW-39?jql=project
> >> %20%3D%20AIRFLOW%20AND%20text%20~%20%22scheduler%22
> >>
> >
> >
> >
> > --
> > Lance Norskog
> > lance.norskog@gmail.com
> > Redwood City, CA
>
>


-- 
Lance Norskog
lance.norskog@gmail.com
Redwood City, CA

Re: Restarting the scheduler regularly - still current advice?

Posted by Bolke de Bruin <bd...@gmail.com>.
I disagree. Num_runs should NOT be used anymore and I would really like to know ‘stuck’ schedulers on release or on master, preferably with celery executor (LocalExecutor can sometimes look stuck but isn’t). Restarting should only be required for clearing up database connections as we are not very good at that yet.

- Bolke

> Op 9 aug. 2016, om 20:30 heeft Lance Norskog <la...@gmail.com> het volgende geschreven:
> 
> Yes, it is still current advice.
> 
> My experience is that after running for (let's say) days, the app develops
> memory corruption. I've seen three different ways that memory corruption
> shows up. The scheduler failure is just one of these three symptoms.
> 
> The other two symptoms are
> 1) the main page of the UI shows a different list of running DAGs than is
> what is really configured,
> 2) a task contains some configuration data that should be in a neighboring
> task, and fails.
> 
> Frankly, I would configure all 5 daemons to restart periodically, not just
> the scheduler daemon.
> 
> 
> On Tue, Aug 9, 2016 at 8:50 AM, Andrew Phillips <an...@apache.org> wrote:
> 
>> Hi all
>> 
>> I just wanted to check to what extent the advice in [1] and [2], namely to
>> restart the scheduler "every once in a while", is still considered accurate?
>> 
>> "Restart your scheduler process to get a clean environment every once in a
>> while. Use --num_runs N scheduler CLI option to make it stop after N runs
>> and have some supervisor ensuring it is always running. See issue 698"
>> 
>> "The scheduler should be restarted frequently
>> 
>> In our experience, a long running scheduler process, at least with the
>> CeleryExecutor, ends up not scheduling some tasks. We still don’t know the
>> exact cause, unfortunately.
>> 
>> Fortunately, airflow has a built-in workaround in the form of the —
>> num_runs flag. It specifies a number of iterations for the scheduler to run
>> of its loop before it quits. We’re running it with 10 iterations, Airbnb
>> runs it with 5. Note that this will cause problems when using the
>> LocalExecutor."
>> 
>> Both documents are pretty now, so I assume this is considered still
>> relevant. Could you give some guidance on what kind of frequency is
>> recommended here, or is that very dependent on the particular installation?
>> 
>> Also, which of the current JIRA issues (if any) is the new version of
>> "issue 698" as mentioned in the first quote? There seem to be quite a few
>> issues relating to the scheduler getting stuck [3] - which one(s) should we
>> follow and/or add information to to best track progress on this topic?
>> 
>> Thanks!
>> 
>> ap
>> 
>> 
>> [1] https://cwiki.apache.org/confluence/display/AIRFLOW/Common+Pitfalls
>> [2] https://medium.com/handy-tech/airflow-tips-tricks-and-pitfal
>> ls-9ba53fba14eb#.ahcprdr9r
>> [3] https://issues.apache.org/jira/browse/AIRFLOW-39?jql=project
>> %20%3D%20AIRFLOW%20AND%20text%20~%20%22scheduler%22
>> 
> 
> 
> 
> -- 
> Lance Norskog
> lance.norskog@gmail.com
> Redwood City, CA

Re: Restarting the scheduler regularly - still current advice?

Posted by Lance Norskog <la...@gmail.com>.
Yes, it is still current advice.

My experience is that after running for (let's say) days, the app develops
memory corruption. I've seen three different ways that memory corruption
shows up. The scheduler failure is just one of these three symptoms.

The other two symptoms are
1) the main page of the UI shows a different list of running DAGs than is
what is really configured,
2) a task contains some configuration data that should be in a neighboring
task, and fails.

Frankly, I would configure all 5 daemons to restart periodically, not just
the scheduler daemon.


On Tue, Aug 9, 2016 at 8:50 AM, Andrew Phillips <an...@apache.org> wrote:

> Hi all
>
> I just wanted to check to what extent the advice in [1] and [2], namely to
> restart the scheduler "every once in a while", is still considered accurate?
>
> "Restart your scheduler process to get a clean environment every once in a
> while. Use --num_runs N scheduler CLI option to make it stop after N runs
> and have some supervisor ensuring it is always running. See issue 698"
>
> "The scheduler should be restarted frequently
>
> In our experience, a long running scheduler process, at least with the
> CeleryExecutor, ends up not scheduling some tasks. We still don’t know the
> exact cause, unfortunately.
>
> Fortunately, airflow has a built-in workaround in the form of the —
> num_runs flag. It specifies a number of iterations for the scheduler to run
> of its loop before it quits. We’re running it with 10 iterations, Airbnb
> runs it with 5. Note that this will cause problems when using the
> LocalExecutor."
>
> Both documents are pretty now, so I assume this is considered still
> relevant. Could you give some guidance on what kind of frequency is
> recommended here, or is that very dependent on the particular installation?
>
> Also, which of the current JIRA issues (if any) is the new version of
> "issue 698" as mentioned in the first quote? There seem to be quite a few
> issues relating to the scheduler getting stuck [3] - which one(s) should we
> follow and/or add information to to best track progress on this topic?
>
> Thanks!
>
> ap
>
>
> [1] https://cwiki.apache.org/confluence/display/AIRFLOW/Common+Pitfalls
> [2] https://medium.com/handy-tech/airflow-tips-tricks-and-pitfal
> ls-9ba53fba14eb#.ahcprdr9r
> [3] https://issues.apache.org/jira/browse/AIRFLOW-39?jql=project
> %20%3D%20AIRFLOW%20AND%20text%20~%20%22scheduler%22
>



-- 
Lance Norskog
lance.norskog@gmail.com
Redwood City, CA

Restarting the scheduler regularly - still current advice?

Posted by Andrew Phillips <an...@apache.org>.
Hi all

I just wanted to check to what extent the advice in [1] and [2], namely 
to restart the scheduler "every once in a while", is still considered 
accurate?

"Restart your scheduler process to get a clean environment every once in 
a while. Use --num_runs N scheduler CLI option to make it stop after N 
runs and have some supervisor ensuring it is always running. See issue 
698"

"The scheduler should be restarted frequently

In our experience, a long running scheduler process, at least with the 
CeleryExecutor, ends up not scheduling some tasks. We still don\u2019t know 
the exact cause, unfortunately.

Fortunately, airflow has a built-in workaround in the form of 
the\u200a\u2014\u200anum_runs flag. It specifies a number of iterations for the 
scheduler to run of its loop before it quits. We\u2019re running it with 10 
iterations, Airbnb runs it with 5. Note that this will cause problems 
when using the LocalExecutor."

Both documents are pretty now, so I assume this is considered still 
relevant. Could you give some guidance on what kind of frequency is 
recommended here, or is that very dependent on the particular 
installation?

Also, which of the current JIRA issues (if any) is the new version of 
"issue 698" as mentioned in the first quote? There seem to be quite a 
few issues relating to the scheduler getting stuck [3] - which one(s) 
should we follow and/or add information to to best track progress on 
this topic?

Thanks!

ap


[1] https://cwiki.apache.org/confluence/display/AIRFLOW/Common+Pitfalls
[2] 
https://medium.com/handy-tech/airflow-tips-tricks-and-pitfalls-9ba53fba14eb#.ahcprdr9r
[3] 
https://issues.apache.org/jira/browse/AIRFLOW-39?jql=project%20%3D%20AIRFLOW%20AND%20text%20~%20%22scheduler%22

Re: A question for the Airflow community

Posted by Andrew Phillips <an...@apache.org>.
> I'd like to ask you all a question in turn: what do you
> know now that you wish you knew when you first deployed
> Airflow?

Mainly questions around suggestions/best practice for addressing 
reasonably common (?) integration challenges:

1. Talking to Airflow from remote/external systems (e.g. on a cloud 
CI/CD service) where the Airflow CLI is not available and/or it is not 
desirable to expose access to the DB directly.

An HTTP API, as already on the roadmap, as far as I know, would be an 
obvious candidate here, I guess, but how are people doing this 
currently? (Cf. [1])

2. Passing parameters to DAG runs beyond the run ID that remain stable 
across the DAG run. Cf. [2]

A description somewhere of what happens when a DAG run actually takes 
place (when and where is the DAG object instantiated, when are task 
instances instantiated, when does template resolution happen, etc.) 
would have helped us to avoid running into gotchas based on mistaken 
assumptions.

For example, we had initially assumed that the DAG object would be 
created *once*, at the beginning of the DAG run, and that any parameters 
passed to it would be "frozen" at that time.

As far as we can tell, the DAG object is actually instantiated 
repeatedly, so e.g. the value passed as the params argument is not 
"frozen" at the beginning of the DAG run, and later task instances may 
see different values from earlier task instances.

Thanks for kicking off this initiative!

Regards

ap

[1] http://markmail.org/message/3yowkhufo7hegzoo
[2] http://markmail.org/message/5jwz7c64ndnhzqyq