[DISCUSS] Change all example_dags's start_date to specific datetime

[Jiajie Zhong <zh...@hotmail.com>]

Hi Airflowers:
    I start this thread proposal to change all example_dags's start_date to specific datetime, including airflow/example_dag and airflow/contrib/example_dags.
I proposal that because in https://airflow.apache.org/faq.html#what-s-the-deal-with-start-date

> We recommend against using dynamic values as start_date, especially datetime.now() as it can be quite confusing. The task is triggered once the period closes, and in theory an @hourly DAG would never get to an hour after now as now() moves along.

we suggest to set start_date to specific datetime, but all example_dags in code base are use dynamic values like `airflow.utils.dates.days_ago(2)` or `datetime.utcnow()`. Most of Airflow starter will use example dag as they template to create their own dag, so I think should change them in correct values.

But if we use a specific datetime in example dag, for example 2019-01-01, starter will use them as template and confuse "Why my dag start run in 2019-01-01?".

So, should we change start_date?

Best wish.
-- Jiajie

Re: [DISCUSS] Change all example_dags's start_date to specific datetime

[Jiajie Zhong <zh...@hotmail.com>]

I know catchup_by=true by default, but I think when user run example dag, their just want to test how Airflow run, and thier would stop it manually.

But if user use start_date=airflow.utils.dates.days_ago(2) as their production dag would cause unexpected behavior.

IMO, if we use fixed date in example dag, dag would backfill, but user could stop it manually, and their would know that "startdate control when dag start", and user will use fixed date to their new dag(template from example dag)

Best wish.
-- Jiajie
________________________________
From: Deng Xiaodong <xd...@gmail.com>
Sent: Friday, May 10, 2019 15:50
To: dev@airflow.apache.org
Subject: Re: [DISCUSS] Change all example_dags's start_date to specific datetime

Please note that catchup_by_default is True by default. Imagine what will
happen if you set these start_date to fixed dates ;-)


XD

On Fri, May 10, 2019 at 3:34 PM Jiajie Zhong <zh...@hotmail.com>
wrote:

> Hi Airflowers:
>     I start this thread proposal to change all example_dags's start_date
> to specific datetime, including airflow/example_dag and
> airflow/contrib/example_dags.
> I proposal that because in
> https://airflow.apache.org/faq.html#what-s-the-deal-with-start-date
>
> > We recommend against using dynamic values as start_date, especially
> datetime.now() as it can be quite confusing. The task is triggered once the
> period closes, and in theory an @hourly DAG would never get to an hour
> after now as now() moves along.
>
> we suggest to set start_date to specific datetime, but all example_dags in
> code base are use dynamic values like `airflow.utils.dates.days_ago(2)` or
> `datetime.utcnow()`. Most of Airflow starter will use example dag as they
> template to create their own dag, so I think should change them in correct
> values.
>
> But if we use a specific datetime in example dag, for example 2019-01-01,
> starter will use them as template and confuse "Why my dag start run in
> 2019-01-01?".
>
> So, should we change start_date?
>
> Best wish.
> -- Jiajie
>

Re: [DISCUSS] Change all example_dags's start_date to specific datetime

[Deng Xiaodong <xd...@gmail.com>]

Please note that catchup_by_default is True by default. Imagine what will
happen if you set these start_date to fixed dates ;-)


XD

On Fri, May 10, 2019 at 3:34 PM Jiajie Zhong <zh...@hotmail.com>
wrote:

> Hi Airflowers:
>     I start this thread proposal to change all example_dags's start_date
> to specific datetime, including airflow/example_dag and
> airflow/contrib/example_dags.
> I proposal that because in
> https://airflow.apache.org/faq.html#what-s-the-deal-with-start-date
>
> > We recommend against using dynamic values as start_date, especially
> datetime.now() as it can be quite confusing. The task is triggered once the
> period closes, and in theory an @hourly DAG would never get to an hour
> after now as now() moves along.
>
> we suggest to set start_date to specific datetime, but all example_dags in
> code base are use dynamic values like `airflow.utils.dates.days_ago(2)` or
> `datetime.utcnow()`. Most of Airflow starter will use example dag as they
> template to create their own dag, so I think should change them in correct
> values.
>
> But if we use a specific datetime in example dag, for example 2019-01-01,
> starter will use them as template and confuse "Why my dag start run in
> 2019-01-01?".
>
> So, should we change start_date?
>
> Best wish.
> -- Jiajie
>