You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jiahao Chen <jh...@google.com.INVALID> on 2019/05/13 18:08:15 UTC
Cron schedule with DST-aware timezone
Hi team,
I have a question about the expected behavior of Airflow scheduler when the
schedule_interval is a cron expression and the start_date is in a timezone
with DST.
Based on the Airflow documentation
https://airflow.apache.org/timezone.html#cron-schedules, the DST change
will be ignored if schedule_interval is a cron expresion (e.g. '0 17 * *
*'). And it gives an example that the GMT offset will not change regardless
how DST changes. If I'm understanding it correctly, that means if I upload
a DAG with a schedule_interval of "0 17 * * *" and a start_date of
2019-03-15 17:00 PST(GMT-8) which is before the DST change on March 10, the
Airflow scheduler will always start the DAG on 5 pm everyday GMT-8 even
after the DST change on March 10.
However, that is not the behavior I've seen with my experimental code (see
attachments). It looks like the the Airflow is actually taking the DST into
account, since the execution time is always 17:00 locally, which is 1 hour
off on the GMT after the DST change.
Could you please confirm the behavior of Airflow scheduler in this use case?
Thank you!
Jiahao
RE: Cron schedule with DST-aware timezone
Posted by Bolke de Bruin <bd...@gmail.com>.
It is assumed that you want to have cron always at the exact *local time*.
So in your case always 17.00h. I see that indeed the documentation is not
correct here. It actually observes DST.
Cheers
Bolke
On 13 May 2019 at 21:38:04, Shaw, Damian P. (damian.shaw.2@credit-suisse.com)
wrote:
Hi,
I’m not part of the Airflow team but I came to the same conclusion, that
the behavior is the opposite of what the documentation specifies. When
using the cron notation then DST is properly honored, when using timedelta
it is not. I played around with the DAG.following_schedule method to
satisfy this was the case.
I’ve had production instance of airflow based on this that successfully
respected the March/April DST changes in many timezones.
Regards
Damian
From: Jiahao Chen [mailto:jhchen@google.com.INVALID]
Sent: Monday, May 13, 2019 2:08 PM
To: dev@airflow.apache.org
Subject: Cron schedule with DST-aware timezone
Hi team,
I have a question about the expected behavior of Airflow scheduler when the
schedule_interval is a cron expression and the start_date is in a timezone
with DST.
Based on the Airflow documentation
https://airflow.apache.org/timezone.html#cron-schedules, the DST change
will be ignored if schedule_interval is a cron expresion (e.g. '0 17 * *
*'). And it gives an example that the GMT offset will not change regardless
how DST changes. If I'm understanding it correctly, that means if I upload
a DAG with a schedule_interval of "0 17 * * *" and a start_date of
2019-03-15 17:00 PST(GMT-8) which is before the DST change on March 10, the
Airflow scheduler will always start the DAG on 5 pm everyday GMT-8 even
after the DST change on March 10.
However, that is not the behavior I've seen with my experimental code (see
attachments). It looks like the the Airflow is actually taking the DST into
account, since the execution time is always 17:00 locally, which is 1 hour
off on the GMT after the DST change.
Could you please confirm the behavior of Airflow scheduler in this use
case?
Thank you!
Jiahao
===============================================================================
Please access the attached hyperlink for an important electronic
communications disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
===============================================================================
RE: Cron schedule with DST-aware timezone
Posted by "Shaw, Damian P. " <da...@credit-suisse.com>.
As an Airflow user I would just like to add I strongly agree with this. The current behavior is intuitive but documented incorrectly.
IMO cron notation is expected to work like cron on modern Linux/Unix, i.e. whatever the system or user timezone is it follows the DST rules. And for jobs you wish to run every 24 hours that starts in a specific time or timezone then timedelta(days=1) makes sense.
Regards
Damian
-----Original Message-----
From: Bolke de Bruin [mailto:bdbruin@gmail.com]
Sent: Tuesday, May 14, 2019 2:13 AM
To: dev@airflow.apache.org
Subject: Re: Cron schedule with DST-aware timezone
The idea is obviously that with a Timedelta you want to say "add 24h". With DST this shifts the actual point in local time (in local time 17.00 can become 16.00/18.00).
Cron schedules specify points in local time so 17.00h always stays 17.00h.
There is nothing to make configurable here. Otherwise condition #1 wouldn't be true anymore and you would be adding either "23h" or "25h".
As a side note: we need to upgrade pendulum, python >3.6 introduced changed behavior.
B.
Sent from my iPhone
> On 14 May 2019, at 06:57, David Klosowski <da...@thinknear.com> wrote:
>
> The distinction could come from being non-UTC VS UTC. You can schedule a
> DAG at any time UTC and it would follow the 24-hour intervals but if it is
> non-UTC it would follow DST changes when relevant (by timezone). You could
> technically make this behavior configurable and follow the former path as
> well (not sure the use cases).
>
> On Mon, May 13, 2019 at 9:17 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> Just to add to that: there are already tests that tests this behaviour:
>>
>> https://github.com/PolideaInternal/airflow/blob/master/tests/models/test_dag.py#L749
>> and
>> indeed - the schedule follows DST changes rather than discards the DST
>> time.
>>
>> I think it is generally a good idea to follow DST (for clarity), but I
>> might miss some context/cases. Maybe it is indeed better to have always 24
>> hour intervals for daily schedules for example rather than 23/25
>> sometimes).
>>
>> BTW. I fixed one of those tests that was failing recently on Python 3.6 CI.
>> Python 3.6 behaved a bit differently than 3.5 at DST change:
>> https://issues.apache.org/jira/browse/AIRFLOW-4308
>>
>> J.
>>
>>
>>
>> On Mon, May 13, 2019 at 11:39 PM Maxime Beauchemin <
>> maximebeauchemin@gmail.com> wrote:
>>
>>> It would be great if people can provide failing unit tests as PR with
>> clear
>>> expectations stated out as code. It makes it easier for people to get
>>> consensus on expectations and for anyone to jump in and implement a fix.
>>>
>>> Max
>>>
>>> On Mon, May 13, 2019 at 12:48 PM David Klosowski <da...@thinknear.com>
>>> wrote:
>>>
>>>> Damian is correct. We've observed that exact behavior and noticed the
>>>> timedelta logic is dubiously broken for DST but works for CRON.
>>>>
>>>> On Mon, May 13, 2019 at 12:38 PM Shaw, Damian P. <
>>>> damian.shaw.2@credit-suisse.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I’m not part of the Airflow team but I came to the same conclusion,
>>> that
>>>>> the behavior is the opposite of what the documentation specifies.
>> When
>>>>> using the cron notation then DST is properly honored, when using
>>>> timedelta
>>>>> it is not. I played around with the DAG.following_schedule method to
>>>>> satisfy this was the case.
>>>>>
>>>>> I’ve had production instance of airflow based on this that
>> successfully
>>>>> respected the March/April DST changes in many timezones.
>>>>>
>>>>> Regards
>>>>> Damian
>>>>>
>>>>> From: Jiahao Chen [mailto:jhchen@google.com.INVALID]
>>>>> Sent: Monday, May 13, 2019 2:08 PM
>>>>> To: dev@airflow.apache.org
>>>>> Subject: Cron schedule with DST-aware timezone
>>>>>
>>>>> Hi team,
>>>>>
>>>>> I have a question about the expected behavior of Airflow scheduler
>> when
>>>>> the schedule_interval is a cron expression and the start_date is in a
>>>>> timezone with DST.
>>>>>
>>>>> Based on the Airflow documentation
>>>>> https://airflow.apache.org/timezone.html#cron-schedules, the DST
>>> change
>>>>> will be ignored if schedule_interval is a cron expresion (e.g. '0 17
>> *
>>> *
>>>>> *'). And it gives an example that the GMT offset will not change
>>>> regardless
>>>>> how DST changes. If I'm understanding it correctly, that means if I
>>>> upload
>>>>> a DAG with a schedule_interval of "0 17 * * *" and a start_date of
>>>>> 2019-03-15 17:00 PST(GMT-8) which is before the DST change on March
>> 10,
>>>> the
>>>>> Airflow scheduler will always start the DAG on 5 pm everyday GMT-8
>> even
>>>>> after the DST change on March 10.
>>>>>
>>>>> However, that is not the behavior I've seen with my experimental code
>>>> (see
>>>>> attachments). It looks like the the Airflow is actually taking the
>> DST
>>>> into
>>>>> account, since the execution time is always 17:00 locally, which is 1
>>>> hour
>>>>> off on the GMT after the DST change.
>>>>>
>>>>> Could you please confirm the behavior of Airflow scheduler in this
>> use
>>>>> case?
>>>>>
>>>>> Thank you!
>>>>> Jiahao
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>> ===============================================================================
>>>>>
>>>>> Please access the attached hyperlink for an important electronic
>>>>> communications disclaimer:
>>>>> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
>>>>>
>>>>
>>>
>> ===============================================================================
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> E: jarek.potiuk@polidea.com
>>
===============================================================================
Please access the attached hyperlink for an important electronic communications disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
===============================================================================
Re: Cron schedule with DST-aware timezone
Posted by Bolke de Bruin <bd...@gmail.com>.
The idea is obviously that with a Timedelta you want to say "add 24h". With DST this shifts the actual point in local time (in local time 17.00 can become 16.00/18.00).
Cron schedules specify points in local time so 17.00h always stays 17.00h.
There is nothing to make configurable here. Otherwise condition #1 wouldn't be true anymore and you would be adding either "23h" or "25h".
As a side note: we need to upgrade pendulum, python >3.6 introduced changed behavior.
B.
Sent from my iPhone
> On 14 May 2019, at 06:57, David Klosowski <da...@thinknear.com> wrote:
>
> The distinction could come from being non-UTC VS UTC. You can schedule a
> DAG at any time UTC and it would follow the 24-hour intervals but if it is
> non-UTC it would follow DST changes when relevant (by timezone). You could
> technically make this behavior configurable and follow the former path as
> well (not sure the use cases).
>
> On Mon, May 13, 2019 at 9:17 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> Just to add to that: there are already tests that tests this behaviour:
>>
>> https://github.com/PolideaInternal/airflow/blob/master/tests/models/test_dag.py#L749
>> and
>> indeed - the schedule follows DST changes rather than discards the DST
>> time.
>>
>> I think it is generally a good idea to follow DST (for clarity), but I
>> might miss some context/cases. Maybe it is indeed better to have always 24
>> hour intervals for daily schedules for example rather than 23/25
>> sometimes).
>>
>> BTW. I fixed one of those tests that was failing recently on Python 3.6 CI.
>> Python 3.6 behaved a bit differently than 3.5 at DST change:
>> https://issues.apache.org/jira/browse/AIRFLOW-4308
>>
>> J.
>>
>>
>>
>> On Mon, May 13, 2019 at 11:39 PM Maxime Beauchemin <
>> maximebeauchemin@gmail.com> wrote:
>>
>>> It would be great if people can provide failing unit tests as PR with
>> clear
>>> expectations stated out as code. It makes it easier for people to get
>>> consensus on expectations and for anyone to jump in and implement a fix.
>>>
>>> Max
>>>
>>> On Mon, May 13, 2019 at 12:48 PM David Klosowski <da...@thinknear.com>
>>> wrote:
>>>
>>>> Damian is correct. We've observed that exact behavior and noticed the
>>>> timedelta logic is dubiously broken for DST but works for CRON.
>>>>
>>>> On Mon, May 13, 2019 at 12:38 PM Shaw, Damian P. <
>>>> damian.shaw.2@credit-suisse.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I’m not part of the Airflow team but I came to the same conclusion,
>>> that
>>>>> the behavior is the opposite of what the documentation specifies.
>> When
>>>>> using the cron notation then DST is properly honored, when using
>>>> timedelta
>>>>> it is not. I played around with the DAG.following_schedule method to
>>>>> satisfy this was the case.
>>>>>
>>>>> I’ve had production instance of airflow based on this that
>> successfully
>>>>> respected the March/April DST changes in many timezones.
>>>>>
>>>>> Regards
>>>>> Damian
>>>>>
>>>>> From: Jiahao Chen [mailto:jhchen@google.com.INVALID]
>>>>> Sent: Monday, May 13, 2019 2:08 PM
>>>>> To: dev@airflow.apache.org
>>>>> Subject: Cron schedule with DST-aware timezone
>>>>>
>>>>> Hi team,
>>>>>
>>>>> I have a question about the expected behavior of Airflow scheduler
>> when
>>>>> the schedule_interval is a cron expression and the start_date is in a
>>>>> timezone with DST.
>>>>>
>>>>> Based on the Airflow documentation
>>>>> https://airflow.apache.org/timezone.html#cron-schedules, the DST
>>> change
>>>>> will be ignored if schedule_interval is a cron expresion (e.g. '0 17
>> *
>>> *
>>>>> *'). And it gives an example that the GMT offset will not change
>>>> regardless
>>>>> how DST changes. If I'm understanding it correctly, that means if I
>>>> upload
>>>>> a DAG with a schedule_interval of "0 17 * * *" and a start_date of
>>>>> 2019-03-15 17:00 PST(GMT-8) which is before the DST change on March
>> 10,
>>>> the
>>>>> Airflow scheduler will always start the DAG on 5 pm everyday GMT-8
>> even
>>>>> after the DST change on March 10.
>>>>>
>>>>> However, that is not the behavior I've seen with my experimental code
>>>> (see
>>>>> attachments). It looks like the the Airflow is actually taking the
>> DST
>>>> into
>>>>> account, since the execution time is always 17:00 locally, which is 1
>>>> hour
>>>>> off on the GMT after the DST change.
>>>>>
>>>>> Could you please confirm the behavior of Airflow scheduler in this
>> use
>>>>> case?
>>>>>
>>>>> Thank you!
>>>>> Jiahao
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>> ===============================================================================
>>>>>
>>>>> Please access the attached hyperlink for an important electronic
>>>>> communications disclaimer:
>>>>> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
>>>>>
>>>>
>>>
>> ===============================================================================
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> E: jarek.potiuk@polidea.com
>>
Re: Cron schedule with DST-aware timezone
Posted by David Klosowski <da...@thinknear.com>.
The distinction could come from being non-UTC VS UTC. You can schedule a
DAG at any time UTC and it would follow the 24-hour intervals but if it is
non-UTC it would follow DST changes when relevant (by timezone). You could
technically make this behavior configurable and follow the former path as
well (not sure the use cases).
On Mon, May 13, 2019 at 9:17 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Just to add to that: there are already tests that tests this behaviour:
>
> https://github.com/PolideaInternal/airflow/blob/master/tests/models/test_dag.py#L749
> and
> indeed - the schedule follows DST changes rather than discards the DST
> time.
>
> I think it is generally a good idea to follow DST (for clarity), but I
> might miss some context/cases. Maybe it is indeed better to have always 24
> hour intervals for daily schedules for example rather than 23/25
> sometimes).
>
> BTW. I fixed one of those tests that was failing recently on Python 3.6 CI.
> Python 3.6 behaved a bit differently than 3.5 at DST change:
> https://issues.apache.org/jira/browse/AIRFLOW-4308
>
> J.
>
>
>
> On Mon, May 13, 2019 at 11:39 PM Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
>
> > It would be great if people can provide failing unit tests as PR with
> clear
> > expectations stated out as code. It makes it easier for people to get
> > consensus on expectations and for anyone to jump in and implement a fix.
> >
> > Max
> >
> > On Mon, May 13, 2019 at 12:48 PM David Klosowski <da...@thinknear.com>
> > wrote:
> >
> > > Damian is correct. We've observed that exact behavior and noticed the
> > > timedelta logic is dubiously broken for DST but works for CRON.
> > >
> > > On Mon, May 13, 2019 at 12:38 PM Shaw, Damian P. <
> > > damian.shaw.2@credit-suisse.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > I’m not part of the Airflow team but I came to the same conclusion,
> > that
> > > > the behavior is the opposite of what the documentation specifies.
> When
> > > > using the cron notation then DST is properly honored, when using
> > > timedelta
> > > > it is not. I played around with the DAG.following_schedule method to
> > > > satisfy this was the case.
> > > >
> > > > I’ve had production instance of airflow based on this that
> successfully
> > > > respected the March/April DST changes in many timezones.
> > > >
> > > > Regards
> > > > Damian
> > > >
> > > > From: Jiahao Chen [mailto:jhchen@google.com.INVALID]
> > > > Sent: Monday, May 13, 2019 2:08 PM
> > > > To: dev@airflow.apache.org
> > > > Subject: Cron schedule with DST-aware timezone
> > > >
> > > > Hi team,
> > > >
> > > > I have a question about the expected behavior of Airflow scheduler
> when
> > > > the schedule_interval is a cron expression and the start_date is in a
> > > > timezone with DST.
> > > >
> > > > Based on the Airflow documentation
> > > > https://airflow.apache.org/timezone.html#cron-schedules, the DST
> > change
> > > > will be ignored if schedule_interval is a cron expresion (e.g. '0 17
> *
> > *
> > > > *'). And it gives an example that the GMT offset will not change
> > > regardless
> > > > how DST changes. If I'm understanding it correctly, that means if I
> > > upload
> > > > a DAG with a schedule_interval of "0 17 * * *" and a start_date of
> > > > 2019-03-15 17:00 PST(GMT-8) which is before the DST change on March
> 10,
> > > the
> > > > Airflow scheduler will always start the DAG on 5 pm everyday GMT-8
> even
> > > > after the DST change on March 10.
> > > >
> > > > However, that is not the behavior I've seen with my experimental code
> > > (see
> > > > attachments). It looks like the the Airflow is actually taking the
> DST
> > > into
> > > > account, since the execution time is always 17:00 locally, which is 1
> > > hour
> > > > off on the GMT after the DST change.
> > > >
> > > > Could you please confirm the behavior of Airflow scheduler in this
> use
> > > > case?
> > > >
> > > > Thank you!
> > > > Jiahao
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
> ===============================================================================
> > > >
> > > > Please access the attached hyperlink for an important electronic
> > > > communications disclaimer:
> > > > http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
> > > >
> > >
> >
> ===============================================================================
> > > >
> > > >
> > >
> >
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> E: jarek.potiuk@polidea.com
>
Re: Cron schedule with DST-aware timezone
Posted by Jarek Potiuk <Ja...@polidea.com>.
Just to add to that: there are already tests that tests this behaviour:
https://github.com/PolideaInternal/airflow/blob/master/tests/models/test_dag.py#L749
and
indeed - the schedule follows DST changes rather than discards the DST time.
I think it is generally a good idea to follow DST (for clarity), but I
might miss some context/cases. Maybe it is indeed better to have always 24
hour intervals for daily schedules for example rather than 23/25 sometimes).
BTW. I fixed one of those tests that was failing recently on Python 3.6 CI.
Python 3.6 behaved a bit differently than 3.5 at DST change:
https://issues.apache.org/jira/browse/AIRFLOW-4308
J.
On Mon, May 13, 2019 at 11:39 PM Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:
> It would be great if people can provide failing unit tests as PR with clear
> expectations stated out as code. It makes it easier for people to get
> consensus on expectations and for anyone to jump in and implement a fix.
>
> Max
>
> On Mon, May 13, 2019 at 12:48 PM David Klosowski <da...@thinknear.com>
> wrote:
>
> > Damian is correct. We've observed that exact behavior and noticed the
> > timedelta logic is dubiously broken for DST but works for CRON.
> >
> > On Mon, May 13, 2019 at 12:38 PM Shaw, Damian P. <
> > damian.shaw.2@credit-suisse.com> wrote:
> >
> > > Hi,
> > >
> > > I’m not part of the Airflow team but I came to the same conclusion,
> that
> > > the behavior is the opposite of what the documentation specifies. When
> > > using the cron notation then DST is properly honored, when using
> > timedelta
> > > it is not. I played around with the DAG.following_schedule method to
> > > satisfy this was the case.
> > >
> > > I’ve had production instance of airflow based on this that successfully
> > > respected the March/April DST changes in many timezones.
> > >
> > > Regards
> > > Damian
> > >
> > > From: Jiahao Chen [mailto:jhchen@google.com.INVALID]
> > > Sent: Monday, May 13, 2019 2:08 PM
> > > To: dev@airflow.apache.org
> > > Subject: Cron schedule with DST-aware timezone
> > >
> > > Hi team,
> > >
> > > I have a question about the expected behavior of Airflow scheduler when
> > > the schedule_interval is a cron expression and the start_date is in a
> > > timezone with DST.
> > >
> > > Based on the Airflow documentation
> > > https://airflow.apache.org/timezone.html#cron-schedules, the DST
> change
> > > will be ignored if schedule_interval is a cron expresion (e.g. '0 17 *
> *
> > > *'). And it gives an example that the GMT offset will not change
> > regardless
> > > how DST changes. If I'm understanding it correctly, that means if I
> > upload
> > > a DAG with a schedule_interval of "0 17 * * *" and a start_date of
> > > 2019-03-15 17:00 PST(GMT-8) which is before the DST change on March 10,
> > the
> > > Airflow scheduler will always start the DAG on 5 pm everyday GMT-8 even
> > > after the DST change on March 10.
> > >
> > > However, that is not the behavior I've seen with my experimental code
> > (see
> > > attachments). It looks like the the Airflow is actually taking the DST
> > into
> > > account, since the execution time is always 17:00 locally, which is 1
> > hour
> > > off on the GMT after the DST change.
> > >
> > > Could you please confirm the behavior of Airflow scheduler in this use
> > > case?
> > >
> > > Thank you!
> > > Jiahao
> > >
> > >
> > >
> > >
> > >
> >
> ===============================================================================
> > >
> > > Please access the attached hyperlink for an important electronic
> > > communications disclaimer:
> > > http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
> > >
> >
> ===============================================================================
> > >
> > >
> >
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
E: jarek.potiuk@polidea.com
Re: Cron schedule with DST-aware timezone
Posted by Maxime Beauchemin <ma...@gmail.com>.
It would be great if people can provide failing unit tests as PR with clear
expectations stated out as code. It makes it easier for people to get
consensus on expectations and for anyone to jump in and implement a fix.
Max
On Mon, May 13, 2019 at 12:48 PM David Klosowski <da...@thinknear.com>
wrote:
> Damian is correct. We've observed that exact behavior and noticed the
> timedelta logic is dubiously broken for DST but works for CRON.
>
> On Mon, May 13, 2019 at 12:38 PM Shaw, Damian P. <
> damian.shaw.2@credit-suisse.com> wrote:
>
> > Hi,
> >
> > I’m not part of the Airflow team but I came to the same conclusion, that
> > the behavior is the opposite of what the documentation specifies. When
> > using the cron notation then DST is properly honored, when using
> timedelta
> > it is not. I played around with the DAG.following_schedule method to
> > satisfy this was the case.
> >
> > I’ve had production instance of airflow based on this that successfully
> > respected the March/April DST changes in many timezones.
> >
> > Regards
> > Damian
> >
> > From: Jiahao Chen [mailto:jhchen@google.com.INVALID]
> > Sent: Monday, May 13, 2019 2:08 PM
> > To: dev@airflow.apache.org
> > Subject: Cron schedule with DST-aware timezone
> >
> > Hi team,
> >
> > I have a question about the expected behavior of Airflow scheduler when
> > the schedule_interval is a cron expression and the start_date is in a
> > timezone with DST.
> >
> > Based on the Airflow documentation
> > https://airflow.apache.org/timezone.html#cron-schedules, the DST change
> > will be ignored if schedule_interval is a cron expresion (e.g. '0 17 * *
> > *'). And it gives an example that the GMT offset will not change
> regardless
> > how DST changes. If I'm understanding it correctly, that means if I
> upload
> > a DAG with a schedule_interval of "0 17 * * *" and a start_date of
> > 2019-03-15 17:00 PST(GMT-8) which is before the DST change on March 10,
> the
> > Airflow scheduler will always start the DAG on 5 pm everyday GMT-8 even
> > after the DST change on March 10.
> >
> > However, that is not the behavior I've seen with my experimental code
> (see
> > attachments). It looks like the the Airflow is actually taking the DST
> into
> > account, since the execution time is always 17:00 locally, which is 1
> hour
> > off on the GMT after the DST change.
> >
> > Could you please confirm the behavior of Airflow scheduler in this use
> > case?
> >
> > Thank you!
> > Jiahao
> >
> >
> >
> >
> >
> ===============================================================================
> >
> > Please access the attached hyperlink for an important electronic
> > communications disclaimer:
> > http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
> >
> ===============================================================================
> >
> >
>
Re: Cron schedule with DST-aware timezone
Posted by David Klosowski <da...@thinknear.com>.
Damian is correct. We've observed that exact behavior and noticed the
timedelta logic is dubiously broken for DST but works for CRON.
On Mon, May 13, 2019 at 12:38 PM Shaw, Damian P. <
damian.shaw.2@credit-suisse.com> wrote:
> Hi,
>
> I’m not part of the Airflow team but I came to the same conclusion, that
> the behavior is the opposite of what the documentation specifies. When
> using the cron notation then DST is properly honored, when using timedelta
> it is not. I played around with the DAG.following_schedule method to
> satisfy this was the case.
>
> I’ve had production instance of airflow based on this that successfully
> respected the March/April DST changes in many timezones.
>
> Regards
> Damian
>
> From: Jiahao Chen [mailto:jhchen@google.com.INVALID]
> Sent: Monday, May 13, 2019 2:08 PM
> To: dev@airflow.apache.org
> Subject: Cron schedule with DST-aware timezone
>
> Hi team,
>
> I have a question about the expected behavior of Airflow scheduler when
> the schedule_interval is a cron expression and the start_date is in a
> timezone with DST.
>
> Based on the Airflow documentation
> https://airflow.apache.org/timezone.html#cron-schedules, the DST change
> will be ignored if schedule_interval is a cron expresion (e.g. '0 17 * *
> *'). And it gives an example that the GMT offset will not change regardless
> how DST changes. If I'm understanding it correctly, that means if I upload
> a DAG with a schedule_interval of "0 17 * * *" and a start_date of
> 2019-03-15 17:00 PST(GMT-8) which is before the DST change on March 10, the
> Airflow scheduler will always start the DAG on 5 pm everyday GMT-8 even
> after the DST change on March 10.
>
> However, that is not the behavior I've seen with my experimental code (see
> attachments). It looks like the the Airflow is actually taking the DST into
> account, since the execution time is always 17:00 locally, which is 1 hour
> off on the GMT after the DST change.
>
> Could you please confirm the behavior of Airflow scheduler in this use
> case?
>
> Thank you!
> Jiahao
>
>
>
>
> ===============================================================================
>
> Please access the attached hyperlink for an important electronic
> communications disclaimer:
> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
> ===============================================================================
>
>
RE: Cron schedule with DST-aware timezone
Posted by "Shaw, Damian P. " <da...@credit-suisse.com>.
Hi,
I’m not part of the Airflow team but I came to the same conclusion, that the behavior is the opposite of what the documentation specifies. When using the cron notation then DST is properly honored, when using timedelta it is not. I played around with the DAG.following_schedule method to satisfy this was the case.
I’ve had production instance of airflow based on this that successfully respected the March/April DST changes in many timezones.
Regards
Damian
From: Jiahao Chen [mailto:jhchen@google.com.INVALID]
Sent: Monday, May 13, 2019 2:08 PM
To: dev@airflow.apache.org
Subject: Cron schedule with DST-aware timezone
Hi team,
I have a question about the expected behavior of Airflow scheduler when the schedule_interval is a cron expression and the start_date is in a timezone with DST.
Based on the Airflow documentation https://airflow.apache.org/timezone.html#cron-schedules, the DST change will be ignored if schedule_interval is a cron expresion (e.g. '0 17 * * *'). And it gives an example that the GMT offset will not change regardless how DST changes. If I'm understanding it correctly, that means if I upload a DAG with a schedule_interval of "0 17 * * *" and a start_date of 2019-03-15 17:00 PST(GMT-8) which is before the DST change on March 10, the Airflow scheduler will always start the DAG on 5 pm everyday GMT-8 even after the DST change on March 10.
However, that is not the behavior I've seen with my experimental code (see attachments). It looks like the the Airflow is actually taking the DST into account, since the execution time is always 17:00 locally, which is 1 hour off on the GMT after the DST change.
Could you please confirm the behavior of Airflow scheduler in this use case?
Thank you!
Jiahao
===============================================================================
Please access the attached hyperlink for an important electronic communications disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
===============================================================================