You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Bolke de Bruin <bd...@gmail.com> on 2018/10/26 15:12:26 UTC
1.10.1 Release?
Hey Ash,
I was wondering if you are picking up the 1.10.1 release? Master is speeding ahead and you were tracking fixes for 1.10.1 right?
B.
Re: 1.10.1 Release?
Posted by Ash Berlin-Taylor <as...@apache.org>.
Hey, yeah I've been working (a bit slowly) on it - we had a few test failures on the v1-10-test branch (conflicts/bad cherry-picks) to unravel to get green builds, but we're there now.
A rough summary of where we are (wide columns, won't look good if it has to line-wrap, https://gist.github.com/ashb/4285c5a6c6b3be616495b1181d3fd63e to see it in another format):
(airflow) (themisto python/incuba…irflow v1-10-test:+)% ./dev/airflow-jira compare 1.10.1
ISSUE ID |TYPE ||PRIORITY ||STATUS |DESCRIPTION |MERGED|PR |COMMIT
AIRFLOW-3238 |Bug ||Major ||Resolved |Dags, removed from the filesystem, are not deactiv|1 |#na |1eeb0a4a24fa8541763a67f84ec9f4b034f66475
AIRFLOW-3237 |Improvement ||Major ||Resolved |Refactor example DAGs |1 |#na |fdfb359e4b95dfadfa3973d43025f61f4aa3b96a
AIRFLOW-3232 |Improvement ||Trivial ||Resolved |Make documentation for GCF Functions operator more|1 |#na |d4dff076a6eaf169424822c0010c802f4af80c6a
AIRFLOW-3203 |Bug ||Critical ||Closed |Bugs in DockerOperator & Some operator test script|1 |#na |3dfc9562d3df127ca8337edb600ab6a0259521ac
AIRFLOW-3197 |Improvement ||Minor ||Resolved |EMR Hook is missing some parameters to valid on th|1 |#na |079b0ee95e4a1d37bdb31c477b531db264242bf7
AIRFLOW-3195 |Improvement ||Trivial ||Resolved |Druid Hook: Log ingestion spec and task id |1 |#na |8e55b499b2dcc6e4dc4d86f8225d1424f6886a0c
AIRFLOW-3187 |Wish ||Major ||Resolved |Update airflow.gif file with a slower version |0 |- |-
AIRFLOW-3183 |Bug ||Minor ||Resolved |Potential Bug in utils/dag_processing/DagFileProce|1 |#na |0e98c60268c805639be8cd8c1cdf9a8909f966bb
AIRFLOW-3178 |Bug ||Major ||Resolved |`airflow run` config doens't cope with % in config|1 |#na |5bc4cfa4a7908818877828ed3db1090b71e65b93
AIRFLOW-3177 |Improvement ||Minor ||Resolved |Change scheduler_heartbeat metric from gauge to co|1 |#na |cab121bc036be24bcb1e397f48ca1672e1f73692
AIRFLOW-3173 |Improvement ||Major ||Resolved |Add _cmd options for password config options |1 |#na |e2a238c3a55be5fcb17b5ec64e7165dd954a4bc0
AIRFLOW-3172 |Bug ||Major ||Open |AttributeError: 'DagModel' object has no attribute|0 |- |-
AIRFLOW-3162 |Bug ||Minor ||Resolved |HttpHook fails to parse URL when port is specified|1 |#na |040707b5d830acb96fc6c9a367039ed96aba7231
AIRFLOW-3147 |New Feature ||Major ||Resolved |Update Flask-AppBuilder version |1 |#na |35f996b65a2e68f4e168e3f466f3a939e4fb904a
AIRFLOW-3138 |Bug ||Major ||Resolved |Migration cc1e65623dc7 creates issues with postgre|1 |#na |1234af2995780327517c770bd09daf684563527d
AIRFLOW-3137 |Improvement ||Trivial ||Resolved |Make ProxyFix middleware optional |0 |- |-
AIRFLOW-3127 |Improvement ||Minor ||Resolved |Celery SSL Documentation is out-dated |0 |- |-
AIRFLOW-3124 |Bug ||Minor ||Resolved |Broken webserver debug mode (RBAC) |1 |#na |f9988147999b73f10978f74d37f704dae8e4f012
AIRFLOW-3119 |Improvement ||Minor ||Resolved |Enable loglevel on celery worker and inherit from |1 |#na |23be2a30cd28f236722433a07c536cb17c8515ee
AIRFLOW-3114 |Improvement ||Minor ||Open |Add feature to create External BigQuery Table for |0 |- |-
AIRFLOW-3112 |Improvement ||Minor ||Resolved |Align SFTP hook with SSH hook |1 |#na |35969a426822e5fc9bc2b941b4fe5c5098a5ee52
AIRFLOW-3111 |Bug ||Minor ||Resolved |Confusing comments and instructions for log templa|0 |- |-
AIRFLOW-3109 |Bug ||Major ||Resolved |Default user permission should contain 'can_clear'|0 |- |-
AIRFLOW-3104 |Improvement ||Minor ||Resolved |.airflowignore feature is not mentioned at all in |0 |- |-
AIRFLOW-3099 |Bug ||Minor ||Resolved |Errors raised when some blocs are missing in airfl|1 |#na |25bb0dff64687948f73b3fb86fae4e8476f4f9ce
AIRFLOW-3090 |Wish ||Minor ||Closed |INFO logs are too verbose |0 |- |-
AIRFLOW-3089 |Bug ||Minor ||Resolved |Google auth doesn't work under http |1 |#na |bd7510094de25f2c2bce2a49a3febae28899f3b6
AIRFLOW-3079 |Bug ||Major ||Resolved |Improve initdb to support MSSQL Server |0 |- |-
AIRFLOW-3078 |New Feature ||Trivial ||Resolved |Basic operators for Google Compute Engine |1 |#na |6aeda0fcb2ecbba647a6cede992bf23556c1b05c
AIRFLOW-3072 |Bug ||Major ||Resolved |Only admin can view logs in RBAC UI |0 |- |-
AIRFLOW-3069 |Improvement ||Minor ||Resolved |Decode output of S3 file transform operator |0 |- |-
AIRFLOW-3067 |Improvement ||Minor ||Resolved |(www_rbac) Flask flash messages are not displayed |0 |- |-
AIRFLOW-3064 |Bug ||Minor ||Resolved |No output from `airflow test` due to default loggi|0 |- |-
AIRFLOW-3046 |Bug ||Major ||Resolved |ECS Operator mistakenly reports success when task |0 |- |-
AIRFLOW-3033 |Bug ||Major ||Open |`airflow upgradedb` should create FAB user tables |0 |- |-
AIRFLOW-3030 |Improvement ||Minor ||Resolved |Command Line docs incorrect subdir |1 |#na |ebf2260472c67edb8c4b95b0fbc21e3351fd365b
AIRFLOW-3025 |Improvement ||Minor ||Resolved |Allow to specify dns and dns-search parameters for|1 |#na |db3bcc624667f1d1be25bdbde903b1fcee996ae0
AIRFLOW-3022 |Improvement ||Minor ||Open |Add volume mount to KubernetesExecutorConfig |0 |- |-
AIRFLOW-3012 |Bug ||Major ||Resolved |Email on sla miss is send only to first address on|1 |#na |769c5b3ed20de75489a76d3282f62b748fe8c910
AIRFLOW-3008 |Improvement ||Major ||Closed |Move Kubernetes related example DAGs to contribe/e|1 |#na |bc97e6ee0b6136368d6aebc5c30df848420bb6b6
AIRFLOW-3006 |Improvement ||Major ||Closed |Error when schedule_interval="None" |1 |#na |bc27aace7d23baad145c75171d105e8cd902ece6
AIRFLOW-3002 |Bug ||Major ||Closed |ValueError in dataflow operators when using GCS ja|0 |- |-
AIRFLOW-2997 |Improvement ||Minor ||Closed |Support for clustered tables in Bigquery hooks/ope|0 |- |-
AIRFLOW-2994 |Bug ||Major ||Resolved |flatten_results in BigQueryOperator/BigQueryHook s|1 |#na |e7f73d20462964bf104c7508d3f3d669e7797e49
AIRFLOW-2990 |Improvement ||Minor ||Resolved |Docstrings for Hooks/Operators are in incorrect fo|1 |#na |4f6f496b763f202fcca0de5fdc71f5655d9474f8
AIRFLOW-2989 |New Feature ||Minor ||Resolved |No Parameter to change bootDiskType for DataprocCl|1 |#na |d9af846e8e07874318d85f3c82a0216c3956249e
AIRFLOW-2984 |Bug ||Major ||Resolved |Cannot convert naive_datetime when task has a naiv|1 |#na |152214ea6dd08533aa40e455415c2741259482e9
AIRFLOW-2981 |Bug ||Major ||Resolved |TypeError in dataflow operators when using GCS jar|1 |#na |abb72481557ff0c8a3b20a6bc1d36f970ae9a343
AIRFLOW-2979 |Bug ||Critical ||Resolved |Deprecated Celery Option not in Options list |1 |#na |f04872ed0f8170da48385882f5e04b3c27c46134
AIRFLOW-2974 |New Feature ||Major ||Resolved |Add Start/Restart/Terminate methods Databricks Hoo|1 |#na |a3ecc0ac3cafc9440e875a61d9ac670079b9d2fa
A few more fixed to cherry-pick to do on the branch, and there are a couple of open tasks that still need fixes writing or merging - (3033, 3172)
> On 26 Oct 2018, at 16:12, Bolke de Bruin <bd...@gmail.com> wrote:
>
> Hey Ash,
>
> I was wondering if you are picking up the 1.10.1 release? Master is speeding ahead and you were tracking fixes for 1.10.1 right?
>
> B.
Re: 1.10.1 Release?
Posted by Bolke de Bruin <bd...@gmail.com>.
I think DAGs will start running by themselves again as soon as the interval has passed that it has already seen. So depending your schedule this can take a while.
B.
> On 30 Oct 2018, at 19:53, ash@apache.org wrote:
>
> 1.10.1 isn't out yet, so 1.10.0 :)
>
> I think this only affects days with a schedule interval that is one hour or more frequently, and where the dag timezone is not UTC.
>
> For anyone stuck on this in prod I think the fix is to manually create (via backfill or trigger) a dag run after the tz change over time. This should unblock the scheduler.
>
> Ash
>
> On 30 October 2018 18:43:30 GMT, David Klosowski <da...@thinknear.com> wrote:
>> Hi Airflow Devs:
>>
>> Is this timezone issue in Airflow version 1.10.0 or only in 1.10.1?
>>
>> Thanks.
>>
>> Regards,
>> David
>>
>> On Tue, Oct 30, 2018 at 11:11 AM Bolke de Bruin <bd...@gmail.com>
>> wrote:
>>
>>> we specifically remove timezone info to determine the next schedule.
>> Ie.
>>> cron sets exact date times so tz info should not make sense. I’m
>> going to
>>> have a look now.
>>>
>>>
>>>> On 30 Oct 2018, at 19:01, Ash Berlin-Taylor <as...@apache.org> wrote:
>>>>
>>>> I've done a bit more digging - the issue is of our tz-aware
>> handling
>>> inside following_schedule (and previous schedule) - causing it to
>> loop.
>>>>
>>>> This section of the croniter docs seems relevant
>>> https://github.com/kiorky/croniter#about-dst
>>>>
>>>> Be sure to init your croniter instance with a TZ aware datetime
>> for
>>> this to work !:
>>>>>>> local_date = tz.localize(datetime(2017, 3, 26))
>>>>>>> val = croniter('0 0 * * *', local_date).get_next(datetime)
>>>>
>>>> I think the problem is that we are _not_ passing a TZ aware dag in
>> and
>>> we should be.
>>>>
>>>>> On 30 Oct 2018, at 17:35, Bolke de Bruin <bd...@gmail.com>
>> wrote:
>>>>>
>>>>> Oh that’s a great environment to start digging. Thanks. I’ll have
>> a
>>> look.
>>>>>
>>>>> B.
>>>>>
>>>>> Verstuurd vanaf mijn iPad
>>>>>
>>>>>> Op 30 okt. 2018 om 18:25 heeft Ash Berlin-Taylor <as...@apache.org>
>> het
>>> volgende geschreven:
>>>>>>
>>>>>> This line in airflow.jobs (line 874 in my checkout) is causing
>> the
>>> loop:
>>>>>>
>>>>>> last_run = dag.get_last_dagrun(session=session)
>>>>>> if last_run and next_run_date:
>>>>>> while next_run_date <= last_run.execution_date:
>>>>>> next_run_date =
>> dag.following_schedule(next_run_date)
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On 30 Oct 2018, at 17:20, Ash Berlin-Taylor <as...@apache.org>
>> wrote:
>>>>>>>
>>>>>>> Hi, kaczors on gitter has produced a minmal reproduction case:
>>> https://github.com/kaczors/airflow_1_10_tz_bug
>>>>>>>
>>>>>>> Rough repro steps: In a VM, with time syncing disabled, and
>>> configured with system timezone of Europe/Zurich (or any other CEST
>> one)
>>> run
>>>>>>>
>>>>>>> - `date 10280250.00`
>>>>>>> - initdb, start scheduler, webserver, enable dag etc.
>>>>>>> - `date 10280259.00`
>>>>>>> - wait 5-10 mins for scheduler to catch up
>>>>>>> - After the on-the-hour task run the scheduler will spin up
>> another
>>> process to parse the dag... and it never returns.
>>>>>>>
>>>>>>> I've only just managed to reproduce it, so haven't dug in to why
>> yet.
>>> A quick hacky debug print shows something is stuck in an infinite
>> loop.
>>>>>>>
>>>>>>> -ash
>>>>>>>
>>>>>>>> On 29 Oct 2018, at 17:59, Bolke de Bruin <bd...@gmail.com>
>> wrote:
>>>>>>>>
>>>>>>>> Can this be confirmed? Then I can have a look at it. Preferably
>> with
>>> dag definition code.
>>>>>>>>
>>>>>>>> On the licensing requirements:
>>>>>>>>
>>>>>>>> 1. Indeed licensing header for markdown documents. It was
>> suggested
>>> to use html comments. I’m not sure how that renders with others like
>> PDF
>>> though.
>>>>>>>> 2. The licensing notifications need to be tied to a specific
>> version
>>> as licenses might change with versions.
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>> Bolke
>>>>>>>>
>>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>>
>>>>>>>>> Op 29 okt. 2018 om 12:39 heeft Ash Berlin-Taylor
>> <as...@apache.org>
>>> het volgende geschreven:
>>>>>>>>>
>>>>>>>>> I was going to make a start on the release, but two people
>> have
>>> reported that there might be an issue around non-UTC dags and the
>> scheduler
>>> changing over from Summer time.
>>>>>>>>>
>>>>>>>>>> 08:45 Emmanuel> Hi there, we are currently experiencing a
>> very
>>> strange issue : we have hourly DAGs with a start_date in a local
>> timezone
>>> (not UTC) and since (Sunday) the last winter time change they don’t
>> run
>>> anymore. Any idea ?
>>>>>>>>>> 09:41 <Emmanuel> it impacted all our DAG that had a run at
>> 3am
>>> (Europe/Paris), the exact time of winter time change :(
>>>>>>>>>
>>>>>>>>> I am going to take a look at this today and see if I can get
>> to the
>>> bottom of it.
>>>>>>>>>
>>>>>>>>> Bolke: are there any outstanding tasks/issues that you know of
>> that
>>> might slow down the vote for a 1.10.1? (i.e. did we sort of out all
>> the
>>> licensing issues that were asked of us? I thought I read something
>> about
>>> license declarations in markdown files?)
>>>>>>>>>
>>>>>>>>> -ash
>>>>>>>>>
>>>>>>>>>> On 28 Oct 2018, at 14:46, Bolke de Bruin <bd...@gmail.com>
>>> wrote:
>>>>>>>>>>
>>>>>>>>>> I agree with that, but I would favor time based releases
>> instead.
>>> We are again at the point that a release takes so much time that the
>> gap is
>>> getting really big again. @ash why not start releasing now and move
>> the
>>> remainder to 1.10.2? I dont think there are real blockers (although
>> we
>>> might find them).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On 28 Oct 2018, at 15:35, airflowuser
>> <ai...@protonmail.com.INVALID>
>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> I was really hoping that
>>> https://github.com/apache/incubator-airflow/pull/4069 will be merged
>> into
>>> 1.10.1
>>>>>>>>>>> Deleting dags was a highly requested feature for 1.10 - this
>> can
>>> fix the problem with it.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>>>>>>>>> On Friday, October 26, 2018 6:12 PM, Bolke de Bruin <
>>> bdbruin@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hey Ash,
>>>>>>>>>>>>
>>>>>>>>>>>> I was wondering if you are picking up the 1.10.1 release?
>> Master
>>> is speeding ahead and you were tracking fixes for 1.10.1 right?
>>>>>>>>>>>>
>>>>>>>>>>>> B.
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>>>
Re: 1.10.1 Release?
Posted by as...@apache.org.
1.10.1 isn't out yet, so 1.10.0 :)
I think this only affects days with a schedule interval that is one hour or more frequently, and where the dag timezone is not UTC.
For anyone stuck on this in prod I think the fix is to manually create (via backfill or trigger) a dag run after the tz change over time. This should unblock the scheduler.
Ash
On 30 October 2018 18:43:30 GMT, David Klosowski <da...@thinknear.com> wrote:
>Hi Airflow Devs:
>
>Is this timezone issue in Airflow version 1.10.0 or only in 1.10.1?
>
>Thanks.
>
>Regards,
>David
>
>On Tue, Oct 30, 2018 at 11:11 AM Bolke de Bruin <bd...@gmail.com>
>wrote:
>
>> we specifically remove timezone info to determine the next schedule.
>Ie.
>> cron sets exact date times so tz info should not make sense. I’m
>going to
>> have a look now.
>>
>>
>> > On 30 Oct 2018, at 19:01, Ash Berlin-Taylor <as...@apache.org> wrote:
>> >
>> > I've done a bit more digging - the issue is of our tz-aware
>handling
>> inside following_schedule (and previous schedule) - causing it to
>loop.
>> >
>> > This section of the croniter docs seems relevant
>> https://github.com/kiorky/croniter#about-dst
>> >
>> > Be sure to init your croniter instance with a TZ aware datetime
>for
>> this to work !:
>> >>>> local_date = tz.localize(datetime(2017, 3, 26))
>> >>>> val = croniter('0 0 * * *', local_date).get_next(datetime)
>> >
>> > I think the problem is that we are _not_ passing a TZ aware dag in
>and
>> we should be.
>> >
>> >> On 30 Oct 2018, at 17:35, Bolke de Bruin <bd...@gmail.com>
>wrote:
>> >>
>> >> Oh that’s a great environment to start digging. Thanks. I’ll have
>a
>> look.
>> >>
>> >> B.
>> >>
>> >> Verstuurd vanaf mijn iPad
>> >>
>> >>> Op 30 okt. 2018 om 18:25 heeft Ash Berlin-Taylor <as...@apache.org>
>het
>> volgende geschreven:
>> >>>
>> >>> This line in airflow.jobs (line 874 in my checkout) is causing
>the
>> loop:
>> >>>
>> >>> last_run = dag.get_last_dagrun(session=session)
>> >>> if last_run and next_run_date:
>> >>> while next_run_date <= last_run.execution_date:
>> >>> next_run_date =
>dag.following_schedule(next_run_date)
>> >>>
>> >>>
>> >>>
>> >>>> On 30 Oct 2018, at 17:20, Ash Berlin-Taylor <as...@apache.org>
>wrote:
>> >>>>
>> >>>> Hi, kaczors on gitter has produced a minmal reproduction case:
>> https://github.com/kaczors/airflow_1_10_tz_bug
>> >>>>
>> >>>> Rough repro steps: In a VM, with time syncing disabled, and
>> configured with system timezone of Europe/Zurich (or any other CEST
>one)
>> run
>> >>>>
>> >>>> - `date 10280250.00`
>> >>>> - initdb, start scheduler, webserver, enable dag etc.
>> >>>> - `date 10280259.00`
>> >>>> - wait 5-10 mins for scheduler to catch up
>> >>>> - After the on-the-hour task run the scheduler will spin up
>another
>> process to parse the dag... and it never returns.
>> >>>>
>> >>>> I've only just managed to reproduce it, so haven't dug in to why
>yet.
>> A quick hacky debug print shows something is stuck in an infinite
>loop.
>> >>>>
>> >>>> -ash
>> >>>>
>> >>>>> On 29 Oct 2018, at 17:59, Bolke de Bruin <bd...@gmail.com>
>wrote:
>> >>>>>
>> >>>>> Can this be confirmed? Then I can have a look at it. Preferably
>with
>> dag definition code.
>> >>>>>
>> >>>>> On the licensing requirements:
>> >>>>>
>> >>>>> 1. Indeed licensing header for markdown documents. It was
>suggested
>> to use html comments. I’m not sure how that renders with others like
>PDF
>> though.
>> >>>>> 2. The licensing notifications need to be tied to a specific
>version
>> as licenses might change with versions.
>> >>>>>
>> >>>>> Cheers
>> >>>>> Bolke
>> >>>>>
>> >>>>> Verstuurd vanaf mijn iPad
>> >>>>>
>> >>>>>> Op 29 okt. 2018 om 12:39 heeft Ash Berlin-Taylor
><as...@apache.org>
>> het volgende geschreven:
>> >>>>>>
>> >>>>>> I was going to make a start on the release, but two people
>have
>> reported that there might be an issue around non-UTC dags and the
>scheduler
>> changing over from Summer time.
>> >>>>>>
>> >>>>>>> 08:45 Emmanuel> Hi there, we are currently experiencing a
>very
>> strange issue : we have hourly DAGs with a start_date in a local
>timezone
>> (not UTC) and since (Sunday) the last winter time change they don’t
>run
>> anymore. Any idea ?
>> >>>>>>> 09:41 <Emmanuel> it impacted all our DAG that had a run at
>3am
>> (Europe/Paris), the exact time of winter time change :(
>> >>>>>>
>> >>>>>> I am going to take a look at this today and see if I can get
>to the
>> bottom of it.
>> >>>>>>
>> >>>>>> Bolke: are there any outstanding tasks/issues that you know of
>that
>> might slow down the vote for a 1.10.1? (i.e. did we sort of out all
>the
>> licensing issues that were asked of us? I thought I read something
>about
>> license declarations in markdown files?)
>> >>>>>>
>> >>>>>> -ash
>> >>>>>>
>> >>>>>>> On 28 Oct 2018, at 14:46, Bolke de Bruin <bd...@gmail.com>
>> wrote:
>> >>>>>>>
>> >>>>>>> I agree with that, but I would favor time based releases
>instead.
>> We are again at the point that a release takes so much time that the
>gap is
>> getting really big again. @ash why not start releasing now and move
>the
>> remainder to 1.10.2? I dont think there are real blockers (although
>we
>> might find them).
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>> On 28 Oct 2018, at 15:35, airflowuser
><ai...@protonmail.com.INVALID>
>> wrote:
>> >>>>>>>>
>> >>>>>>>> I was really hoping that
>> https://github.com/apache/incubator-airflow/pull/4069 will be merged
>into
>> 1.10.1
>> >>>>>>>> Deleting dags was a highly requested feature for 1.10 - this
>can
>> fix the problem with it.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>> >>>>>>>>>> On Friday, October 26, 2018 6:12 PM, Bolke de Bruin <
>> bdbruin@gmail.com> wrote:
>> >>>>>>>>>
>> >>>>>>>>> Hey Ash,
>> >>>>>>>>>
>> >>>>>>>>> I was wondering if you are picking up the 1.10.1 release?
>Master
>> is speeding ahead and you were tracking fixes for 1.10.1 right?
>> >>>>>>>>>
>> >>>>>>>>> B.
>> >>>>>>
>> >>>>
>> >>>
>> >
>>
>>
Re: 1.10.1 Release?
Posted by David Klosowski <da...@thinknear.com>.
Hi Airflow Devs:
Is this timezone issue in Airflow version 1.10.0 or only in 1.10.1?
Thanks.
Regards,
David
On Tue, Oct 30, 2018 at 11:11 AM Bolke de Bruin <bd...@gmail.com> wrote:
> we specifically remove timezone info to determine the next schedule. Ie.
> cron sets exact date times so tz info should not make sense. I’m going to
> have a look now.
>
>
> > On 30 Oct 2018, at 19:01, Ash Berlin-Taylor <as...@apache.org> wrote:
> >
> > I've done a bit more digging - the issue is of our tz-aware handling
> inside following_schedule (and previous schedule) - causing it to loop.
> >
> > This section of the croniter docs seems relevant
> https://github.com/kiorky/croniter#about-dst
> >
> > Be sure to init your croniter instance with a TZ aware datetime for
> this to work !:
> >>>> local_date = tz.localize(datetime(2017, 3, 26))
> >>>> val = croniter('0 0 * * *', local_date).get_next(datetime)
> >
> > I think the problem is that we are _not_ passing a TZ aware dag in and
> we should be.
> >
> >> On 30 Oct 2018, at 17:35, Bolke de Bruin <bd...@gmail.com> wrote:
> >>
> >> Oh that’s a great environment to start digging. Thanks. I’ll have a
> look.
> >>
> >> B.
> >>
> >> Verstuurd vanaf mijn iPad
> >>
> >>> Op 30 okt. 2018 om 18:25 heeft Ash Berlin-Taylor <as...@apache.org> het
> volgende geschreven:
> >>>
> >>> This line in airflow.jobs (line 874 in my checkout) is causing the
> loop:
> >>>
> >>> last_run = dag.get_last_dagrun(session=session)
> >>> if last_run and next_run_date:
> >>> while next_run_date <= last_run.execution_date:
> >>> next_run_date = dag.following_schedule(next_run_date)
> >>>
> >>>
> >>>
> >>>> On 30 Oct 2018, at 17:20, Ash Berlin-Taylor <as...@apache.org> wrote:
> >>>>
> >>>> Hi, kaczors on gitter has produced a minmal reproduction case:
> https://github.com/kaczors/airflow_1_10_tz_bug
> >>>>
> >>>> Rough repro steps: In a VM, with time syncing disabled, and
> configured with system timezone of Europe/Zurich (or any other CEST one)
> run
> >>>>
> >>>> - `date 10280250.00`
> >>>> - initdb, start scheduler, webserver, enable dag etc.
> >>>> - `date 10280259.00`
> >>>> - wait 5-10 mins for scheduler to catch up
> >>>> - After the on-the-hour task run the scheduler will spin up another
> process to parse the dag... and it never returns.
> >>>>
> >>>> I've only just managed to reproduce it, so haven't dug in to why yet.
> A quick hacky debug print shows something is stuck in an infinite loop.
> >>>>
> >>>> -ash
> >>>>
> >>>>> On 29 Oct 2018, at 17:59, Bolke de Bruin <bd...@gmail.com> wrote:
> >>>>>
> >>>>> Can this be confirmed? Then I can have a look at it. Preferably with
> dag definition code.
> >>>>>
> >>>>> On the licensing requirements:
> >>>>>
> >>>>> 1. Indeed licensing header for markdown documents. It was suggested
> to use html comments. I’m not sure how that renders with others like PDF
> though.
> >>>>> 2. The licensing notifications need to be tied to a specific version
> as licenses might change with versions.
> >>>>>
> >>>>> Cheers
> >>>>> Bolke
> >>>>>
> >>>>> Verstuurd vanaf mijn iPad
> >>>>>
> >>>>>> Op 29 okt. 2018 om 12:39 heeft Ash Berlin-Taylor <as...@apache.org>
> het volgende geschreven:
> >>>>>>
> >>>>>> I was going to make a start on the release, but two people have
> reported that there might be an issue around non-UTC dags and the scheduler
> changing over from Summer time.
> >>>>>>
> >>>>>>> 08:45 Emmanuel> Hi there, we are currently experiencing a very
> strange issue : we have hourly DAGs with a start_date in a local timezone
> (not UTC) and since (Sunday) the last winter time change they don’t run
> anymore. Any idea ?
> >>>>>>> 09:41 <Emmanuel> it impacted all our DAG that had a run at 3am
> (Europe/Paris), the exact time of winter time change :(
> >>>>>>
> >>>>>> I am going to take a look at this today and see if I can get to the
> bottom of it.
> >>>>>>
> >>>>>> Bolke: are there any outstanding tasks/issues that you know of that
> might slow down the vote for a 1.10.1? (i.e. did we sort of out all the
> licensing issues that were asked of us? I thought I read something about
> license declarations in markdown files?)
> >>>>>>
> >>>>>> -ash
> >>>>>>
> >>>>>>> On 28 Oct 2018, at 14:46, Bolke de Bruin <bd...@gmail.com>
> wrote:
> >>>>>>>
> >>>>>>> I agree with that, but I would favor time based releases instead.
> We are again at the point that a release takes so much time that the gap is
> getting really big again. @ash why not start releasing now and move the
> remainder to 1.10.2? I dont think there are real blockers (although we
> might find them).
> >>>>>>>
> >>>>>>>
> >>>>>>>> On 28 Oct 2018, at 15:35, airflowuser <ai...@protonmail.com.INVALID>
> wrote:
> >>>>>>>>
> >>>>>>>> I was really hoping that
> https://github.com/apache/incubator-airflow/pull/4069 will be merged into
> 1.10.1
> >>>>>>>> Deleting dags was a highly requested feature for 1.10 - this can
> fix the problem with it.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> >>>>>>>>>> On Friday, October 26, 2018 6:12 PM, Bolke de Bruin <
> bdbruin@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>> Hey Ash,
> >>>>>>>>>
> >>>>>>>>> I was wondering if you are picking up the 1.10.1 release? Master
> is speeding ahead and you were tracking fixes for 1.10.1 right?
> >>>>>>>>>
> >>>>>>>>> B.
> >>>>>>
> >>>>
> >>>
> >
>
>
Re: 1.10.1 Release?
Posted by Bolke de Bruin <bd...@gmail.com>.
we specifically remove timezone info to determine the next schedule. Ie. cron sets exact date times so tz info should not make sense. I’m going to have a look now.
> On 30 Oct 2018, at 19:01, Ash Berlin-Taylor <as...@apache.org> wrote:
>
> I've done a bit more digging - the issue is of our tz-aware handling inside following_schedule (and previous schedule) - causing it to loop.
>
> This section of the croniter docs seems relevant https://github.com/kiorky/croniter#about-dst
>
> Be sure to init your croniter instance with a TZ aware datetime for this to work !:
>>>> local_date = tz.localize(datetime(2017, 3, 26))
>>>> val = croniter('0 0 * * *', local_date).get_next(datetime)
>
> I think the problem is that we are _not_ passing a TZ aware dag in and we should be.
>
>> On 30 Oct 2018, at 17:35, Bolke de Bruin <bd...@gmail.com> wrote:
>>
>> Oh that’s a great environment to start digging. Thanks. I’ll have a look.
>>
>> B.
>>
>> Verstuurd vanaf mijn iPad
>>
>>> Op 30 okt. 2018 om 18:25 heeft Ash Berlin-Taylor <as...@apache.org> het volgende geschreven:
>>>
>>> This line in airflow.jobs (line 874 in my checkout) is causing the loop:
>>>
>>> last_run = dag.get_last_dagrun(session=session)
>>> if last_run and next_run_date:
>>> while next_run_date <= last_run.execution_date:
>>> next_run_date = dag.following_schedule(next_run_date)
>>>
>>>
>>>
>>>> On 30 Oct 2018, at 17:20, Ash Berlin-Taylor <as...@apache.org> wrote:
>>>>
>>>> Hi, kaczors on gitter has produced a minmal reproduction case: https://github.com/kaczors/airflow_1_10_tz_bug
>>>>
>>>> Rough repro steps: In a VM, with time syncing disabled, and configured with system timezone of Europe/Zurich (or any other CEST one) run
>>>>
>>>> - `date 10280250.00`
>>>> - initdb, start scheduler, webserver, enable dag etc.
>>>> - `date 10280259.00`
>>>> - wait 5-10 mins for scheduler to catch up
>>>> - After the on-the-hour task run the scheduler will spin up another process to parse the dag... and it never returns.
>>>>
>>>> I've only just managed to reproduce it, so haven't dug in to why yet. A quick hacky debug print shows something is stuck in an infinite loop.
>>>>
>>>> -ash
>>>>
>>>>> On 29 Oct 2018, at 17:59, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>>
>>>>> Can this be confirmed? Then I can have a look at it. Preferably with dag definition code.
>>>>>
>>>>> On the licensing requirements:
>>>>>
>>>>> 1. Indeed licensing header for markdown documents. It was suggested to use html comments. I’m not sure how that renders with others like PDF though.
>>>>> 2. The licensing notifications need to be tied to a specific version as licenses might change with versions.
>>>>>
>>>>> Cheers
>>>>> Bolke
>>>>>
>>>>> Verstuurd vanaf mijn iPad
>>>>>
>>>>>> Op 29 okt. 2018 om 12:39 heeft Ash Berlin-Taylor <as...@apache.org> het volgende geschreven:
>>>>>>
>>>>>> I was going to make a start on the release, but two people have reported that there might be an issue around non-UTC dags and the scheduler changing over from Summer time.
>>>>>>
>>>>>>> 08:45 Emmanuel> Hi there, we are currently experiencing a very strange issue : we have hourly DAGs with a start_date in a local timezone (not UTC) and since (Sunday) the last winter time change they don’t run anymore. Any idea ?
>>>>>>> 09:41 <Emmanuel> it impacted all our DAG that had a run at 3am (Europe/Paris), the exact time of winter time change :(
>>>>>>
>>>>>> I am going to take a look at this today and see if I can get to the bottom of it.
>>>>>>
>>>>>> Bolke: are there any outstanding tasks/issues that you know of that might slow down the vote for a 1.10.1? (i.e. did we sort of out all the licensing issues that were asked of us? I thought I read something about license declarations in markdown files?)
>>>>>>
>>>>>> -ash
>>>>>>
>>>>>>> On 28 Oct 2018, at 14:46, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>>>>
>>>>>>> I agree with that, but I would favor time based releases instead. We are again at the point that a release takes so much time that the gap is getting really big again. @ash why not start releasing now and move the remainder to 1.10.2? I dont think there are real blockers (although we might find them).
>>>>>>>
>>>>>>>
>>>>>>>> On 28 Oct 2018, at 15:35, airflowuser <ai...@protonmail.com.INVALID> wrote:
>>>>>>>>
>>>>>>>> I was really hoping that https://github.com/apache/incubator-airflow/pull/4069 will be merged into 1.10.1
>>>>>>>> Deleting dags was a highly requested feature for 1.10 - this can fix the problem with it.
>>>>>>>>
>>>>>>>>
>>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>>>>>> On Friday, October 26, 2018 6:12 PM, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Hey Ash,
>>>>>>>>>
>>>>>>>>> I was wondering if you are picking up the 1.10.1 release? Master is speeding ahead and you were tracking fixes for 1.10.1 right?
>>>>>>>>>
>>>>>>>>> B.
>>>>>>
>>>>
>>>
>
Re: 1.10.1 Release?
Posted by Bolke de Bruin <bd...@gmail.com>.
The fix is in master and should work across all DST changes. It will be included in 1.10.1.
B.
Sent from my iPhone
> On 5 Nov 2018, at 19:54, Dave Fisher <wa...@apache.org> wrote:
>
>
>
>> On 2018/10/28 00:09:05, Bolke de Bruin <bd...@gmail.com> wrote:
>> I wonder how to treat this:
>>
>> This is what I think happens (need to verify more, but I am pretty sure) the specified DAG should run every 5 minutes. At DST change (3AM -> 2AM)
>
> FYI - In the US the DST change is 2AM -> 1AM. Yes, TZ is hard stuff.
>
> we basically hit a schedule that we have already seen. 2AM -> 3AM has already happened. Obviously the intention is to run every 5 minutes. But what do we do with the execution_date? Is this still idempotent? Should we indeed reschedule?
>>
>> B.
>>
>>> On 30 Oct 2018, at 19:01, Ash Berlin-Taylor <as...@apache.org> wrote:
>>>
>>> I've done a bit more digging - the issue is of our tz-aware handling inside following_schedule (and previous schedule) - causing it to loop.
>>>
>>> This section of the croniter docs seems relevant https://github.com/kiorky/croniter#about-dst
>>>
>>> Be sure to init your croniter instance with a TZ aware datetime for this to work !:
>>>>>> local_date = tz.localize(datetime(2017, 3, 26))
>>>>>> val = croniter('0 0 * * *', local_date).get_next(datetime)
>>>
>>> I think the problem is that we are _not_ passing a TZ aware dag in and we should be.
>>>
>>>> On 30 Oct 2018, at 17:35, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>
>>>> Oh that’s a great environment to start digging. Thanks. I’ll have a look.
>>>>
>>>> B.
>>>>
>>>> Verstuurd vanaf mijn iPad
>>>>
>>>>> Op 30 okt. 2018 om 18:25 heeft Ash Berlin-Taylor <as...@apache.org> het volgende geschreven:
>>>>>
>>>>> This line in airflow.jobs (line 874 in my checkout) is causing the loop:
>>>>>
>>>>> last_run = dag.get_last_dagrun(session=session)
>>>>> if last_run and next_run_date:
>>>>> while next_run_date <= last_run.execution_date:
>>>>> next_run_date = dag.following_schedule(next_run_date)
>>>>>
>>>>>
>>>>>
>>>>>> On 30 Oct 2018, at 17:20, Ash Berlin-Taylor <as...@apache.org> wrote:
>>>>>>
>>>>>> Hi, kaczors on gitter has produced a minmal reproduction case: https://github.com/kaczors/airflow_1_10_tz_bug
>>>>>>
>>>>>> Rough repro steps: In a VM, with time syncing disabled, and configured with system timezone of Europe/Zurich (or any other CEST one) run
>>>>>>
>>>>>> - `date 10280250.00`
>>>>>> - initdb, start scheduler, webserver, enable dag etc.
>>>>>> - `date 10280259.00`
>>>>>> - wait 5-10 mins for scheduler to catch up
>>>>>> - After the on-the-hour task run the scheduler will spin up another process to parse the dag... and it never returns.
>>>>>>
>>>>>> I've only just managed to reproduce it, so haven't dug in to why yet. A quick hacky debug print shows something is stuck in an infinite loop.
>>>>>>
>>>>>> -ash
>>>>>>
>>>>>>> On 29 Oct 2018, at 17:59, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>>>>
>>>>>>> Can this be confirmed? Then I can have a look at it. Preferably with dag definition code.
>>>>>>>
>>>>>>> On the licensing requirements:
>>>>>>>
>>>>>>> 1. Indeed licensing header for markdown documents. It was suggested to use html comments. I’m not sure how that renders with others like PDF though.
>>>>>>> 2. The licensing notifications need to be tied to a specific version as licenses might change with versions.
>>>>>>>
>>>>>>> Cheers
>>>>>>> Bolke
>>>>>>>
>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>
>>>>>>>> Op 29 okt. 2018 om 12:39 heeft Ash Berlin-Taylor <as...@apache.org> het volgende geschreven:
>>>>>>>>
>>>>>>>> I was going to make a start on the release, but two people have reported that there might be an issue around non-UTC dags and the scheduler changing over from Summer time.
>>>>>>>>
>>>>>>>>> 08:45 Emmanuel> Hi there, we are currently experiencing a very strange issue : we have hourly DAGs with a start_date in a local timezone (not UTC) and since (Sunday) the last winter time change they don’t run anymore. Any idea ?
>>>>>>>>> 09:41 <Emmanuel> it impacted all our DAG that had a run at 3am (Europe/Paris), the exact time of winter time change :(
>>>>>>>>
>>>>>>>> I am going to take a look at this today and see if I can get to the bottom of it.
>>>>>>>>
>>>>>>>> Bolke: are there any outstanding tasks/issues that you know of that might slow down the vote for a 1.10.1? (i.e. did we sort of out all the licensing issues that were asked of us? I thought I read something about license declarations in markdown files?)
>>>>>>>>
>>>>>>>> -ash
>>>>>>>>
>>>>>>>>> On 28 Oct 2018, at 14:46, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> I agree with that, but I would favor time based releases instead. We are again at the point that a release takes so much time that the gap is getting really big again. @ash why not start releasing now and move the remainder to 1.10.2? I dont think there are real blockers (although we might find them).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On 28 Oct 2018, at 15:35, airflowuser <ai...@protonmail.com.INVALID> wrote:
>>>>>>>>>>
>>>>>>>>>> I was really hoping that https://github.com/apache/incubator-airflow/pull/4069 will be merged into 1.10.1
>>>>>>>>>> Deleting dags was a highly requested feature for 1.10 - this can fix the problem with it.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>>>>>>>> On Friday, October 26, 2018 6:12 PM, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hey Ash,
>>>>>>>>>>>
>>>>>>>>>>> I was wondering if you are picking up the 1.10.1 release? Master is speeding ahead and you were tracking fixes for 1.10.1 right?
>>>>>>>>>>>
>>>>>>>>>>> B.
>>>>>>>>
>>>>>>
>>>>>
>>>
>>
>>
Re: 1.10.1 Release?
Posted by Dave Fisher <wa...@apache.org>.
On 2018/10/28 00:09:05, Bolke de Bruin <bd...@gmail.com> wrote:
> I wonder how to treat this:
>
> This is what I think happens (need to verify more, but I am pretty sure) the specified DAG should run every 5 minutes. At DST change (3AM -> 2AM)
FYI - In the US the DST change is 2AM -> 1AM. Yes, TZ is hard stuff.
we basically hit a schedule that we have already seen. 2AM -> 3AM has already happened. Obviously the intention is to run every 5 minutes. But what do we do with the execution_date? Is this still idempotent? Should we indeed reschedule?
>
> B.
>
> > On 30 Oct 2018, at 19:01, Ash Berlin-Taylor <as...@apache.org> wrote:
> >
> > I've done a bit more digging - the issue is of our tz-aware handling inside following_schedule (and previous schedule) - causing it to loop.
> >
> > This section of the croniter docs seems relevant https://github.com/kiorky/croniter#about-dst
> >
> > Be sure to init your croniter instance with a TZ aware datetime for this to work !:
> >>>> local_date = tz.localize(datetime(2017, 3, 26))
> >>>> val = croniter('0 0 * * *', local_date).get_next(datetime)
> >
> > I think the problem is that we are _not_ passing a TZ aware dag in and we should be.
> >
> >> On 30 Oct 2018, at 17:35, Bolke de Bruin <bd...@gmail.com> wrote:
> >>
> >> Oh that’s a great environment to start digging. Thanks. I’ll have a look.
> >>
> >> B.
> >>
> >> Verstuurd vanaf mijn iPad
> >>
> >>> Op 30 okt. 2018 om 18:25 heeft Ash Berlin-Taylor <as...@apache.org> het volgende geschreven:
> >>>
> >>> This line in airflow.jobs (line 874 in my checkout) is causing the loop:
> >>>
> >>> last_run = dag.get_last_dagrun(session=session)
> >>> if last_run and next_run_date:
> >>> while next_run_date <= last_run.execution_date:
> >>> next_run_date = dag.following_schedule(next_run_date)
> >>>
> >>>
> >>>
> >>>> On 30 Oct 2018, at 17:20, Ash Berlin-Taylor <as...@apache.org> wrote:
> >>>>
> >>>> Hi, kaczors on gitter has produced a minmal reproduction case: https://github.com/kaczors/airflow_1_10_tz_bug
> >>>>
> >>>> Rough repro steps: In a VM, with time syncing disabled, and configured with system timezone of Europe/Zurich (or any other CEST one) run
> >>>>
> >>>> - `date 10280250.00`
> >>>> - initdb, start scheduler, webserver, enable dag etc.
> >>>> - `date 10280259.00`
> >>>> - wait 5-10 mins for scheduler to catch up
> >>>> - After the on-the-hour task run the scheduler will spin up another process to parse the dag... and it never returns.
> >>>>
> >>>> I've only just managed to reproduce it, so haven't dug in to why yet. A quick hacky debug print shows something is stuck in an infinite loop.
> >>>>
> >>>> -ash
> >>>>
> >>>>> On 29 Oct 2018, at 17:59, Bolke de Bruin <bd...@gmail.com> wrote:
> >>>>>
> >>>>> Can this be confirmed? Then I can have a look at it. Preferably with dag definition code.
> >>>>>
> >>>>> On the licensing requirements:
> >>>>>
> >>>>> 1. Indeed licensing header for markdown documents. It was suggested to use html comments. I’m not sure how that renders with others like PDF though.
> >>>>> 2. The licensing notifications need to be tied to a specific version as licenses might change with versions.
> >>>>>
> >>>>> Cheers
> >>>>> Bolke
> >>>>>
> >>>>> Verstuurd vanaf mijn iPad
> >>>>>
> >>>>>> Op 29 okt. 2018 om 12:39 heeft Ash Berlin-Taylor <as...@apache.org> het volgende geschreven:
> >>>>>>
> >>>>>> I was going to make a start on the release, but two people have reported that there might be an issue around non-UTC dags and the scheduler changing over from Summer time.
> >>>>>>
> >>>>>>> 08:45 Emmanuel> Hi there, we are currently experiencing a very strange issue : we have hourly DAGs with a start_date in a local timezone (not UTC) and since (Sunday) the last winter time change they don’t run anymore. Any idea ?
> >>>>>>> 09:41 <Emmanuel> it impacted all our DAG that had a run at 3am (Europe/Paris), the exact time of winter time change :(
> >>>>>>
> >>>>>> I am going to take a look at this today and see if I can get to the bottom of it.
> >>>>>>
> >>>>>> Bolke: are there any outstanding tasks/issues that you know of that might slow down the vote for a 1.10.1? (i.e. did we sort of out all the licensing issues that were asked of us? I thought I read something about license declarations in markdown files?)
> >>>>>>
> >>>>>> -ash
> >>>>>>
> >>>>>>> On 28 Oct 2018, at 14:46, Bolke de Bruin <bd...@gmail.com> wrote:
> >>>>>>>
> >>>>>>> I agree with that, but I would favor time based releases instead. We are again at the point that a release takes so much time that the gap is getting really big again. @ash why not start releasing now and move the remainder to 1.10.2? I dont think there are real blockers (although we might find them).
> >>>>>>>
> >>>>>>>
> >>>>>>>> On 28 Oct 2018, at 15:35, airflowuser <ai...@protonmail.com.INVALID> wrote:
> >>>>>>>>
> >>>>>>>> I was really hoping that https://github.com/apache/incubator-airflow/pull/4069 will be merged into 1.10.1
> >>>>>>>> Deleting dags was a highly requested feature for 1.10 - this can fix the problem with it.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> >>>>>>>>>> On Friday, October 26, 2018 6:12 PM, Bolke de Bruin <bd...@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>> Hey Ash,
> >>>>>>>>>
> >>>>>>>>> I was wondering if you are picking up the 1.10.1 release? Master is speeding ahead and you were tracking fixes for 1.10.1 right?
> >>>>>>>>>
> >>>>>>>>> B.
> >>>>>>
> >>>>
> >>>
> >
>
>
Re: 1.10.1 Release?
Posted by Bolke de Bruin <bd...@gmail.com>.
They are :-)
I might have to adjust some tests, let's see. Will do that tomorrow if required.
B.
> On 30 Oct 2018, at 21:53, Ash Berlin-Taylor <as...@apache.org> wrote:
>
> Fair :) Timezones are _hard_
>
> Giving it a look now.
>
> -ash
>
>> On 30 Oct 2018, at 20:50, Bolke de Bruin <bd...@gmail.com> wrote:
>>
>> The reason for not passing a TZ aware object is, is that many libraries make mistakes (pytz, arrow etc) when doing transitions hence to use of pendulum which seem most complete. I don’t know what croniter is relying on and I don’t want to find out ;-).
>>
>> B.
>>
>>> On 30 Oct 2018, at 21:13, Ash Berlin-Taylor <ash@firemirror.com <ma...@firemirror.com>> wrote:
>>>
>>> I think if we give croniter a tz-aware DT in the local tz it will deal with DST (i.e. will give 2:55 CEST followed by 2:00 CET) and then we convert it to UTC for return - but right now we are giving it a TZ-unaware local time.
>>>
>>> I think.
>>>
>>> Ash
>>>
>>> On 30 October 2018 19:40:27 GMT, Bolke de Bruin <bd...@gmail.com> wrote:
>>> I think we should use the UTC date for cron instead of the naive local date time. I will check of croniter implements this so we can rely on that.
>>>
>>> B.
>>>
>>> On 28 Oct 2018, at 02:09, Bolke de Bruin <bd...@gmail.com> wrote:
>>>
>>> I wonder how to treat this:
>>>
>>> This is what I think happens (need to verify more, but I am pretty sure) the specified DAG should run every 5 minutes. At DST change (3AM -> 2AM) we basically hit a schedule that we have already seen. 2AM -> 3AM has already happened. Obviously the intention is to run every 5 minutes. But what do we do with the execution_date? Is this still idempotent? Should we indeed reschedule?
>>>
>>> B.
>>>
>>> On 30 Oct 2018, at 19:01, Ash Berlin-Taylor <as...@apache.org> wrote:
>>>
>>> I've done a bit more digging - the issue is of our tz-aware handling inside following_schedule (and previous schedule) - causing it to loop.
>>>
>>> This section of the croniter docs seems relevant https://github.com/kiorky/croniter#about-dst <https://github.com/kiorky/croniter#about-dst><https://github.com/kiorky/croniter#about-dst <https://github.com/kiorky/croniter#about-dst>>
>>>
>>> Be sure to init your croniter instance with a TZ aware datetime for this to work !:
>>> local_date = tz.localize(datetime(2017, 3, 26))
>>> val = croniter('0 0 * * *', local_date).get_next(datetime)
>>>
>>> I think the problem is that we are _not_ passing a TZ aware dag in and we should be.
>>>
>>> On 30 Oct 2018, at 17:35, Bolke de Bruin <bdbruin@gmail.com <ma...@gmail.com>> wrote:
>>>
>>> Oh that’s a great environment to start digging. Thanks. I’ll have a look.
>>>
>>> B.
>>>
>>> Verstuurd vanaf mijn iPad
>>>
>>> Op 30 okt. 2018 om 18:25 heeft Ash Berlin-Taylor <ash@apache.org <ma...@apache.org>> het volgende geschreven:
>>>
>>> This line in airflow.jobs (line 874 in my checkout) is causing the loop:
>>>
>>> last_run = dag.get_last_dagrun(session=session)
>>> if last_run and next_run_date:
>>> while next_run_date <= last_run.execution_date:
>>> next_run_date = dag.following_schedule(next_run_date)
>>>
>>>
>>>
>>> On 30 Oct 2018, at 17:20, Ash Berlin-Taylor <ash@apache.org <ma...@apache.org>> wrote:
>>>
>>> Hi, kaczors on gitter has produced a minmal reproduction case: https://github.com/kaczors/airflow_1_10_tz_bug <https://github.com/kaczors/airflow_1_10_tz_bug> <https://github.com/kaczors/airflow_1_10_tz_bug <https://github.com/kaczors/airflow_1_10_tz_bug>>
>>>
>>> Rough repro steps: In a VM, with time syncing disabled, and configured with system timezone of Europe/Zurich (or any other CEST one) run
>>>
>>> - `date 10280250.00`
>>> - initdb, start scheduler, webserver, enable dag etc.
>>> - `date 10280259.00`
>>> - wait 5-10 mins for scheduler to catch up
>>> - After the on-the-hour task run the scheduler will spin up another process to parse the dag... and it never returns.
>>>
>>> I've only just managed to reproduce it, so haven't dug in to why yet. A quick hacky debug print shows something is stuck in an infinite loop.
>>>
>>> -ash
>>>
>>> On 29 Oct 2018, at 17:59, Bolke de Bruin <bdbruin@gmail.com <ma...@gmail.com>> wrote:
>>>
>>> Can this be confirmed? Then I can have a look at it. Preferably with dag definition code.
>>>
>>> On the licensing requirements:
>>>
>>> 1. Indeed licensing header for markdown documents. It was suggested to use html comments. I’m not sure how that renders with others like PDF though.
>>> 2. The licensing notifications need to be tied to a specific version as licenses might change with versions.
>>>
>>> Cheers
>>> Bolke
>>>
>>> Verstuurd vanaf mijn iPad
>>>
>>> Op 29 okt. 2018 om 12:39 heeft Ash Berlin-Taylor <ash@apache.org <ma...@apache.org>> het volgende geschreven:
>>>
>>> I was going to make a start on the release, but two people have reported that there might be an issue around non-UTC dags and the scheduler changing over from Summer time.
>>>
>>> 08:45 Emmanuel> Hi there, we are currently experiencing a very strange issue : we have hourly DAGs with a start_date in a local timezone (not UTC) and since (Sunday) the last winter time change they don’t run anymore. Any idea ?
>>> 09:41 <Emmanuel> it impacted all our DAG that had a run at 3am (Europe/Paris), the exact time of winter time change :(
>>>
>>> I am going to take a look at this today and see if I can get to the bottom of it.
>>>
>>> Bolke: are there any outstanding tasks/issues that you know of that might slow down the vote for a 1.10.1? (i.e. did we sort of out all the licensing issues that were asked of us? I thought I read something about license declarations in markdown files?)
>>>
>>> -ash
>>>
>>> On 28 Oct 2018, at 14:46, Bolke de Bruin <bdbruin@gmail.com <ma...@gmail.com>> wrote:
>>>
>>> I agree with that, but I would favor time based releases instead. We are again at the point that a release takes so much time that the gap is getting really big again. @ash why not start releasing now and move the remainder to 1.10.2? I dont think there are real blockers (although we might find them).
>>>
>>>
>>> On 28 Oct 2018, at 15:35, airflowuser <airflowuser@protonmail.com.INVALID <ma...@protonmail.com.INVALID>> wrote:
>>>
>>> I was really hoping that https://github.com/apache/incubator-airflow/pull/4069 <https://github.com/apache/incubator-airflow/pull/4069><https://github.com/apache/incubator-airflow/pull/4069 <https://github.com/apache/incubator-airflow/pull/4069>> will be merged into 1.10.1
>>> Deleting dags was a highly requested feature for 1.10 - this can fix the problem with it.
>>>
>>>
>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>> On Friday, October 26, 2018 6:12 PM, Bolke de Bruin <bd...@gmail.com> wrote:
>>>
>>> Hey Ash,
>>>
>>> I was wondering if you are picking up the 1.10.1 release? Master is speeding ahead and you were tracking fixes for 1.10.1 right?
>>>
>>> B.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>
Re: 1.10.1 Release?
Posted by Ash Berlin-Taylor <as...@apache.org>.
Fair :) Timezones are _hard_
Giving it a look now.
-ash
> On 30 Oct 2018, at 20:50, Bolke de Bruin <bd...@gmail.com> wrote:
>
> The reason for not passing a TZ aware object is, is that many libraries make mistakes (pytz, arrow etc) when doing transitions hence to use of pendulum which seem most complete. I don’t know what croniter is relying on and I don’t want to find out ;-).
>
> B.
>
>> On 30 Oct 2018, at 21:13, Ash Berlin-Taylor <ash@firemirror.com <ma...@firemirror.com>> wrote:
>>
>> I think if we give croniter a tz-aware DT in the local tz it will deal with DST (i.e. will give 2:55 CEST followed by 2:00 CET) and then we convert it to UTC for return - but right now we are giving it a TZ-unaware local time.
>>
>> I think.
>>
>> Ash
>>
>> On 30 October 2018 19:40:27 GMT, Bolke de Bruin <bd...@gmail.com> wrote:
>> I think we should use the UTC date for cron instead of the naive local date time. I will check of croniter implements this so we can rely on that.
>>
>> B.
>>
>> On 28 Oct 2018, at 02:09, Bolke de Bruin <bd...@gmail.com> wrote:
>>
>> I wonder how to treat this:
>>
>> This is what I think happens (need to verify more, but I am pretty sure) the specified DAG should run every 5 minutes. At DST change (3AM -> 2AM) we basically hit a schedule that we have already seen. 2AM -> 3AM has already happened. Obviously the intention is to run every 5 minutes. But what do we do with the execution_date? Is this still idempotent? Should we indeed reschedule?
>>
>> B.
>>
>> On 30 Oct 2018, at 19:01, Ash Berlin-Taylor <as...@apache.org> wrote:
>>
>> I've done a bit more digging - the issue is of our tz-aware handling inside following_schedule (and previous schedule) - causing it to loop.
>>
>> This section of the croniter docs seems relevant https://github.com/kiorky/croniter#about-dst <https://github.com/kiorky/croniter#about-dst><https://github.com/kiorky/croniter#about-dst <https://github.com/kiorky/croniter#about-dst>>
>>
>> Be sure to init your croniter instance with a TZ aware datetime for this to work !:
>> local_date = tz.localize(datetime(2017, 3, 26))
>> val = croniter('0 0 * * *', local_date).get_next(datetime)
>>
>> I think the problem is that we are _not_ passing a TZ aware dag in and we should be.
>>
>> On 30 Oct 2018, at 17:35, Bolke de Bruin <bdbruin@gmail.com <ma...@gmail.com>> wrote:
>>
>> Oh that’s a great environment to start digging. Thanks. I’ll have a look.
>>
>> B.
>>
>> Verstuurd vanaf mijn iPad
>>
>> Op 30 okt. 2018 om 18:25 heeft Ash Berlin-Taylor <ash@apache.org <ma...@apache.org>> het volgende geschreven:
>>
>> This line in airflow.jobs (line 874 in my checkout) is causing the loop:
>>
>> last_run = dag.get_last_dagrun(session=session)
>> if last_run and next_run_date:
>> while next_run_date <= last_run.execution_date:
>> next_run_date = dag.following_schedule(next_run_date)
>>
>>
>>
>> On 30 Oct 2018, at 17:20, Ash Berlin-Taylor <ash@apache.org <ma...@apache.org>> wrote:
>>
>> Hi, kaczors on gitter has produced a minmal reproduction case: https://github.com/kaczors/airflow_1_10_tz_bug <https://github.com/kaczors/airflow_1_10_tz_bug> <https://github.com/kaczors/airflow_1_10_tz_bug <https://github.com/kaczors/airflow_1_10_tz_bug>>
>>
>> Rough repro steps: In a VM, with time syncing disabled, and configured with system timezone of Europe/Zurich (or any other CEST one) run
>>
>> - `date 10280250.00`
>> - initdb, start scheduler, webserver, enable dag etc.
>> - `date 10280259.00`
>> - wait 5-10 mins for scheduler to catch up
>> - After the on-the-hour task run the scheduler will spin up another process to parse the dag... and it never returns.
>>
>> I've only just managed to reproduce it, so haven't dug in to why yet. A quick hacky debug print shows something is stuck in an infinite loop.
>>
>> -ash
>>
>> On 29 Oct 2018, at 17:59, Bolke de Bruin <bdbruin@gmail.com <ma...@gmail.com>> wrote:
>>
>> Can this be confirmed? Then I can have a look at it. Preferably with dag definition code.
>>
>> On the licensing requirements:
>>
>> 1. Indeed licensing header for markdown documents. It was suggested to use html comments. I’m not sure how that renders with others like PDF though.
>> 2. The licensing notifications need to be tied to a specific version as licenses might change with versions.
>>
>> Cheers
>> Bolke
>>
>> Verstuurd vanaf mijn iPad
>>
>> Op 29 okt. 2018 om 12:39 heeft Ash Berlin-Taylor <ash@apache.org <ma...@apache.org>> het volgende geschreven:
>>
>> I was going to make a start on the release, but two people have reported that there might be an issue around non-UTC dags and the scheduler changing over from Summer time.
>>
>> 08:45 Emmanuel> Hi there, we are currently experiencing a very strange issue : we have hourly DAGs with a start_date in a local timezone (not UTC) and since (Sunday) the last winter time change they don’t run anymore. Any idea ?
>> 09:41 <Emmanuel> it impacted all our DAG that had a run at 3am (Europe/Paris), the exact time of winter time change :(
>>
>> I am going to take a look at this today and see if I can get to the bottom of it.
>>
>> Bolke: are there any outstanding tasks/issues that you know of that might slow down the vote for a 1.10.1? (i.e. did we sort of out all the licensing issues that were asked of us? I thought I read something about license declarations in markdown files?)
>>
>> -ash
>>
>> On 28 Oct 2018, at 14:46, Bolke de Bruin <bdbruin@gmail.com <ma...@gmail.com>> wrote:
>>
>> I agree with that, but I would favor time based releases instead. We are again at the point that a release takes so much time that the gap is getting really big again. @ash why not start releasing now and move the remainder to 1.10.2? I dont think there are real blockers (although we might find them).
>>
>>
>> On 28 Oct 2018, at 15:35, airflowuser <airflowuser@protonmail.com.INVALID <ma...@protonmail.com.INVALID>> wrote:
>>
>> I was really hoping that https://github.com/apache/incubator-airflow/pull/4069 <https://github.com/apache/incubator-airflow/pull/4069><https://github.com/apache/incubator-airflow/pull/4069 <https://github.com/apache/incubator-airflow/pull/4069>> will be merged into 1.10.1
>> Deleting dags was a highly requested feature for 1.10 - this can fix the problem with it.
>>
>>
>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>> On Friday, October 26, 2018 6:12 PM, Bolke de Bruin <bd...@gmail.com> wrote:
>>
>> Hey Ash,
>>
>> I was wondering if you are picking up the 1.10.1 release? Master is speeding ahead and you were tracking fixes for 1.10.1 right?
>>
>> B.
>>
>>
>>
>>
>>
>>
>>
>> --
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: 1.10.1 Release?
Posted by Bolke de Bruin <bd...@gmail.com>.
The reason for not passing a TZ aware object is, is that many libraries make mistakes (pytz, arrow etc) when doing transitions hence to use of pendulum which seem most complete. I don’t know what croniter is relying on and I don’t want to find out ;-).
B.
> On 30 Oct 2018, at 21:13, Ash Berlin-Taylor <as...@firemirror.com> wrote:
>
> I think if we give croniter a tz-aware DT in the local tz it will deal with DST (i.e. will give 2:55 CEST followed by 2:00 CET) and then we convert it to UTC for return - but right now we are giving it a TZ-unaware local time.
>
> I think.
>
> Ash
>
> On 30 October 2018 19:40:27 GMT, Bolke de Bruin <bd...@gmail.com> wrote:
> I think we should use the UTC date for cron instead of the naive local date time. I will check of croniter implements this so we can rely on that.
>
> B.
>
> On 28 Oct 2018, at 02:09, Bolke de Bruin <bd...@gmail.com> wrote:
>
> I wonder how to treat this:
>
> This is what I think happens (need to verify more, but I am pretty sure) the specified DAG should run every 5 minutes. At DST change (3AM -> 2AM) we basically hit a schedule that we have already seen. 2AM -> 3AM has already happened. Obviously the intention is to run every 5 minutes. But what do we do with the execution_date? Is this still idempotent? Should we indeed reschedule?
>
> B.
>
> On 30 Oct 2018, at 19:01, Ash Berlin-Taylor <as...@apache.org> wrote:
>
> I've done a bit more digging - the issue is of our tz-aware handling inside following_schedule (and previous schedule) - causing it to loop.
>
> This section of the croniter docs seems relevant https://github.com/kiorky/croniter#about-dst <https://github.com/kiorky/croniter#about-dst>
>
> Be sure to init your croniter instance with a TZ aware datetime for this to work !:
> local_date = tz.localize(datetime(2017, 3, 26))
> val = croniter('0 0 * * *', local_date).get_next(datetime)
>
> I think the problem is that we are _not_ passing a TZ aware dag in and we should be.
>
> On 30 Oct 2018, at 17:35, Bolke de Bruin <bd...@gmail.com> wrote:
>
> Oh that’s a great environment to start digging. Thanks. I’ll have a look.
>
> B.
>
> Verstuurd vanaf mijn iPad
>
> Op 30 okt. 2018 om 18:25 heeft Ash Berlin-Taylor <as...@apache.org> het volgende geschreven:
>
> This line in airflow.jobs (line 874 in my checkout) is causing the loop:
>
> last_run = dag.get_last_dagrun(session=session)
> if last_run and next_run_date:
> while next_run_date <= last_run.execution_date:
> next_run_date = dag.following_schedule(next_run_date)
>
>
>
> On 30 Oct 2018, at 17:20, Ash Berlin-Taylor <as...@apache.org> wrote:
>
> Hi, kaczors on gitter has produced a minmal reproduction case: https://github.com/kaczors/airflow_1_10_tz_bug <https://github.com/kaczors/airflow_1_10_tz_bug>
>
> Rough repro steps: In a VM, with time syncing disabled, and configured with system timezone of Europe/Zurich (or any other CEST one) run
>
> - `date 10280250.00`
> - initdb, start scheduler, webserver, enable dag etc.
> - `date 10280259.00`
> - wait 5-10 mins for scheduler to catch up
> - After the on-the-hour task run the scheduler will spin up another process to parse the dag... and it never returns.
>
> I've only just managed to reproduce it, so haven't dug in to why yet. A quick hacky debug print shows something is stuck in an infinite loop.
>
> -ash
>
> On 29 Oct 2018, at 17:59, Bolke de Bruin <bd...@gmail.com> wrote:
>
> Can this be confirmed? Then I can have a look at it. Preferably with dag definition code.
>
> On the licensing requirements:
>
> 1. Indeed licensing header for markdown documents. It was suggested to use html comments. I’m not sure how that renders with others like PDF though.
> 2. The licensing notifications need to be tied to a specific version as licenses might change with versions.
>
> Cheers
> Bolke
>
> Verstuurd vanaf mijn iPad
>
> Op 29 okt. 2018 om 12:39 heeft Ash Berlin-Taylor <as...@apache.org> het volgende geschreven:
>
> I was going to make a start on the release, but two people have reported that there might be an issue around non-UTC dags and the scheduler changing over from Summer time.
>
> 08:45 Emmanuel> Hi there, we are currently experiencing a very strange issue : we have hourly DAGs with a start_date in a local timezone (not UTC) and since (Sunday) the last winter time change they don’t run anymore. Any idea ?
> 09:41 <Emmanuel> it impacted all our DAG that had a run at 3am (Europe/Paris), the exact time of winter time change :(
>
> I am going to take a look at this today and see if I can get to the bottom of it.
>
> Bolke: are there any outstanding tasks/issues that you know of that might slow down the vote for a 1.10.1? (i.e. did we sort of out all the licensing issues that were asked of us? I thought I read something about license declarations in markdown files?)
>
> -ash
>
> On 28 Oct 2018, at 14:46, Bolke de Bruin <bd...@gmail.com> wrote:
>
> I agree with that, but I would favor time based releases instead. We are again at the point that a release takes so much time that the gap is getting really big again. @ash why not start releasing now and move the remainder to 1.10.2? I dont think there are real blockers (although we might find them).
>
>
> On 28 Oct 2018, at 15:35, airflowuser <ai...@protonmail.com.INVALID> wrote:
>
> I was really hoping that https://github.com/apache/incubator-airflow/pull/4069 <https://github.com/apache/incubator-airflow/pull/4069> will be merged into 1.10.1
> Deleting dags was a highly requested feature for 1.10 - this can fix the problem with it.
>
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Friday, October 26, 2018 6:12 PM, Bolke de Bruin <bd...@gmail.com> wrote:
>
> Hey Ash,
>
> I was wondering if you are picking up the 1.10.1 release? Master is speeding ahead and you were tracking fixes for 1.10.1 right?
>
> B.
>
>
>
>
>
>
>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: 1.10.1 Release?
Posted by Bolke de Bruin <bd...@gmail.com>.
Patch available at:
https://github.com/apache/incubator-airflow/pull/4117
please test.
B.
> On 30 Oct 2018, at 21:14, Bolke de Bruin <bd...@gmail.com> wrote:
>
> We should just pass it the UTC date (we should never use local time except at the user interface). I’m testing a patch right now.
>
> B.
>
>> On 30 Oct 2018, at 21:13, Ash Berlin-Taylor <ash@firemirror.com <ma...@firemirror.com>> wrote:
>>
>> I think if we give croniter a tz-aware DT in the local tz it will deal with DST (i.e. will give 2:55 CEST followed by 2:00 CET) and then we convert it to UTC for return - but right now we are giving it a TZ-unaware local time.
>>
>> I think.
>>
>> Ash
>>
>> On 30 October 2018 19:40:27 GMT, Bolke de Bruin <bdbruin@gmail.com <ma...@gmail.com>> wrote:
>> I think we should use the UTC date for cron instead of the naive local date time. I will check of croniter implements this so we can rely on that.
>>
>> B.
>>
>> On 28 Oct 2018, at 02:09, Bolke de Bruin <bdbruin@gmail.com <ma...@gmail.com>> wrote:
>>
>> I wonder how to treat this:
>>
>> This is what I think happens (need to verify more, but I am pretty sure) the specified DAG should run every 5 minutes. At DST change (3AM -> 2AM) we basically hit a schedule that we have already seen. 2AM -> 3AM has already happened. Obviously the intention is to run every 5 minutes. But what do we do with the execution_date? Is this still idempotent? Should we indeed reschedule?
>>
>> B.
>>
>> On 30 Oct 2018, at 19:01, Ash Berlin-Taylor <ash@apache.org <ma...@apache.org>> wrote:
>>
>> I've done a bit more digging - the issue is of our tz-aware handling inside following_schedule (and previous schedule) - causing it to loop.
>>
>> This section of the croniter docs seems relevant https://github.com/kiorky/croniter#about-dst <https://github.com/kiorky/croniter#about-dst>
>>
>> Be sure to init your croniter instance with a TZ aware datetime for this to work !:
>> local_date = tz.localize(datetime(2017, 3, 26))
>> val = croniter('0 0 * * *', local_date).get_next(datetime)
>>
>> I think the problem is that we are _not_ passing a TZ aware dag in and we should be.
>>
>> On 30 Oct 2018, at 17:35, Bolke de Bruin <bdbruin@gmail.com <ma...@gmail.com>> wrote:
>>
>> Oh that’s a great environment to start digging. Thanks. I’ll have a look.
>>
>> B.
>>
>> Verstuurd vanaf mijn iPad
>>
>> Op 30 okt. 2018 om 18:25 heeft Ash Berlin-Taylor <ash@apache.org <ma...@apache.org>> het volgende geschreven:
>>
>> This line in airflow.jobs (line 874 in my checkout) is causing the loop:
>>
>> last_run = dag.get_last_dagrun(session=session)
>> if last_run and next_run_date:
>> while next_run_date <= last_run.execution_date:
>> next_run_date = dag.following_schedule(next_run_date)
>>
>>
>>
>> On 30 Oct 2018, at 17:20, Ash Berlin-Taylor <ash@apache.org <ma...@apache.org>> wrote:
>>
>> Hi, kaczors on gitter has produced a minmal reproduction case: https://github.com/kaczors/airflow_1_10_tz_bug <https://github.com/kaczors/airflow_1_10_tz_bug>
>>
>> Rough repro steps: In a VM, with time syncing disabled, and configured with system timezone of Europe/Zurich (or any other CEST one) run
>>
>> - `date 10280250.00`
>> - initdb, start scheduler, webserver, enable dag etc.
>> - `date 10280259.00`
>> - wait 5-10 mins for scheduler to catch up
>> - After the on-the-hour task run the scheduler will spin up another process to parse the dag... and it never returns.
>>
>> I've only just managed to reproduce it, so haven't dug in to why yet. A quick hacky debug print shows something is stuck in an infinite loop.
>>
>> -ash
>>
>> On 29 Oct 2018, at 17:59, Bolke de Bruin <bdbruin@gmail.com <ma...@gmail.com>> wrote:
>>
>> Can this be confirmed? Then I can have a look at it. Preferably with dag definition code.
>>
>> On the licensing requirements:
>>
>> 1. Indeed licensing header for markdown documents. It was suggested to use html comments. I’m not sure how that renders with others like PDF though.
>> 2. The licensing notifications need to be tied to a specific version as licenses might change with versions.
>>
>> Cheers
>> Bolke
>>
>> Verstuurd vanaf mijn iPad
>>
>> Op 29 okt. 2018 om 12:39 heeft Ash Berlin-Taylor <ash@apache.org <ma...@apache.org>> het volgende geschreven:
>>
>> I was going to make a start on the release, but two people have reported that there might be an issue around non-UTC dags and the scheduler changing over from Summer time.
>>
>> 08:45 Emmanuel> Hi there, we are currently experiencing a very strange issue : we have hourly DAGs with a start_date in a local timezone (not UTC) and since (Sunday) the last winter time change they don’t run anymore. Any idea ?
>> 09:41 <Emmanuel> it impacted all our DAG that had a run at 3am (Europe/Paris), the exact time of winter time change :(
>>
>> I am going to take a look at this today and see if I can get to the bottom of it.
>>
>> Bolke: are there any outstanding tasks/issues that you know of that might slow down the vote for a 1.10.1? (i.e. did we sort of out all the licensing issues that were asked of us? I thought I read something about license declarations in markdown files?)
>>
>> -ash
>>
>> On 28 Oct 2018, at 14:46, Bolke de Bruin <bdbruin@gmail.com <ma...@gmail.com>> wrote:
>>
>> I agree with that, but I would favor time based releases instead. We are again at the point that a release takes so much time that the gap is getting really big again. @ash why not start releasing now and move the remainder to 1.10.2? I dont think there are real blockers (although we might find them).
>>
>>
>> On 28 Oct 2018, at 15:35, airflowuser <airflowuser@protonmail.com.INVALID <ma...@protonmail.com.INVALID>> wrote:
>>
>> I was really hoping that https://github.com/apache/incubator-airflow/pull/4069 <https://github.com/apache/incubator-airflow/pull/4069> will be merged into 1.10.1
>> Deleting dags was a highly requested feature for 1.10 - this can fix the problem with it.
>>
>>
>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>> On Friday, October 26, 2018 6:12 PM, Bolke de Bruin <bdbruin@gmail.com <ma...@gmail.com>> wrote:
>>
>> Hey Ash,
>>
>> I was wondering if you are picking up the 1.10.1 release? Master is speeding ahead and you were tracking fixes for 1.10.1 right?
>>
>> B.
>>
>>
>>
>>
>>
>>
>>
>> --
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>
Re: 1.10.1 Release?
Posted by Bolke de Bruin <bd...@gmail.com>.
We should just pass it the UTC date (we should never use local time except at the user interface). I’m testing a patch right now.
B.
> On 30 Oct 2018, at 21:13, Ash Berlin-Taylor <as...@firemirror.com> wrote:
>
> I think if we give croniter a tz-aware DT in the local tz it will deal with DST (i.e. will give 2:55 CEST followed by 2:00 CET) and then we convert it to UTC for return - but right now we are giving it a TZ-unaware local time.
>
> I think.
>
> Ash
>
> On 30 October 2018 19:40:27 GMT, Bolke de Bruin <bd...@gmail.com> wrote:
> I think we should use the UTC date for cron instead of the naive local date time. I will check of croniter implements this so we can rely on that.
>
> B.
>
> On 28 Oct 2018, at 02:09, Bolke de Bruin <bd...@gmail.com> wrote:
>
> I wonder how to treat this:
>
> This is what I think happens (need to verify more, but I am pretty sure) the specified DAG should run every 5 minutes. At DST change (3AM -> 2AM) we basically hit a schedule that we have already seen. 2AM -> 3AM has already happened. Obviously the intention is to run every 5 minutes. But what do we do with the execution_date? Is this still idempotent? Should we indeed reschedule?
>
> B.
>
> On 30 Oct 2018, at 19:01, Ash Berlin-Taylor <as...@apache.org> wrote:
>
> I've done a bit more digging - the issue is of our tz-aware handling inside following_schedule (and previous schedule) - causing it to loop.
>
> This section of the croniter docs seems relevant https://github.com/kiorky/croniter#about-dst <https://github.com/kiorky/croniter#about-dst>
>
> Be sure to init your croniter instance with a TZ aware datetime for this to work !:
> local_date = tz.localize(datetime(2017, 3, 26))
> val = croniter('0 0 * * *', local_date).get_next(datetime)
>
> I think the problem is that we are _not_ passing a TZ aware dag in and we should be.
>
> On 30 Oct 2018, at 17:35, Bolke de Bruin <bd...@gmail.com> wrote:
>
> Oh that’s a great environment to start digging. Thanks. I’ll have a look.
>
> B.
>
> Verstuurd vanaf mijn iPad
>
> Op 30 okt. 2018 om 18:25 heeft Ash Berlin-Taylor <as...@apache.org> het volgende geschreven:
>
> This line in airflow.jobs (line 874 in my checkout) is causing the loop:
>
> last_run = dag.get_last_dagrun(session=session)
> if last_run and next_run_date:
> while next_run_date <= last_run.execution_date:
> next_run_date = dag.following_schedule(next_run_date)
>
>
>
> On 30 Oct 2018, at 17:20, Ash Berlin-Taylor <as...@apache.org> wrote:
>
> Hi, kaczors on gitter has produced a minmal reproduction case: https://github.com/kaczors/airflow_1_10_tz_bug <https://github.com/kaczors/airflow_1_10_tz_bug>
>
> Rough repro steps: In a VM, with time syncing disabled, and configured with system timezone of Europe/Zurich (or any other CEST one) run
>
> - `date 10280250.00`
> - initdb, start scheduler, webserver, enable dag etc.
> - `date 10280259.00`
> - wait 5-10 mins for scheduler to catch up
> - After the on-the-hour task run the scheduler will spin up another process to parse the dag... and it never returns.
>
> I've only just managed to reproduce it, so haven't dug in to why yet. A quick hacky debug print shows something is stuck in an infinite loop.
>
> -ash
>
> On 29 Oct 2018, at 17:59, Bolke de Bruin <bd...@gmail.com> wrote:
>
> Can this be confirmed? Then I can have a look at it. Preferably with dag definition code.
>
> On the licensing requirements:
>
> 1. Indeed licensing header for markdown documents. It was suggested to use html comments. I’m not sure how that renders with others like PDF though.
> 2. The licensing notifications need to be tied to a specific version as licenses might change with versions.
>
> Cheers
> Bolke
>
> Verstuurd vanaf mijn iPad
>
> Op 29 okt. 2018 om 12:39 heeft Ash Berlin-Taylor <as...@apache.org> het volgende geschreven:
>
> I was going to make a start on the release, but two people have reported that there might be an issue around non-UTC dags and the scheduler changing over from Summer time.
>
> 08:45 Emmanuel> Hi there, we are currently experiencing a very strange issue : we have hourly DAGs with a start_date in a local timezone (not UTC) and since (Sunday) the last winter time change they don’t run anymore. Any idea ?
> 09:41 <Emmanuel> it impacted all our DAG that had a run at 3am (Europe/Paris), the exact time of winter time change :(
>
> I am going to take a look at this today and see if I can get to the bottom of it.
>
> Bolke: are there any outstanding tasks/issues that you know of that might slow down the vote for a 1.10.1? (i.e. did we sort of out all the licensing issues that were asked of us? I thought I read something about license declarations in markdown files?)
>
> -ash
>
> On 28 Oct 2018, at 14:46, Bolke de Bruin <bd...@gmail.com> wrote:
>
> I agree with that, but I would favor time based releases instead. We are again at the point that a release takes so much time that the gap is getting really big again. @ash why not start releasing now and move the remainder to 1.10.2? I dont think there are real blockers (although we might find them).
>
>
> On 28 Oct 2018, at 15:35, airflowuser <ai...@protonmail.com.INVALID> wrote:
>
> I was really hoping that https://github.com/apache/incubator-airflow/pull/4069 <https://github.com/apache/incubator-airflow/pull/4069> will be merged into 1.10.1
> Deleting dags was a highly requested feature for 1.10 - this can fix the problem with it.
>
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Friday, October 26, 2018 6:12 PM, Bolke de Bruin <bd...@gmail.com> wrote:
>
> Hey Ash,
>
> I was wondering if you are picking up the 1.10.1 release? Master is speeding ahead and you were tracking fixes for 1.10.1 right?
>
> B.
>
>
>
>
>
>
>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: 1.10.1 Release?
Posted by Ash Berlin-Taylor <as...@firemirror.com>.
I think if we give croniter a tz-aware DT in the local tz it will deal with DST (i.e. will give 2:55 CEST followed by 2:00 CET) and then we convert it to UTC for return - but right now we are giving it a TZ-unaware local time.
I think.
Ash
On 30 October 2018 19:40:27 GMT, Bolke de Bruin <bd...@gmail.com> wrote:
>I think we should use the UTC date for cron instead of the naive local
>date time. I will check of croniter implements this so we can rely on
>that.
>
>B.
>
>> On 28 Oct 2018, at 02:09, Bolke de Bruin <bd...@gmail.com> wrote:
>>
>> I wonder how to treat this:
>>
>> This is what I think happens (need to verify more, but I am pretty
>sure) the specified DAG should run every 5 minutes. At DST change (3AM
>-> 2AM) we basically hit a schedule that we have already seen. 2AM ->
>3AM has already happened. Obviously the intention is to run every 5
>minutes. But what do we do with the execution_date? Is this still
>idempotent? Should we indeed reschedule?
>>
>> B.
>>
>>> On 30 Oct 2018, at 19:01, Ash Berlin-Taylor <as...@apache.org> wrote:
>>>
>>> I've done a bit more digging - the issue is of our tz-aware handling
>inside following_schedule (and previous schedule) - causing it to loop.
>>>
>>> This section of the croniter docs seems relevant
>https://github.com/kiorky/croniter#about-dst
>>>
>>> Be sure to init your croniter instance with a TZ aware datetime
>for this to work !:
>>>>>> local_date = tz.localize(datetime(2017, 3, 26))
>>>>>> val = croniter('0 0 * * *', local_date).get_next(datetime)
>>>
>>> I think the problem is that we are _not_ passing a TZ aware dag in
>and we should be.
>>>
>>>> On 30 Oct 2018, at 17:35, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>
>>>> Oh that’s a great environment to start digging. Thanks. I’ll have a
>look.
>>>>
>>>> B.
>>>>
>>>> Verstuurd vanaf mijn iPad
>>>>
>>>>> Op 30 okt. 2018 om 18:25 heeft Ash Berlin-Taylor <as...@apache.org>
>het volgende geschreven:
>>>>>
>>>>> This line in airflow.jobs (line 874 in my checkout) is causing the
>loop:
>>>>>
>>>>> last_run = dag.get_last_dagrun(session=session)
>>>>> if last_run and next_run_date:
>>>>> while next_run_date <= last_run.execution_date:
>>>>> next_run_date =
>dag.following_schedule(next_run_date)
>>>>>
>>>>>
>>>>>
>>>>>> On 30 Oct 2018, at 17:20, Ash Berlin-Taylor <as...@apache.org>
>wrote:
>>>>>>
>>>>>> Hi, kaczors on gitter has produced a minmal reproduction case:
>https://github.com/kaczors/airflow_1_10_tz_bug
>>>>>>
>>>>>> Rough repro steps: In a VM, with time syncing disabled, and
>configured with system timezone of Europe/Zurich (or any other CEST
>one) run
>>>>>>
>>>>>> - `date 10280250.00`
>>>>>> - initdb, start scheduler, webserver, enable dag etc.
>>>>>> - `date 10280259.00`
>>>>>> - wait 5-10 mins for scheduler to catch up
>>>>>> - After the on-the-hour task run the scheduler will spin up
>another process to parse the dag... and it never returns.
>>>>>>
>>>>>> I've only just managed to reproduce it, so haven't dug in to why
>yet. A quick hacky debug print shows something is stuck in an infinite
>loop.
>>>>>>
>>>>>> -ash
>>>>>>
>>>>>>> On 29 Oct 2018, at 17:59, Bolke de Bruin <bd...@gmail.com>
>wrote:
>>>>>>>
>>>>>>> Can this be confirmed? Then I can have a look at it. Preferably
>with dag definition code.
>>>>>>>
>>>>>>> On the licensing requirements:
>>>>>>>
>>>>>>> 1. Indeed licensing header for markdown documents. It was
>suggested to use html comments. I’m not sure how that renders with
>others like PDF though.
>>>>>>> 2. The licensing notifications need to be tied to a specific
>version as licenses might change with versions.
>>>>>>>
>>>>>>> Cheers
>>>>>>> Bolke
>>>>>>>
>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>
>>>>>>>> Op 29 okt. 2018 om 12:39 heeft Ash Berlin-Taylor
><as...@apache.org> het volgende geschreven:
>>>>>>>>
>>>>>>>> I was going to make a start on the release, but two people have
>reported that there might be an issue around non-UTC dags and the
>scheduler changing over from Summer time.
>>>>>>>>
>>>>>>>>> 08:45 Emmanuel> Hi there, we are currently experiencing a very
>strange issue : we have hourly DAGs with a start_date in a local
>timezone (not UTC) and since (Sunday) the last winter time change they
>don’t run anymore. Any idea ?
>>>>>>>>> 09:41 <Emmanuel> it impacted all our DAG that had a run at 3am
>(Europe/Paris), the exact time of winter time change :(
>>>>>>>>
>>>>>>>> I am going to take a look at this today and see if I can get to
>the bottom of it.
>>>>>>>>
>>>>>>>> Bolke: are there any outstanding tasks/issues that you know of
>that might slow down the vote for a 1.10.1? (i.e. did we sort of out
>all the licensing issues that were asked of us? I thought I read
>something about license declarations in markdown files?)
>>>>>>>>
>>>>>>>> -ash
>>>>>>>>
>>>>>>>>> On 28 Oct 2018, at 14:46, Bolke de Bruin <bd...@gmail.com>
>wrote:
>>>>>>>>>
>>>>>>>>> I agree with that, but I would favor time based releases
>instead. We are again at the point that a release takes so much time
>that the gap is getting really big again. @ash why not start releasing
>now and move the remainder to 1.10.2? I dont think there are real
>blockers (although we might find them).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On 28 Oct 2018, at 15:35, airflowuser
><ai...@protonmail.com.INVALID> wrote:
>>>>>>>>>>
>>>>>>>>>> I was really hoping that
>https://github.com/apache/incubator-airflow/pull/4069 will be merged
>into 1.10.1
>>>>>>>>>> Deleting dags was a highly requested feature for 1.10 - this
>can fix the problem with it.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>>>>>>>> On Friday, October 26, 2018 6:12 PM, Bolke de Bruin
><bd...@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hey Ash,
>>>>>>>>>>>
>>>>>>>>>>> I was wondering if you are picking up the 1.10.1 release?
>Master is speeding ahead and you were tracking fixes for 1.10.1 right?
>>>>>>>>>>>
>>>>>>>>>>> B.
>>>>>>>>
>>>>>>
>>>>>
>>>
>>
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: 1.10.1 Release?
Posted by Bolke de Bruin <bd...@gmail.com>.
I think we should use the UTC date for cron instead of the naive local date time. I will check of croniter implements this so we can rely on that.
B.
> On 28 Oct 2018, at 02:09, Bolke de Bruin <bd...@gmail.com> wrote:
>
> I wonder how to treat this:
>
> This is what I think happens (need to verify more, but I am pretty sure) the specified DAG should run every 5 minutes. At DST change (3AM -> 2AM) we basically hit a schedule that we have already seen. 2AM -> 3AM has already happened. Obviously the intention is to run every 5 minutes. But what do we do with the execution_date? Is this still idempotent? Should we indeed reschedule?
>
> B.
>
>> On 30 Oct 2018, at 19:01, Ash Berlin-Taylor <as...@apache.org> wrote:
>>
>> I've done a bit more digging - the issue is of our tz-aware handling inside following_schedule (and previous schedule) - causing it to loop.
>>
>> This section of the croniter docs seems relevant https://github.com/kiorky/croniter#about-dst
>>
>> Be sure to init your croniter instance with a TZ aware datetime for this to work !:
>>>>> local_date = tz.localize(datetime(2017, 3, 26))
>>>>> val = croniter('0 0 * * *', local_date).get_next(datetime)
>>
>> I think the problem is that we are _not_ passing a TZ aware dag in and we should be.
>>
>>> On 30 Oct 2018, at 17:35, Bolke de Bruin <bd...@gmail.com> wrote:
>>>
>>> Oh that’s a great environment to start digging. Thanks. I’ll have a look.
>>>
>>> B.
>>>
>>> Verstuurd vanaf mijn iPad
>>>
>>>> Op 30 okt. 2018 om 18:25 heeft Ash Berlin-Taylor <as...@apache.org> het volgende geschreven:
>>>>
>>>> This line in airflow.jobs (line 874 in my checkout) is causing the loop:
>>>>
>>>> last_run = dag.get_last_dagrun(session=session)
>>>> if last_run and next_run_date:
>>>> while next_run_date <= last_run.execution_date:
>>>> next_run_date = dag.following_schedule(next_run_date)
>>>>
>>>>
>>>>
>>>>> On 30 Oct 2018, at 17:20, Ash Berlin-Taylor <as...@apache.org> wrote:
>>>>>
>>>>> Hi, kaczors on gitter has produced a minmal reproduction case: https://github.com/kaczors/airflow_1_10_tz_bug
>>>>>
>>>>> Rough repro steps: In a VM, with time syncing disabled, and configured with system timezone of Europe/Zurich (or any other CEST one) run
>>>>>
>>>>> - `date 10280250.00`
>>>>> - initdb, start scheduler, webserver, enable dag etc.
>>>>> - `date 10280259.00`
>>>>> - wait 5-10 mins for scheduler to catch up
>>>>> - After the on-the-hour task run the scheduler will spin up another process to parse the dag... and it never returns.
>>>>>
>>>>> I've only just managed to reproduce it, so haven't dug in to why yet. A quick hacky debug print shows something is stuck in an infinite loop.
>>>>>
>>>>> -ash
>>>>>
>>>>>> On 29 Oct 2018, at 17:59, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>>>
>>>>>> Can this be confirmed? Then I can have a look at it. Preferably with dag definition code.
>>>>>>
>>>>>> On the licensing requirements:
>>>>>>
>>>>>> 1. Indeed licensing header for markdown documents. It was suggested to use html comments. I’m not sure how that renders with others like PDF though.
>>>>>> 2. The licensing notifications need to be tied to a specific version as licenses might change with versions.
>>>>>>
>>>>>> Cheers
>>>>>> Bolke
>>>>>>
>>>>>> Verstuurd vanaf mijn iPad
>>>>>>
>>>>>>> Op 29 okt. 2018 om 12:39 heeft Ash Berlin-Taylor <as...@apache.org> het volgende geschreven:
>>>>>>>
>>>>>>> I was going to make a start on the release, but two people have reported that there might be an issue around non-UTC dags and the scheduler changing over from Summer time.
>>>>>>>
>>>>>>>> 08:45 Emmanuel> Hi there, we are currently experiencing a very strange issue : we have hourly DAGs with a start_date in a local timezone (not UTC) and since (Sunday) the last winter time change they don’t run anymore. Any idea ?
>>>>>>>> 09:41 <Emmanuel> it impacted all our DAG that had a run at 3am (Europe/Paris), the exact time of winter time change :(
>>>>>>>
>>>>>>> I am going to take a look at this today and see if I can get to the bottom of it.
>>>>>>>
>>>>>>> Bolke: are there any outstanding tasks/issues that you know of that might slow down the vote for a 1.10.1? (i.e. did we sort of out all the licensing issues that were asked of us? I thought I read something about license declarations in markdown files?)
>>>>>>>
>>>>>>> -ash
>>>>>>>
>>>>>>>> On 28 Oct 2018, at 14:46, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> I agree with that, but I would favor time based releases instead. We are again at the point that a release takes so much time that the gap is getting really big again. @ash why not start releasing now and move the remainder to 1.10.2? I dont think there are real blockers (although we might find them).
>>>>>>>>
>>>>>>>>
>>>>>>>>> On 28 Oct 2018, at 15:35, airflowuser <ai...@protonmail.com.INVALID> wrote:
>>>>>>>>>
>>>>>>>>> I was really hoping that https://github.com/apache/incubator-airflow/pull/4069 will be merged into 1.10.1
>>>>>>>>> Deleting dags was a highly requested feature for 1.10 - this can fix the problem with it.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>>>>>>> On Friday, October 26, 2018 6:12 PM, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Hey Ash,
>>>>>>>>>>
>>>>>>>>>> I was wondering if you are picking up the 1.10.1 release? Master is speeding ahead and you were tracking fixes for 1.10.1 right?
>>>>>>>>>>
>>>>>>>>>> B.
>>>>>>>
>>>>>
>>>>
>>
>
Re: 1.10.1 Release?
Posted by Bolke de Bruin <bd...@gmail.com>.
I wonder how to treat this:
This is what I think happens (need to verify more, but I am pretty sure) the specified DAG should run every 5 minutes. At DST change (3AM -> 2AM) we basically hit a schedule that we have already seen. 2AM -> 3AM has already happened. Obviously the intention is to run every 5 minutes. But what do we do with the execution_date? Is this still idempotent? Should we indeed reschedule?
B.
> On 30 Oct 2018, at 19:01, Ash Berlin-Taylor <as...@apache.org> wrote:
>
> I've done a bit more digging - the issue is of our tz-aware handling inside following_schedule (and previous schedule) - causing it to loop.
>
> This section of the croniter docs seems relevant https://github.com/kiorky/croniter#about-dst
>
> Be sure to init your croniter instance with a TZ aware datetime for this to work !:
>>>> local_date = tz.localize(datetime(2017, 3, 26))
>>>> val = croniter('0 0 * * *', local_date).get_next(datetime)
>
> I think the problem is that we are _not_ passing a TZ aware dag in and we should be.
>
>> On 30 Oct 2018, at 17:35, Bolke de Bruin <bd...@gmail.com> wrote:
>>
>> Oh that’s a great environment to start digging. Thanks. I’ll have a look.
>>
>> B.
>>
>> Verstuurd vanaf mijn iPad
>>
>>> Op 30 okt. 2018 om 18:25 heeft Ash Berlin-Taylor <as...@apache.org> het volgende geschreven:
>>>
>>> This line in airflow.jobs (line 874 in my checkout) is causing the loop:
>>>
>>> last_run = dag.get_last_dagrun(session=session)
>>> if last_run and next_run_date:
>>> while next_run_date <= last_run.execution_date:
>>> next_run_date = dag.following_schedule(next_run_date)
>>>
>>>
>>>
>>>> On 30 Oct 2018, at 17:20, Ash Berlin-Taylor <as...@apache.org> wrote:
>>>>
>>>> Hi, kaczors on gitter has produced a minmal reproduction case: https://github.com/kaczors/airflow_1_10_tz_bug
>>>>
>>>> Rough repro steps: In a VM, with time syncing disabled, and configured with system timezone of Europe/Zurich (or any other CEST one) run
>>>>
>>>> - `date 10280250.00`
>>>> - initdb, start scheduler, webserver, enable dag etc.
>>>> - `date 10280259.00`
>>>> - wait 5-10 mins for scheduler to catch up
>>>> - After the on-the-hour task run the scheduler will spin up another process to parse the dag... and it never returns.
>>>>
>>>> I've only just managed to reproduce it, so haven't dug in to why yet. A quick hacky debug print shows something is stuck in an infinite loop.
>>>>
>>>> -ash
>>>>
>>>>> On 29 Oct 2018, at 17:59, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>>
>>>>> Can this be confirmed? Then I can have a look at it. Preferably with dag definition code.
>>>>>
>>>>> On the licensing requirements:
>>>>>
>>>>> 1. Indeed licensing header for markdown documents. It was suggested to use html comments. I’m not sure how that renders with others like PDF though.
>>>>> 2. The licensing notifications need to be tied to a specific version as licenses might change with versions.
>>>>>
>>>>> Cheers
>>>>> Bolke
>>>>>
>>>>> Verstuurd vanaf mijn iPad
>>>>>
>>>>>> Op 29 okt. 2018 om 12:39 heeft Ash Berlin-Taylor <as...@apache.org> het volgende geschreven:
>>>>>>
>>>>>> I was going to make a start on the release, but two people have reported that there might be an issue around non-UTC dags and the scheduler changing over from Summer time.
>>>>>>
>>>>>>> 08:45 Emmanuel> Hi there, we are currently experiencing a very strange issue : we have hourly DAGs with a start_date in a local timezone (not UTC) and since (Sunday) the last winter time change they don’t run anymore. Any idea ?
>>>>>>> 09:41 <Emmanuel> it impacted all our DAG that had a run at 3am (Europe/Paris), the exact time of winter time change :(
>>>>>>
>>>>>> I am going to take a look at this today and see if I can get to the bottom of it.
>>>>>>
>>>>>> Bolke: are there any outstanding tasks/issues that you know of that might slow down the vote for a 1.10.1? (i.e. did we sort of out all the licensing issues that were asked of us? I thought I read something about license declarations in markdown files?)
>>>>>>
>>>>>> -ash
>>>>>>
>>>>>>> On 28 Oct 2018, at 14:46, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>>>>
>>>>>>> I agree with that, but I would favor time based releases instead. We are again at the point that a release takes so much time that the gap is getting really big again. @ash why not start releasing now and move the remainder to 1.10.2? I dont think there are real blockers (although we might find them).
>>>>>>>
>>>>>>>
>>>>>>>> On 28 Oct 2018, at 15:35, airflowuser <ai...@protonmail.com.INVALID> wrote:
>>>>>>>>
>>>>>>>> I was really hoping that https://github.com/apache/incubator-airflow/pull/4069 will be merged into 1.10.1
>>>>>>>> Deleting dags was a highly requested feature for 1.10 - this can fix the problem with it.
>>>>>>>>
>>>>>>>>
>>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>>>>>> On Friday, October 26, 2018 6:12 PM, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Hey Ash,
>>>>>>>>>
>>>>>>>>> I was wondering if you are picking up the 1.10.1 release? Master is speeding ahead and you were tracking fixes for 1.10.1 right?
>>>>>>>>>
>>>>>>>>> B.
>>>>>>
>>>>
>>>
>
Re: 1.10.1 Release?
Posted by Ash Berlin-Taylor <as...@apache.org>.
I've done a bit more digging - the issue is of our tz-aware handling inside following_schedule (and previous schedule) - causing it to loop.
This section of the croniter docs seems relevant https://github.com/kiorky/croniter#about-dst
Be sure to init your croniter instance with a TZ aware datetime for this to work !:
>>> local_date = tz.localize(datetime(2017, 3, 26))
>>> val = croniter('0 0 * * *', local_date).get_next(datetime)
I think the problem is that we are _not_ passing a TZ aware dag in and we should be.
> On 30 Oct 2018, at 17:35, Bolke de Bruin <bd...@gmail.com> wrote:
>
> Oh that’s a great environment to start digging. Thanks. I’ll have a look.
>
> B.
>
> Verstuurd vanaf mijn iPad
>
>> Op 30 okt. 2018 om 18:25 heeft Ash Berlin-Taylor <as...@apache.org> het volgende geschreven:
>>
>> This line in airflow.jobs (line 874 in my checkout) is causing the loop:
>>
>> last_run = dag.get_last_dagrun(session=session)
>> if last_run and next_run_date:
>> while next_run_date <= last_run.execution_date:
>> next_run_date = dag.following_schedule(next_run_date)
>>
>>
>>
>>> On 30 Oct 2018, at 17:20, Ash Berlin-Taylor <as...@apache.org> wrote:
>>>
>>> Hi, kaczors on gitter has produced a minmal reproduction case: https://github.com/kaczors/airflow_1_10_tz_bug
>>>
>>> Rough repro steps: In a VM, with time syncing disabled, and configured with system timezone of Europe/Zurich (or any other CEST one) run
>>>
>>> - `date 10280250.00`
>>> - initdb, start scheduler, webserver, enable dag etc.
>>> - `date 10280259.00`
>>> - wait 5-10 mins for scheduler to catch up
>>> - After the on-the-hour task run the scheduler will spin up another process to parse the dag... and it never returns.
>>>
>>> I've only just managed to reproduce it, so haven't dug in to why yet. A quick hacky debug print shows something is stuck in an infinite loop.
>>>
>>> -ash
>>>
>>>> On 29 Oct 2018, at 17:59, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>
>>>> Can this be confirmed? Then I can have a look at it. Preferably with dag definition code.
>>>>
>>>> On the licensing requirements:
>>>>
>>>> 1. Indeed licensing header for markdown documents. It was suggested to use html comments. I’m not sure how that renders with others like PDF though.
>>>> 2. The licensing notifications need to be tied to a specific version as licenses might change with versions.
>>>>
>>>> Cheers
>>>> Bolke
>>>>
>>>> Verstuurd vanaf mijn iPad
>>>>
>>>>> Op 29 okt. 2018 om 12:39 heeft Ash Berlin-Taylor <as...@apache.org> het volgende geschreven:
>>>>>
>>>>> I was going to make a start on the release, but two people have reported that there might be an issue around non-UTC dags and the scheduler changing over from Summer time.
>>>>>
>>>>>> 08:45 Emmanuel> Hi there, we are currently experiencing a very strange issue : we have hourly DAGs with a start_date in a local timezone (not UTC) and since (Sunday) the last winter time change they don’t run anymore. Any idea ?
>>>>>> 09:41 <Emmanuel> it impacted all our DAG that had a run at 3am (Europe/Paris), the exact time of winter time change :(
>>>>>
>>>>> I am going to take a look at this today and see if I can get to the bottom of it.
>>>>>
>>>>> Bolke: are there any outstanding tasks/issues that you know of that might slow down the vote for a 1.10.1? (i.e. did we sort of out all the licensing issues that were asked of us? I thought I read something about license declarations in markdown files?)
>>>>>
>>>>> -ash
>>>>>
>>>>>> On 28 Oct 2018, at 14:46, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>>>
>>>>>> I agree with that, but I would favor time based releases instead. We are again at the point that a release takes so much time that the gap is getting really big again. @ash why not start releasing now and move the remainder to 1.10.2? I dont think there are real blockers (although we might find them).
>>>>>>
>>>>>>
>>>>>>> On 28 Oct 2018, at 15:35, airflowuser <ai...@protonmail.com.INVALID> wrote:
>>>>>>>
>>>>>>> I was really hoping that https://github.com/apache/incubator-airflow/pull/4069 will be merged into 1.10.1
>>>>>>> Deleting dags was a highly requested feature for 1.10 - this can fix the problem with it.
>>>>>>>
>>>>>>>
>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>>>>> On Friday, October 26, 2018 6:12 PM, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Hey Ash,
>>>>>>>>
>>>>>>>> I was wondering if you are picking up the 1.10.1 release? Master is speeding ahead and you were tracking fixes for 1.10.1 right?
>>>>>>>>
>>>>>>>> B.
>>>>>
>>>
>>
Re: 1.10.1 Release?
Posted by Bolke de Bruin <bd...@gmail.com>.
Oh that’s a great environment to start digging. Thanks. I’ll have a look.
B.
Verstuurd vanaf mijn iPad
> Op 30 okt. 2018 om 18:25 heeft Ash Berlin-Taylor <as...@apache.org> het volgende geschreven:
>
> This line in airflow.jobs (line 874 in my checkout) is causing the loop:
>
> last_run = dag.get_last_dagrun(session=session)
> if last_run and next_run_date:
> while next_run_date <= last_run.execution_date:
> next_run_date = dag.following_schedule(next_run_date)
>
>
>
>> On 30 Oct 2018, at 17:20, Ash Berlin-Taylor <as...@apache.org> wrote:
>>
>> Hi, kaczors on gitter has produced a minmal reproduction case: https://github.com/kaczors/airflow_1_10_tz_bug
>>
>> Rough repro steps: In a VM, with time syncing disabled, and configured with system timezone of Europe/Zurich (or any other CEST one) run
>>
>> - `date 10280250.00`
>> - initdb, start scheduler, webserver, enable dag etc.
>> - `date 10280259.00`
>> - wait 5-10 mins for scheduler to catch up
>> - After the on-the-hour task run the scheduler will spin up another process to parse the dag... and it never returns.
>>
>> I've only just managed to reproduce it, so haven't dug in to why yet. A quick hacky debug print shows something is stuck in an infinite loop.
>>
>> -ash
>>
>>> On 29 Oct 2018, at 17:59, Bolke de Bruin <bd...@gmail.com> wrote:
>>>
>>> Can this be confirmed? Then I can have a look at it. Preferably with dag definition code.
>>>
>>> On the licensing requirements:
>>>
>>> 1. Indeed licensing header for markdown documents. It was suggested to use html comments. I’m not sure how that renders with others like PDF though.
>>> 2. The licensing notifications need to be tied to a specific version as licenses might change with versions.
>>>
>>> Cheers
>>> Bolke
>>>
>>> Verstuurd vanaf mijn iPad
>>>
>>>> Op 29 okt. 2018 om 12:39 heeft Ash Berlin-Taylor <as...@apache.org> het volgende geschreven:
>>>>
>>>> I was going to make a start on the release, but two people have reported that there might be an issue around non-UTC dags and the scheduler changing over from Summer time.
>>>>
>>>>> 08:45 Emmanuel> Hi there, we are currently experiencing a very strange issue : we have hourly DAGs with a start_date in a local timezone (not UTC) and since (Sunday) the last winter time change they don’t run anymore. Any idea ?
>>>>> 09:41 <Emmanuel> it impacted all our DAG that had a run at 3am (Europe/Paris), the exact time of winter time change :(
>>>>
>>>> I am going to take a look at this today and see if I can get to the bottom of it.
>>>>
>>>> Bolke: are there any outstanding tasks/issues that you know of that might slow down the vote for a 1.10.1? (i.e. did we sort of out all the licensing issues that were asked of us? I thought I read something about license declarations in markdown files?)
>>>>
>>>> -ash
>>>>
>>>>> On 28 Oct 2018, at 14:46, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>>
>>>>> I agree with that, but I would favor time based releases instead. We are again at the point that a release takes so much time that the gap is getting really big again. @ash why not start releasing now and move the remainder to 1.10.2? I dont think there are real blockers (although we might find them).
>>>>>
>>>>>
>>>>>> On 28 Oct 2018, at 15:35, airflowuser <ai...@protonmail.com.INVALID> wrote:
>>>>>>
>>>>>> I was really hoping that https://github.com/apache/incubator-airflow/pull/4069 will be merged into 1.10.1
>>>>>> Deleting dags was a highly requested feature for 1.10 - this can fix the problem with it.
>>>>>>
>>>>>>
>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>>>> On Friday, October 26, 2018 6:12 PM, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>>>>
>>>>>>> Hey Ash,
>>>>>>>
>>>>>>> I was wondering if you are picking up the 1.10.1 release? Master is speeding ahead and you were tracking fixes for 1.10.1 right?
>>>>>>>
>>>>>>> B.
>>>>
>>
>
Re: 1.10.1 Release?
Posted by Ash Berlin-Taylor <as...@apache.org>.
This line in airflow.jobs (line 874 in my checkout) is causing the loop:
last_run = dag.get_last_dagrun(session=session)
if last_run and next_run_date:
while next_run_date <= last_run.execution_date:
next_run_date = dag.following_schedule(next_run_date)
> On 30 Oct 2018, at 17:20, Ash Berlin-Taylor <as...@apache.org> wrote:
>
> Hi, kaczors on gitter has produced a minmal reproduction case: https://github.com/kaczors/airflow_1_10_tz_bug
>
> Rough repro steps: In a VM, with time syncing disabled, and configured with system timezone of Europe/Zurich (or any other CEST one) run
>
> - `date 10280250.00`
> - initdb, start scheduler, webserver, enable dag etc.
> - `date 10280259.00`
> - wait 5-10 mins for scheduler to catch up
> - After the on-the-hour task run the scheduler will spin up another process to parse the dag... and it never returns.
>
> I've only just managed to reproduce it, so haven't dug in to why yet. A quick hacky debug print shows something is stuck in an infinite loop.
>
> -ash
>
>> On 29 Oct 2018, at 17:59, Bolke de Bruin <bd...@gmail.com> wrote:
>>
>> Can this be confirmed? Then I can have a look at it. Preferably with dag definition code.
>>
>> On the licensing requirements:
>>
>> 1. Indeed licensing header for markdown documents. It was suggested to use html comments. I’m not sure how that renders with others like PDF though.
>> 2. The licensing notifications need to be tied to a specific version as licenses might change with versions.
>>
>> Cheers
>> Bolke
>>
>> Verstuurd vanaf mijn iPad
>>
>>> Op 29 okt. 2018 om 12:39 heeft Ash Berlin-Taylor <as...@apache.org> het volgende geschreven:
>>>
>>> I was going to make a start on the release, but two people have reported that there might be an issue around non-UTC dags and the scheduler changing over from Summer time.
>>>
>>>> 08:45 Emmanuel> Hi there, we are currently experiencing a very strange issue : we have hourly DAGs with a start_date in a local timezone (not UTC) and since (Sunday) the last winter time change they don’t run anymore. Any idea ?
>>>> 09:41 <Emmanuel> it impacted all our DAG that had a run at 3am (Europe/Paris), the exact time of winter time change :(
>>>
>>> I am going to take a look at this today and see if I can get to the bottom of it.
>>>
>>> Bolke: are there any outstanding tasks/issues that you know of that might slow down the vote for a 1.10.1? (i.e. did we sort of out all the licensing issues that were asked of us? I thought I read something about license declarations in markdown files?)
>>>
>>> -ash
>>>
>>>> On 28 Oct 2018, at 14:46, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>
>>>> I agree with that, but I would favor time based releases instead. We are again at the point that a release takes so much time that the gap is getting really big again. @ash why not start releasing now and move the remainder to 1.10.2? I dont think there are real blockers (although we might find them).
>>>>
>>>>
>>>>> On 28 Oct 2018, at 15:35, airflowuser <ai...@protonmail.com.INVALID> wrote:
>>>>>
>>>>> I was really hoping that https://github.com/apache/incubator-airflow/pull/4069 will be merged into 1.10.1
>>>>> Deleting dags was a highly requested feature for 1.10 - this can fix the problem with it.
>>>>>
>>>>>
>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>>> On Friday, October 26, 2018 6:12 PM, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>>>
>>>>>> Hey Ash,
>>>>>>
>>>>>> I was wondering if you are picking up the 1.10.1 release? Master is speeding ahead and you were tracking fixes for 1.10.1 right?
>>>>>>
>>>>>> B.
>>>
>
Re: 1.10.1 Release?
Posted by Ash Berlin-Taylor <as...@apache.org>.
Hi, kaczors on gitter has produced a minmal reproduction case: https://github.com/kaczors/airflow_1_10_tz_bug
Rough repro steps: In a VM, with time syncing disabled, and configured with system timezone of Europe/Zurich (or any other CEST one) run
- `date 10280250.00`
- initdb, start scheduler, webserver, enable dag etc.
- `date 10280259.00`
- wait 5-10 mins for scheduler to catch up
- After the on-the-hour task run the scheduler will spin up another process to parse the dag... and it never returns.
I've only just managed to reproduce it, so haven't dug in to why yet. A quick hacky debug print shows something is stuck in an infinite loop.
-ash
> On 29 Oct 2018, at 17:59, Bolke de Bruin <bd...@gmail.com> wrote:
>
> Can this be confirmed? Then I can have a look at it. Preferably with dag definition code.
>
> On the licensing requirements:
>
> 1. Indeed licensing header for markdown documents. It was suggested to use html comments. I’m not sure how that renders with others like PDF though.
> 2. The licensing notifications need to be tied to a specific version as licenses might change with versions.
>
> Cheers
> Bolke
>
> Verstuurd vanaf mijn iPad
>
>> Op 29 okt. 2018 om 12:39 heeft Ash Berlin-Taylor <as...@apache.org> het volgende geschreven:
>>
>> I was going to make a start on the release, but two people have reported that there might be an issue around non-UTC dags and the scheduler changing over from Summer time.
>>
>>> 08:45 Emmanuel> Hi there, we are currently experiencing a very strange issue : we have hourly DAGs with a start_date in a local timezone (not UTC) and since (Sunday) the last winter time change they don’t run anymore. Any idea ?
>>> 09:41 <Emmanuel> it impacted all our DAG that had a run at 3am (Europe/Paris), the exact time of winter time change :(
>>
>> I am going to take a look at this today and see if I can get to the bottom of it.
>>
>> Bolke: are there any outstanding tasks/issues that you know of that might slow down the vote for a 1.10.1? (i.e. did we sort of out all the licensing issues that were asked of us? I thought I read something about license declarations in markdown files?)
>>
>> -ash
>>
>>> On 28 Oct 2018, at 14:46, Bolke de Bruin <bd...@gmail.com> wrote:
>>>
>>> I agree with that, but I would favor time based releases instead. We are again at the point that a release takes so much time that the gap is getting really big again. @ash why not start releasing now and move the remainder to 1.10.2? I dont think there are real blockers (although we might find them).
>>>
>>>
>>>> On 28 Oct 2018, at 15:35, airflowuser <ai...@protonmail.com.INVALID> wrote:
>>>>
>>>> I was really hoping that https://github.com/apache/incubator-airflow/pull/4069 will be merged into 1.10.1
>>>> Deleting dags was a highly requested feature for 1.10 - this can fix the problem with it.
>>>>
>>>>
>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>> On Friday, October 26, 2018 6:12 PM, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>>
>>>>> Hey Ash,
>>>>>
>>>>> I was wondering if you are picking up the 1.10.1 release? Master is speeding ahead and you were tracking fixes for 1.10.1 right?
>>>>>
>>>>> B.
>>
Re: 1.10.1 Release?
Posted by Bolke de Bruin <bd...@gmail.com>.
Can this be confirmed? Then I can have a look at it. Preferably with dag definition code.
On the licensing requirements:
1. Indeed licensing header for markdown documents. It was suggested to use html comments. I’m not sure how that renders with others like PDF though.
2. The licensing notifications need to be tied to a specific version as licenses might change with versions.
Cheers
Bolke
Verstuurd vanaf mijn iPad
> Op 29 okt. 2018 om 12:39 heeft Ash Berlin-Taylor <as...@apache.org> het volgende geschreven:
>
> I was going to make a start on the release, but two people have reported that there might be an issue around non-UTC dags and the scheduler changing over from Summer time.
>
>> 08:45 Emmanuel> Hi there, we are currently experiencing a very strange issue : we have hourly DAGs with a start_date in a local timezone (not UTC) and since (Sunday) the last winter time change they don’t run anymore. Any idea ?
>> 09:41 <Emmanuel> it impacted all our DAG that had a run at 3am (Europe/Paris), the exact time of winter time change :(
>
> I am going to take a look at this today and see if I can get to the bottom of it.
>
> Bolke: are there any outstanding tasks/issues that you know of that might slow down the vote for a 1.10.1? (i.e. did we sort of out all the licensing issues that were asked of us? I thought I read something about license declarations in markdown files?)
>
> -ash
>
>> On 28 Oct 2018, at 14:46, Bolke de Bruin <bd...@gmail.com> wrote:
>>
>> I agree with that, but I would favor time based releases instead. We are again at the point that a release takes so much time that the gap is getting really big again. @ash why not start releasing now and move the remainder to 1.10.2? I dont think there are real blockers (although we might find them).
>>
>>
>>> On 28 Oct 2018, at 15:35, airflowuser <ai...@protonmail.com.INVALID> wrote:
>>>
>>> I was really hoping that https://github.com/apache/incubator-airflow/pull/4069 will be merged into 1.10.1
>>> Deleting dags was a highly requested feature for 1.10 - this can fix the problem with it.
>>>
>>>
>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>> On Friday, October 26, 2018 6:12 PM, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>
>>>> Hey Ash,
>>>>
>>>> I was wondering if you are picking up the 1.10.1 release? Master is speeding ahead and you were tracking fixes for 1.10.1 right?
>>>>
>>>> B.
>
Re: 1.10.1 Release?
Posted by Ash Berlin-Taylor <as...@apache.org>.
I was going to make a start on the release, but two people have reported that there might be an issue around non-UTC dags and the scheduler changing over from Summer time.
> 08:45 Emmanuel> Hi there, we are currently experiencing a very strange issue : we have hourly DAGs with a start_date in a local timezone (not UTC) and since (Sunday) the last winter time change they don’t run anymore. Any idea ?
> 09:41 <Emmanuel> it impacted all our DAG that had a run at 3am (Europe/Paris), the exact time of winter time change :(
I am going to take a look at this today and see if I can get to the bottom of it.
Bolke: are there any outstanding tasks/issues that you know of that might slow down the vote for a 1.10.1? (i.e. did we sort of out all the licensing issues that were asked of us? I thought I read something about license declarations in markdown files?)
-ash
> On 28 Oct 2018, at 14:46, Bolke de Bruin <bd...@gmail.com> wrote:
>
> I agree with that, but I would favor time based releases instead. We are again at the point that a release takes so much time that the gap is getting really big again. @ash why not start releasing now and move the remainder to 1.10.2? I dont think there are real blockers (although we might find them).
>
>
>> On 28 Oct 2018, at 15:35, airflowuser <ai...@protonmail.com.INVALID> wrote:
>>
>> I was really hoping that https://github.com/apache/incubator-airflow/pull/4069 will be merged into 1.10.1
>> Deleting dags was a highly requested feature for 1.10 - this can fix the problem with it.
>>
>>
>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>> On Friday, October 26, 2018 6:12 PM, Bolke de Bruin <bd...@gmail.com> wrote:
>>
>>> Hey Ash,
>>>
>>> I was wondering if you are picking up the 1.10.1 release? Master is speeding ahead and you were tracking fixes for 1.10.1 right?
>>>
>>> B.
>>
>>
>
Re: 1.10.1 Release?
Posted by Bolke de Bruin <bd...@gmail.com>.
I agree with that, but I would favor time based releases instead. We are again at the point that a release takes so much time that the gap is getting really big again. @ash why not start releasing now and move the remainder to 1.10.2? I dont think there are real blockers (although we might find them).
> On 28 Oct 2018, at 15:35, airflowuser <ai...@protonmail.com.INVALID> wrote:
>
> I was really hoping that https://github.com/apache/incubator-airflow/pull/4069 will be merged into 1.10.1
> Deleting dags was a highly requested feature for 1.10 - this can fix the problem with it.
>
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Friday, October 26, 2018 6:12 PM, Bolke de Bruin <bd...@gmail.com> wrote:
>
>> Hey Ash,
>>
>> I was wondering if you are picking up the 1.10.1 release? Master is speeding ahead and you were tracking fixes for 1.10.1 right?
>>
>> B.
>
>
Re: 1.10.1 Release?
Posted by airflowuser <ai...@protonmail.com.INVALID>.
I was really hoping that https://github.com/apache/incubator-airflow/pull/4069 will be merged into 1.10.1
Deleting dags was a highly requested feature for 1.10 - this can fix the problem with it.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, October 26, 2018 6:12 PM, Bolke de Bruin <bd...@gmail.com> wrote:
> Hey Ash,
>
> I was wondering if you are picking up the 1.10.1 release? Master is speeding ahead and you were tracking fixes for 1.10.1 right?
>
> B.