You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Andrey Anshin <an...@taragol.is> on 2023/12/01 12:08:53 UTC
Re: [DISCUSS] Future of Pendulum in Airflow
IMHO, if we would like to replace pendulum to alternatives, the better
alternative would be combination of datetime.datetime + zoneinfo +
datetime.timezone
We still do hacks around pendulum functional, which makes hacks around
datetime.datetime, so in my perspective better remove redundant layers and
use native objects directly.
What confuses me the most is how we get around type changes in Tasks
Context. All dates return as pendulum.DateTime for a very long time
(literally for ages) and it is probably could be classified as breaking
changes, because it might break quite a bit users pipelines. It still could
be resolved as optional pendulum dependency + config option for datetime
types in context, which might help to get familiar types for a while.
On Wed, 29 Nov 2023 at 05:58, Jarek Potiuk <ja...@potiuk.com> wrote:
> "infamous fork of " should be "fork of infamous Akka" - just to be clear :)
>
> On Wed, Nov 29, 2023 at 2:57 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > It's a very rare occurrence that ASF accepts forks. Usually it has to be
> > willingly donated by those who own the IP rights, - mainly because of
> > trademark issues. Licence is one thing, but carrying things like the name
> > of the project is not possible without clearing IP and donating the
> > project.
> >
> > It happened last time last year - with Pekko project - infamous fork or
> > Akka after Lightbend changed licensing for Akka -
> > https://www.lightbend.com/blog/why-we-are-changing-the-license-for-akka
> > making it impossible to be used in ASF projects with the new licence. But
> > it was a really "strong" case and the "we do not accept forks usually"
> was
> > quite a contentious issue (I followed the discussion - it was fascinating
> > actually). It tool a LOT of time and effort for those who led the effort
> > eventually - and also mostly because things had to be crystal-clear and
> not
> > "litigable" because the other side had some business reasons to make the
> > licensing changes and they did not understood why it is impossible to use
> > Akka as-is even with all the exclusions they offered to ASF - and
> > effectively it was a "hostile" move.
> >
> > I don't think it's "worth" it in this case to be honest - especially if
> > the maintainers are effectively "silent" and seem to ignore problems of
> one
> > of the really serious users they have. If they were willing to cooperate,
> > we could do a lot, but when we are left alone, I think it will be far
> more
> > convenient for us to look for alternatives - to be honest.
> >
> > J.
> >
> >
> > On Wed, Nov 29, 2023 at 2:46 AM Austin Bennett <au...@apache.org>
> wrote:
> >
> >> An option would be to fork Pendulum? It is MIT Licensed, I don't know
> >> whether that poses problems to get in ASF?
> >>
> >> If forking (?) which is somewhat non-ideal, would we want that 'in'
> >> airflow?
> >>
> >> If not 'in' airflow, I wonder if ASF incubator would accept a forked
> >> project? [ anecdotally Linux Foundation has incubated Starrocks which
> is
> >> a
> >> fork of Apache Doris, so there's precedence for this to be OK in the
> wider
> >> OSS ecosystem ].
> >>
> >>
> >> On Tue, Nov 28, 2023 at 6:41 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >>
> >> > I have not heard back from maintainers, just a comment from someone
> else
> >> > who suggested donating a pendulum to the ASF (which is kinda
> interesting
> >> > idea). I followed up, let's see. I think if we do not hear back and
> >> there
> >> > will be another week or two where "Pendulum 3 supporting 3.12 support
> >> "is
> >> > coming" but no-one knows when, effectively means that we should look
> >> for an
> >> > alternative like NOW.
> >> >
> >> > J.
> >> >
> >> >
> >> >
> >> > On Wed, Nov 22, 2023 at 3:27 AM Daniel Standish
> >> > <da...@astronomer.io.invalid> wrote:
> >> >
> >> > > Thanks Jarek
> >> > >
> >> > > On Tue, Nov 21, 2023, 4:34 PM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> >> > >
> >> > > > I think we miss important insight - straight from the source.
> >> > > >
> >> > > > I believe it's time to be candid and simply ask questions for the
> >> > future
> >> > > of
> >> > > > Pendulum directly where we should - ie. we should just ask
> >> > maintainers.
> >> > > >
> >> > > > I've just started a very candid and open discussion there -
> >> > > > https://github.com/sdispater/pendulum/discussions/771 .
> >> > > >
> >> > > > I linked back to this discussion and explained where we are coming
> >> > from.
> >> > > I
> >> > > > even offered an option, that maybe - if they accept some help and
> >> would
> >> > > be
> >> > > > open to make a few of us maintainers of Pendulum, that I will ask
> if
> >> > some
> >> > > > of the maintainers here in Airflow would like to step up (I would
> be
> >> > > > willing to - for one) and help to at least move Pendulum through
> >> 3.12
> >> > and
> >> > > > maybe keep it running for as long as it will be needed to maintain
> >> it.
> >> > > >
> >> > > > I think if they would be open to some of us helping them - this
> >> might
> >> > be
> >> > > > actually the "cheapest" option for us to be honest at least in a
> >> > mid-term
> >> > > > if we could gain influence and even take part and speed up
> releases
> >> and
> >> > > > merging of issues that are blocking us (or would be blocking us in
> >> the
> >> > > > future).
> >> > > >
> >> > > > I tried to be very friendly but candid and direct on what kind of
> >> > > problems
> >> > > > it creates us and also expressed our thanks for supporting us for
> so
> >> > long
> >> > > > (I do believe they deserve it). I think the sole (pretty much)
> >> > > maintainer
> >> > > > of Pendulum might really have a hard time maintaining and keeping
> >> it up
> >> > > for
> >> > > > years - without being paid for and thanked. I hope - maybe all
> that
> >> > they
> >> > > > need are a few words of encouragement and thanks and realising
> that
> >> > > others
> >> > > > depend on their work - but also seeing that they are understood,
> >> > > empathised
> >> > > > with and that there are viable alternatives they might follow
> might
> >> be
> >> > > > helpful for them to be also candid and respond. I hope my
> intentions
> >> > will
> >> > > > be understood - that I am not complaining, but even trying to
> help.
> >> > > >
> >> > > > Let\s see where it gets us.
> >> > > >
> >> > > > J.
> >> > > >
> >> > > > On Fri, Nov 17, 2023 at 5:32 PM Andrey Anshin <
> >> > andrey.anshin@taragol.is>
> >> > > > wrote:
> >> > > >
> >> > > > > Just to clarify I'd like us to consider the possibility that no
> >> new
> >> > > > > pendulum would be released or released at the end of 2024, like
> a
> >> > > > > pessimistic scenario:
> >> > > > > - What should we do in this case?
> >> > > > > - Work out a backup plan.
> >> > > > >
> >> > > > >
> >> > > > > ----
> >> > > > > Best Wishes
> >> > > > > *Andrey Anshin*
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > On Fri, 17 Nov 2023 at 16:33, Jarek Potiuk <ja...@potiuk.com>
> >> wrote:
> >> > > > >
> >> > > > > > Also I think TP - had a document in the past (years ago)
> >> > > describing a
> >> > > > > > draft of a more complete alternative we can take to approach
> >> > datetime
> >> > > > vs.
> >> > > > > > pendulum dichotomy. I cannot easily find the document and
> >> > discussion
> >> > > -
> >> > > > > but
> >> > > > > > I do remember it was proposing some interesting changes in the
> >> > > approach
> >> > > > > of
> >> > > > > > Airflow to have an abstraction layer on top of it (as far as I
> >> > > > remember).
> >> > > > > > Maybe we can resurrect that idea if TP might find the
> proposal ?
> >> > > > > >
> >> > > > > > On Fri, Nov 17, 2023 at 1:06 PM Bolke de Bruin <
> >> bdbruin@gmail.com>
> >> > > > > wrote:
> >> > > > > >
> >> > > > > > > Hi,
> >> > > > > > >
> >> > > > > > > I agree that the current speed of development of Pendulum
> >> leaves
> >> > > > > > something
> >> > > > > > > to be desired. However, I think we should not underestimate
> >> the
> >> > > > effort
> >> > > > > of
> >> > > > > > > replacing it. It is not just a matter of
> >> %s/pendulum/datetime/g
> >> > so
> >> > > to
> >> > > > > > say.
> >> > > > > > > If we are *truly* thinking about moving to native datetime /
> >> > > zoneinfo
> >> > > > > etc
> >> > > > > > > we need *extensive* tests, basically copying what pendulum
> >> does
> >> > to
> >> > > > > check
> >> > > > > > > its behavior. The reason is that native implementations in
> the
> >> > past
> >> > > > > made
> >> > > > > > > serious mistakes and I do not put a lot of trust in them.
> >> > > > > > >
> >> > > > > > > An abstraction or vendoring in could be alternatives, but
> they
> >> > > bring
> >> > > > > > their
> >> > > > > > > own significant challenges.
> >> > > > > > >
> >> > > > > > > I have re-raised the issue here:
> >> > > > > > >
> >> > > > > > > https://github.com/sdispater/pendulum/issues/753
> >> > > > > > >
> >> > > > > > > The upside is that it seems the amount of issues with the
> >> beta is
> >> > > > > > limited,
> >> > > > > > > so hopefully the maintainers can spend a couple of cycles to
> >> > > address
> >> > > > > > them.
> >> > > > > > >
> >> > > > > > > Bolke
> >> > > > > > >
> >> > > > > > > On Fri, 17 Nov 2023 at 10:28, Andrey Anshin <
> >> > > > andrey.anshin@taragol.is>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > There is no changes in stable pendulum so let's try to
> >> continue
> >> > > > this
> >> > > > > > > > discussion and start think about "Plan B"
> >> > > > > > > >
> >> > > > > > > > Just a reminder:
> >> > > > > > > > - pendulum 2.1.2 released 3 years ago (at the time Airflow
> >> > > 1.10.x)
> >> > > > > > > > - pendulum 2 doesn't work well in Python 3.12, this is a
> >> > > > showstopper
> >> > > > > > for
> >> > > > > > > > the support Python 3.12
> >> > > > > > > > - pendulum 2 have memory leaks (
> >> > > > > > > > https://github.com/sdispater/pendulum/issues/720), and
> >> Airflow
> >> > > use
> >> > > > > > > > approach
> >> > > > > > > > to achieve this leaks, especially in K8S executor
> >> > > > > > > > - pendulum 2 doesn't use system tzdata by default, but we
> >> have
> >> > a
> >> > > > > > > > workaround, thanks Bolke for the documentation
> >> > > > > > > > - pendulum it is a core dependency of Airflow, and I guess
> >> we
> >> > > can't
> >> > > > > > > > remove/replace it without breaking changes.
> >> > > > > > > >
> >> > > > > > > > So my proposal if things won't change in the near future
> >> then
> >> > we
> >> > > > need
> >> > > > > > to
> >> > > > > > > > start removing the pendulum from the core and replace it
> >> with
> >> > the
> >> > > > > > native
> >> > > > > > > > datetime / zoneinfo / tzinfo. But maybe we have another
> way
> >> to
> >> > > > > resolve
> >> > > > > > > this
> >> > > > > > > > without breaking changes? Because for me it would be a
> >> little
> >> > > weird
> >> > > > > if
> >> > > > > > > > removal pendulum would be a main driver for release
> Airflow
> >> 3
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > ----
> >> > > > > > > > Best Wishes
> >> > > > > > > > *Andrey Anshin*
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Thu, 28 Sept 2023 at 13:01, Andrey Anshin <
> >> > > > > andrey.anshin@taragol.is
> >> > > > > > >
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > This discussion is more about the known problem of
> >> pendulum
> >> > and
> >> > > > how
> >> > > > > > we
> >> > > > > > > > > could deal with it and maybe how we (as Community) might
> >> help
> >> > > > > autor.
> >> > > > > > > > >
> >> > > > > > > > > The library is mostly supported by a single author
> >> Sébastien
> >> > > > > Eustace
> >> > > > > > (
> >> > > > > > > > > https://github.com/sdispater) and it seems like we bump
> >> into
> >> > > the
> >> > > > > > > > > situation which is described in xkcd #2347 (
> >> > > > > > > > > https://imgs.xkcd.com/comics/dependency.png). To be
> >> honest
> >> > it
> >> > > is
> >> > > > > not
> >> > > > > > > > > something new when library mainly supported by one
> author
> >> so
> >> > > > there
> >> > > > > is
> >> > > > > > > > > always a risk that the library will no longer be
> >> supported /
> >> > > > > > abandoned
> >> > > > > > > > > And if takes in account that pendulum provides core
> >> > > functionality
> >> > > > > in
> >> > > > > > > > > Airflow it could have dramatical impact in the future.
> >> > > > > > > > >
> >> > > > > > > > > Pendulum is a really nice library which helps a lot of
> >> > > developers
> >> > > > > to
> >> > > > > > > work
> >> > > > > > > > > with dates/datetimes. However there is one major
> problem,
> >> the
> >> > > > last
> >> > > > > > > > release
> >> > > > > > > > > of this library happened more than 3 years ago (
> >> > > > > > > > > https://pypi.org/project/pendulum/#history) in the time
> >> when
> >> > > > > Airflow
> >> > > > > > > > > 1.10.11 was released
> >> > > > > > > > >
> >> > > > > > > > > Fortunately, the project is not abandoned and on a
> regular
> >> > > basis
> >> > > > > > > commits
> >> > > > > > > > > add into the master branch. However these commits are
> not
> >> > > > included
> >> > > > > > into
> >> > > > > > > > any
> >> > > > > > > > > final release and that's why some things related to
> >> datetime
> >> > > > don't
> >> > > > > > work
> >> > > > > > > > as
> >> > > > > > > > > expected in Airflow. There are list of known (for me)
> >> issues
> >> > > > which
> >> > > > > > are
> >> > > > > > > > > affect Airflow
> >> > > > > > > > >
> >> > > > > > > > > *Memory Leak on parse*:
> >> > > > > > > > > - https://github.com/sdispater/pendulum/issues/720,
> this
> >> one
> >> > > > fixed
> >> > > > > > 2
> >> > > > > > > > > years ago but not available yet (
> >> > > > > > > > > https://github.com/sdispater/pendulum/pull/563). Since
> we
> >> > use
> >> > > > > parse
> >> > > > > > > > dates
> >> > > > > > > > > in airflow codebase: datetime parameters and datetime in
> >> logs
> >> > > > this
> >> > > > > > one
> >> > > > > > > > > could be a reason for memory leakage in Airflow:
> >> > > > > > > > > - https://github.com/apache/airflow/discussions/24694
> >> > > > > > > > > - https://github.com/apache/airflow/discussions/28597
> >> > > > > > > > >
> >> > > > > > > > > *Incorrect time zones*, known issues and should be
> already
> >> > > fixed
> >> > > > in
> >> > > > > > > > > master branch
> >> > > > > > > > > - https://github.com/sdispater/pendulum/issues/700,
> >> Mexico
> >> > do
> >> > > > not
> >> > > > > > use
> >> > > > > > > > DST
> >> > > > > > > > > anymore
> >> > > > > > > > > - https://github.com/sdispater/pendulum/issues/706,
> Egypt
> >> > > > > reinstate
> >> > > > > > > DST
> >> > > > > > > > >
> >> > > > > > > > > We add clarification in
> >> > > > > https://github.com/apache/airflow/pull/30467
> >> > > > > > ,
> >> > > > > > > > > however it seems like there is no other way rather than
> >> > > patching
> >> > > > > > > Pendulum
> >> > > > > > > > > right now.
> >> > > > > > > > >
> >> > > > > > > > > All these issues should be solved as soon as pendulum 3
> is
> >> > > > > released.
> >> > > > > > > The
> >> > > > > > > > > current announced estimation is end of september/
> >> beginning
> >> > of
> >> > > > > > October:
> >> > > > > > > > >
> >> > > > > > >
> >> > > > >
> >> > >
> >>
> https://github.com/sdispater/pendulum/issues/600#issuecomment-1711299677
> >> > > > > > > > >
> >> > > > > > > > > So in theory we would have a fixed version of pendulum
> >> soon,
> >> > > and
> >> > > > it
> >> > > > > > > might
> >> > > > > > > > > break something in Airflow but from my point of view it
> is
> >> > > better
> >> > > > > > than
> >> > > > > > > > > current status.
> >> > > > > > > > >
> >> > > > > > > > > However there might be a situation where the release of
> >> the
> >> > > > > pendulum
> >> > > > > > > > would
> >> > > > > > > > > be postponed, so maybe better to have a backup plan.
> What
> >> > could
> >> > > > we
> >> > > > > do
> >> > > > > > > in
> >> > > > > > > > > this case?
> >> > > > > > > > >
> >> > > > > > > > > Maybe we should start to use zoneinfo.ZoneInfo instead
> of
> >> > > > pendulum
> >> > > > > > > > > datetime?
> https://github.com/apache/airflow/issues/19450
> >> > > > > > > > > Pros:
> >> > > > > > > > > - stdlib (python 3.9+)
> >> > > > > > > > > - In pendulum 3.0 Timezone based on zoneinfo.Zoneinfo
> >> > > > > > > > >
> >> > > > > > > > > Cons:
> >> > > > > > > > > - Current serialization model can't deal with backport
> >> > > packages.
> >> > > > > E.g.
> >> > > > > > > > > timezone which are serialized in backport_zoneinfo can't
> >> be
> >> > > > > > > deserialized
> >> > > > > > > > in
> >> > > > > > > > > zoneinfo
> >> > > > > > > > >
> >> > > > > > > > > Maybe we should replace parse datetime with another
> >> solution.
> >> > > > Does
> >> > > > > > > anyone
> >> > > > > > > > > know a good replacement?
> >> > > > > > > > >
> >> > > > > > > > > Maybe someone from Airflow Community could propose their
> >> help
> >> > > > with
> >> > > > > > > > > maintenance of library:
> >> > > > > > > > > - https://github.com/sdispater/pendulum/issues/590
> >> > > > > > > > >
> >> > > > > > > > > Maybe we should get rid of the pendulum at all, as a
> last
> >> > > resort
> >> > > > > > > > solution.
> >> > > > > > > > > I can't imagine how we could do that, because a lot of
> >> stuff
> >> > > > > depends
> >> > > > > > on
> >> > > > > > > > the
> >> > > > > > > > > pendulum and removing it would be a breaking change.
> >> > > > > > > > >
> >> > > > > > > > > ----
> >> > > > > > > > > Best Wishes
> >> > > > > > > > > *Andrey Anshin*
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > --
> >> > > > > > >
> >> > > > > > > --
> >> > > > > > > Bolke de Bruin
> >> > > > > > > bdbruin@gmail.com
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>
RE: [DISCUSS] Future of Pendulum in Airflow
Posted by Damian Shaw <ds...@striketechnologies.com>.
FYI, I think the most common pain point for users will be using pendulum methods in macros, an example I have is this macro I have depended on very heavily:
{{ data_interval_end.in_tz(dag.timezone).format("YYYY_MM_DD") }}
This one is easy for me to fix, and fortunately some time back I consolidated all my team's date macro usage into a single file that gets sourced across all our DAGs, but likely across all Airflow users all of pendulum's methods will be used in macros somewhere.
I don't have an easy answer for that other than please throw deprecation warnings as soon and early as possible.
Damian
-----Original Message-----
From: Wei Lee <we...@gmail.com>
Sent: Tuesday, December 19, 2023 9:55 AM
To: dev@airflow.apache.org
Subject: Re: [DISCUSS] Future of Pendulum in Airflow
I’d vote for 2-a. Removing it in the upcoming minor release may not give enough time for airflow users to adjust. On the other hand, we would also want to decrease the amount of time we spend supporting various versions of Pendulum. 2.9 might be a reasonable compromise.
Best,
Wei
> On Dec 19, 2023, at 9:52 PM, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> I am for a) with deprecation warning when Pendulum 2 is installed in
> Airflow 2.8.1+.
>
> I understand we have a long way before we can get rid of Pendulum
> dependency in general as it will be a major pain for our users for a
> number of reasons, so I think getting rid of Pendulum is likely not
> easy / feasible in the short/medium term (though it would likely be
> best).
> Pendulum 3 solves the biggest pain (one of the things that blocks us
> from officially having Python 3.12 support) and when we consider the
> scope of the features we use from Pendulum, it's not very likely it
> will cause us similar pains even if we keep it for quite some time. So
> supporting Pendulum 3 while giving our users (and 3rd-party libraries)
> a chance to upgrade to Pendulum 3 seems a good balance between cost
> and risk - at least for a year or so until Python 3.13 will be out
> (with possibly some more breaking changes) or until some security
> issues will be discovered (and not fixed for some time) in Pendulum.
>
> J,
>
> On Tue, Dec 19, 2023 at 2:43 PM Andrey Anshin <an...@taragol.is> wrote:
>>
>> Greetings everyone!
>>
>> Because pendulum 3 was released we probably need to discuss about
>> support for the previous version of pendulum in Airflow core.
>>
>> There is not much options here
>>
>> Option 1: Drop support of next patch version of Airflow, e.g. 2.8.1.
>> It might (or might not) break someone's pipelines. In general we do
>> not count change upstream library version as breaking changes,
>> however this could be an exclusion, because `pendulum 2` first time
>> released in May 9, 2018 and there is potentially could be a lot of
>> user code which do not work properly with `pendulum 3`
>>
>>
>> Option 2: Keep support of both pendulum 2 and pendulum 3 for a while.
>> There are different suboptions a). Support pendulum 2 until Airflow
>> 2.9 b). Support pendulum 2 until the min Airflow version in providers
>> reaches Airflow 2.9, a bit more than 1 year. In Airflow 2.9 add user
>> warning about limited time of support c). Support pendulum 2 until it
>> is not painful d). Support pendulum 2 until Airflow 3 (Forever?)
>>
>>
>> On Mon, 18 Dec 2023 at 16:53, Bolke de Bruin <bd...@gmail.com> wrote:
>>
>>> Pendulum 3 was released.
>>>
>>> https://pypi.org/project/pendulum/
>>>
>>> Bolke
>>>
>>> On Fri, 1 Dec 2023 at 18:38, Aritra Basu <ar...@gmail.com> wrote:
>>>
>>>> I agree with wanting to stay away from having to deal with dst but
>>>> I am
>>> on
>>>> the side of biting the bullet and transitioning to datetime and
>>>> testing thoroughly, especially considering our current predicament
>>>> with pendulum and the potential of us being in the same place again
>>>> potentially in the future.
>>>>
>>>> --
>>>> Regards,
>>>> Aritra Basu
>>>>
>>>> On Fri, Dec 1, 2023, 10:27 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>
>>>>> Yeah. Agree it's a challenge but I think it's also an opportunity
>>>>> to fix those - because whether we admit it, or not - we already
>>>>> have those problems - regardless of Pendulum's use.
>>>>>
>>>>> If you look at the issues (especially ones created around DST
>>>>> time) -
>>>> there
>>>>> are always one or two issues (even in latest versions of Airflow)
>>>>> where
>>>> DST
>>>>> transition is causing people problems. And they are rarely
>>>>> solved/addressed.
>>>>>
>>>>> Example from last October:
>>>>>
>>>>> https://github.com/apache/airflow/issues/35272
>>>>>
>>>>> Example from May 2022:
>>>>>
>>>>> https://github.com/apache/airflow/issues/23400
>>>>>
>>>>> And Example from March 2020 ( which might or might not be fixed by
>>>>> the
>>>> open
>>>>> PR):
>>>>>
>>>>> https://github.com/apache/airflow/issues/7999
>>>>>
>>>>> J.
>>>>>
>>>>>
>>>>> On Fri, Dec 1, 2023 at 5:41 PM Bolke de Bruin <bd...@gmail.com>
>>> wrote:
>>>>>
>>>>>> The alternative is to vendor in or fork pendulum as part of the
>>> Airflow
>>>>>> project. Also a major undertaking but maybe less effort.
>>>>>>
>>>>>> We shouldn't just address this as a technical challenge. The
>>>>>> question
>>>>>> becomes: does that combination fornally do DST transitions
>>>>>> correctly,
>>>>> does
>>>>>> it do leap seconds correctly, does it add dates correctly, does
>>>>>> it do timezone traversal correctly and many many other things.
>>>>>>
>>>>>> So if we switch to the datetime + zone info + timezone we need to
>>> have
>>>> a
>>>>>> test suite that verifies this. Otherwise we potentially expose
>>>>>> our
>>>> users
>>>>> to
>>>>>> different behavior IN their data. Not just a schedule by in their
>>>>>> SQL
>>>> or
>>>>> in
>>>>>> some scripting where logical date / execution date is used or
>>>>>> where
>>>> date
>>>>>> arithmetic is applied.
>>>>>>
>>>>>> So it is not changing one car for another. It is changing the
>>>>>> means
>>> of
>>>>>> transportation and not knowing if you can get to all the places
>>>>>> you
>>>> want
>>>>> /
>>>>>> need to go as before.
>>>>>>
>>>>>> Honestly, I'm a bit scared (I do see the lack of responsiveness).
>>>>>> I
>>>> hope
>>>>>> someone can take that away.
>>>>>>
>>>>>> Bolke
>>>>>>
>>>>>>
>>>>>> Sent from my iPhone
>>>>>>
>>>>>>> On 1 Dec 2023, at 13:39, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>>>
>>>>>>> Agree. I think this is the best way forward.- we have to bite
>>>>>>> the
>>>>>> bullet of
>>>>>>> potential backwards compatibility issues and separating out a
>>> compat
>>>>>> layer
>>>>>>> where we could optionally use pendulum instead would be the most
>>>> "easy"
>>>>>>> approach for our users. Most of them would not even notice if we
>>> do a
>>>>>> good
>>>>>>> job, but those who somehow depend on pendulum objects would get
>>>>>>> a possibility to switch back.
>>>>>>>
>>>>>>>> On Fri, Dec 1, 2023 at 1:09 PM Andrey Anshin <
>>>>> andrey.anshin@taragol.is>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> IMHO, if we would like to replace pendulum to alternatives, the
>>>> better
>>>>>>>> alternative would be combination of datetime.datetime +
>>>>>>>> zoneinfo + datetime.timezone
>>>>>>>>
>>>>>>>> We still do hacks around pendulum functional, which makes hacks
>>>> around
>>>>>>>> datetime.datetime, so in my perspective better remove redundant
>>>> layers
>>>>>> and
>>>>>>>> use native objects directly.
>>>>>>>>
>>>>>>>> What confuses me the most is how we get around type changes in
>>> Tasks
>>>>>>>> Context. All dates return as pendulum.DateTime for a very long
>>> time
>>>>>>>> (literally for ages) and it is probably could be classified as
>>>>> breaking
>>>>>>>> changes, because it might break quite a bit users pipelines. It
>>>> still
>>>>>> could
>>>>>>>> be resolved as optional pendulum dependency + config option for
>>>>> datetime
>>>>>>>> types in context, which might help to get familiar types for a
>>>> while.
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Wed, 29 Nov 2023 at 05:58, Jarek Potiuk <ja...@potiuk.com>
>>>> wrote:
>>>>>>>>>
>>>>>>>>> "infamous fork of " should be "fork of infamous Akka" - just
>>>>>>>>> to
>>> be
>>>>>> clear
>>>>>>>> :)
>>>>>>>>>
>>>>>>>>>> On Wed, Nov 29, 2023 at 2:57 AM Jarek Potiuk
>>>>>>>>>> <ja...@potiuk.com>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> It's a very rare occurrence that ASF accepts forks. Usually
>>>>>>>>>> it
>>> has
>>>>> to
>>>>>>>> be
>>>>>>>>>> willingly donated by those who own the IP rights, - mainly
>>> because
>>>>> of
>>>>>>>>>> trademark issues. Licence is one thing, but carrying things
>>>>>>>>>> like
>>>> the
>>>>>>>> name
>>>>>>>>>> of the project is not possible without clearing IP and
>>>>>>>>>> donating
>>>> the
>>>>>>>>>> project.
>>>>>>>>>>
>>>>>>>>>> It happened last time last year - with Pekko project -
>>>>>>>>>> infamous
>>>> fork
>>>>>> or
>>>>>>>>>> Akka after Lightbend changed licensing for Akka -
>>>>>>>>>>
>>>>>>>>
>>>>>
>>> https://www.lightbend.com/blog/why-we-are-changing-the-license-for-a
>>> kka
>>>>>>>>>> making it impossible to be used in ASF projects with the new
>>>>> licence.
>>>>>>>> But
>>>>>>>>>> it was a really "strong" case and the "we do not accept forks
>>>>> usually"
>>>>>>>>> was
>>>>>>>>>> quite a contentious issue (I followed the discussion - it was
>>>>>>>> fascinating
>>>>>>>>>> actually). It tool a LOT of time and effort for those who led
>>> the
>>>>>>>> effort
>>>>>>>>>> eventually - and also mostly because things had to be
>>>> crystal-clear
>>>>>> and
>>>>>>>>> not
>>>>>>>>>> "litigable" because the other side had some business reasons
>>>>>>>>>> to
>>>> make
>>>>>>>> the
>>>>>>>>>> licensing changes and they did not understood why it is
>>> impossible
>>>>> to
>>>>>>>> use
>>>>>>>>>> Akka as-is even with all the exclusions they offered to ASF -
>>> and
>>>>>>>>>> effectively it was a "hostile" move.
>>>>>>>>>>
>>>>>>>>>> I don't think it's "worth" it in this case to be honest -
>>>> especially
>>>>>> if
>>>>>>>>>> the maintainers are effectively "silent" and seem to ignore
>>>> problems
>>>>>> of
>>>>>>>>> one
>>>>>>>>>> of the really serious users they have. If they were willing
>>>>>>>>>> to
>>>>>>>> cooperate,
>>>>>>>>>> we could do a lot, but when we are left alone, I think it
>>>>>>>>>> will
>>> be
>>>>> far
>>>>>>>>> more
>>>>>>>>>> convenient for us to look for alternatives - to be honest.
>>>>>>>>>>
>>>>>>>>>> J.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Nov 29, 2023 at 2:46 AM Austin Bennett <
>>> austin@apache.org
>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> An option would be to fork Pendulum? It is MIT Licensed, I
>>> don't
>>>>>> know
>>>>>>>>>>> whether that poses problems to get in ASF?
>>>>>>>>>>>
>>>>>>>>>>> If forking (?) which is somewhat non-ideal, would we want
>>>>>>>>>>> that
>>>> 'in'
>>>>>>>>>>> airflow?
>>>>>>>>>>>
>>>>>>>>>>> If not 'in' airflow, I wonder if ASF incubator would accept
>>>>>>>>>>> a
>>>>> forked
>>>>>>>>>>> project? [ anecdotally Linux Foundation has incubated
>>> Starrocks
>>>>>> which
>>>>>>>>> is
>>>>>>>>>>> a
>>>>>>>>>>> fork of Apache Doris, so there's precedence for this to be
>>>>>>>>>>> OK
>>> in
>>>>> the
>>>>>>>>> wider
>>>>>>>>>>> OSS ecosystem ].
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Nov 28, 2023 at 6:41 PM Jarek Potiuk
>>>>>>>>>>> <jarek@potiuk.com
>>>>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I have not heard back from maintainers, just a comment from
>>>>> someone
>>>>>>>>> else
>>>>>>>>>>>> who suggested donating a pendulum to the ASF (which is
>>>>>>>>>>>> kinda
>>>>>>>>> interesting
>>>>>>>>>>>> idea). I followed up, let's see. I think if we do not hear
>>> back
>>>>> and
>>>>>>>>>>> there
>>>>>>>>>>>> will be another week or two where "Pendulum 3 supporting
>>>>>>>>>>>> 3.12
>>>>>>>> support
>>>>>>>>>>> "is
>>>>>>>>>>>> coming" but no-one knows when, effectively means that we
>>> should
>>>>> look
>>>>>>>>>>> for an
>>>>>>>>>>>> alternative like NOW.
>>>>>>>>>>>>
>>>>>>>>>>>> J.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Nov 22, 2023 at 3:27 AM Daniel Standish
>>>>>>>>>>>> <da...@astronomer.io.invalid> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks Jarek
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Nov 21, 2023, 4:34 PM Jarek Potiuk
>>>>>>>>>>>>> <jarek@potiuk.com
>>>>
>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think we miss important insight - straight from the
>>> source.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I believe it's time to be candid and simply ask questions
>>> for
>>>>>>>> the
>>>>>>>>>>>> future
>>>>>>>>>>>>> of
>>>>>>>>>>>>>> Pendulum directly where we should - ie. we should just
>>>>>>>>>>>>>> ask
>>>>>>>>>>>> maintainers.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've just started a very candid and open discussion there
>>>>>>>>>>>>>> -
>>>>>>>>>>>>>> https://github.com/sdispater/pendulum/discussions/771 .
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I linked back to this discussion and explained where we
>>>>>>>>>>>>>> are
>>>>>>>> coming
>>>>>>>>>>>> from.
>>>>>>>>>>>>> I
>>>>>>>>>>>>>> even offered an option, that maybe - if they accept some
>>> help
>>>>>>>> and
>>>>>>>>>>> would
>>>>>>>>>>>>> be
>>>>>>>>>>>>>> open to make a few of us maintainers of Pendulum, that I
>>> will
>>>>>>>> ask
>>>>>>>>> if
>>>>>>>>>>>> some
>>>>>>>>>>>>>> of the maintainers here in Airflow would like to step up
>>>>>>>>>>>>>> (I
>>>>>>>> would
>>>>>>>>> be
>>>>>>>>>>>>>> willing to - for one) and help to at least move Pendulum
>>>> through
>>>>>>>>>>> 3.12
>>>>>>>>>>>> and
>>>>>>>>>>>>>> maybe keep it running for as long as it will be needed to
>>>>>>>> maintain
>>>>>>>>>>> it.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think if they would be open to some of us helping them
>>>>>>>>>>>>>> -
>>>> this
>>>>>>>>>>> might
>>>>>>>>>>>> be
>>>>>>>>>>>>>> actually the "cheapest" option for us to be honest at
>>>>>>>>>>>>>> least
>>>> in a
>>>>>>>>>>>> mid-term
>>>>>>>>>>>>>> if we could gain influence and even take part and speed
>>>>>>>>>>>>>> up
>>>>>>>>> releases
>>>>>>>>>>> and
>>>>>>>>>>>>>> merging of issues that are blocking us (or would be
>>>>>>>>>>>>>> blocking
>>>> us
>>>>>>>> in
>>>>>>>>>>> the
>>>>>>>>>>>>>> future).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I tried to be very friendly but candid and direct on what
>>> kind
>>>>>>>> of
>>>>>>>>>>>>> problems
>>>>>>>>>>>>>> it creates us and also expressed our thanks for
>>>>>>>>>>>>>> supporting
>>> us
>>>>>>>> for
>>>>>>>>> so
>>>>>>>>>>>> long
>>>>>>>>>>>>>> (I do believe they deserve it). I think the sole (pretty
>>>> much)
>>>>>>>>>>>>> maintainer
>>>>>>>>>>>>>> of Pendulum might really have a hard time maintaining and
>>>>>>>> keeping
>>>>>>>>>>> it up
>>>>>>>>>>>>> for
>>>>>>>>>>>>>> years - without being paid for and thanked. I hope -
>>>>>>>>>>>>>> maybe
>>> all
>>>>>>>>> that
>>>>>>>>>>>> they
>>>>>>>>>>>>>> need are a few words of encouragement and thanks and
>>> realising
>>>>>>>>> that
>>>>>>>>>>>>> others
>>>>>>>>>>>>>> depend on their work - but also seeing that they are
>>>> understood,
>>>>>>>>>>>>> empathised
>>>>>>>>>>>>>> with and that there are viable alternatives they might
>>> follow
>>>>>>>>> might
>>>>>>>>>>> be
>>>>>>>>>>>>>> helpful for them to be also candid and respond. I hope my
>>>>>>>>> intentions
>>>>>>>>>>>> will
>>>>>>>>>>>>>> be understood - that I am not complaining, but even
>>>>>>>>>>>>>> trying
>>> to
>>>>>>>>> help.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Let\s see where it gets us.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> J.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Nov 17, 2023 at 5:32 PM Andrey Anshin <
>>>>>>>>>>>> andrey.anshin@taragol.is>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Just to clarify I'd like us to consider the possibility
>>> that
>>>>>>>> no
>>>>>>>>>>> new
>>>>>>>>>>>>>>> pendulum would be released or released at the end of
>>>>>>>>>>>>>>> 2024,
>>>>>>>> like
>>>>>>>>> a
>>>>>>>>>>>>>>> pessimistic scenario:
>>>>>>>>>>>>>>> - What should we do in this case?
>>>>>>>>>>>>>>> - Work out a backup plan.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ----
>>>>>>>>>>>>>>> Best Wishes
>>>>>>>>>>>>>>> *Andrey Anshin*
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, 17 Nov 2023 at 16:33, Jarek Potiuk <
>>> jarek@potiuk.com
>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Also I think TP - had a document in the past (years
>>>>>>>>>>>>>>>> ago)
>>>>>>>>>>>>> describing a
>>>>>>>>>>>>>>>> draft of a more complete alternative we can take to
>>> approach
>>>>>>>>>>>> datetime
>>>>>>>>>>>>>> vs.
>>>>>>>>>>>>>>>> pendulum dichotomy. I cannot easily find the document
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>> discussion
>>>>>>>>>>>>> -
>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>> I do remember it was proposing some interesting changes
>>>>>>>>>>>>>>>> in
>>>>>>>> the
>>>>>>>>>>>>> approach
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> Airflow to have an abstraction layer on top of it (as
>>>>>>>>>>>>>>>> far
>>>>>>>> as I
>>>>>>>>>>>>>> remember).
>>>>>>>>>>>>>>>> Maybe we can resurrect that idea if TP might find the
>>>>>>>>> proposal ?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Nov 17, 2023 at 1:06 PM Bolke de Bruin <
>>>>>>>>>>> bdbruin@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I agree that the current speed of development of
>>>>>>>>>>>>>>>>> Pendulum
>>>>>>>>>>> leaves
>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>> to be desired. However, I think we should not
>>>>>>>> underestimate
>>>>>>>>>>> the
>>>>>>>>>>>>>> effort
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> replacing it. It is not just a matter of
>>>>>>>>>>> %s/pendulum/datetime/g
>>>>>>>>>>>> so
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> say.
>>>>>>>>>>>>>>>>> If we are *truly* thinking about moving to native
>>>>>>>> datetime /
>>>>>>>>>>>>> zoneinfo
>>>>>>>>>>>>>>> etc
>>>>>>>>>>>>>>>>> we need *extensive* tests, basically copying what
>>> pendulum
>>>>>>>>>>> does
>>>>>>>>>>>> to
>>>>>>>>>>>>>>> check
>>>>>>>>>>>>>>>>> its behavior. The reason is that native
>>>>>>>>>>>>>>>>> implementations
>>> in
>>>>>>>>> the
>>>>>>>>>>>> past
>>>>>>>>>>>>>>> made
>>>>>>>>>>>>>>>>> serious mistakes and I do not put a lot of trust in them.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> An abstraction or vendoring in could be alternatives,
>>>>>>>>>>>>>>>>> but
>>>>>>>>> they
>>>>>>>>>>>>> bring
>>>>>>>>>>>>>>>> their
>>>>>>>>>>>>>>>>> own significant challenges.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I have re-raised the issue here:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> https://github.com/sdispater/pendulum/issues/753
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The upside is that it seems the amount of issues with
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>> beta is
>>>>>>>>>>>>>>>> limited,
>>>>>>>>>>>>>>>>> so hopefully the maintainers can spend a couple of
>>>>>>>>>>>>>>>>> cycles
>>>>>>>> to
>>>>>>>>>>>>> address
>>>>>>>>>>>>>>>> them.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, 17 Nov 2023 at 10:28, Andrey Anshin <
>>>>>>>>>>>>>> andrey.anshin@taragol.is>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> There is no changes in stable pendulum so let's try
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>> continue
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> discussion and start think about "Plan B"
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Just a reminder:
>>>>>>>>>>>>>>>>>> - pendulum 2.1.2 released 3 years ago (at the time
>>>>>>>> Airflow
>>>>>>>>>>>>> 1.10.x)
>>>>>>>>>>>>>>>>>> - pendulum 2 doesn't work well in Python 3.12, this
>>>>>>>>>>>>>>>>>> is a
>>>>>>>>>>>>>> showstopper
>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>> the support Python 3.12
>>>>>>>>>>>>>>>>>> - pendulum 2 have memory leaks (
>>>>>>>>>>>>>>>>>> https://github.com/sdispater/pendulum/issues/720),
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>> Airflow
>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>> approach
>>>>>>>>>>>>>>>>>> to achieve this leaks, especially in K8S executor
>>>>>>>>>>>>>>>>>> - pendulum 2 doesn't use system tzdata by default,
>>>>>>>>>>>>>>>>>> but
>>>>>>>> we
>>>>>>>>>>> have
>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>> workaround, thanks Bolke for the documentation
>>>>>>>>>>>>>>>>>> - pendulum it is a core dependency of Airflow, and I
>>>>>>>> guess
>>>>>>>>>>> we
>>>>>>>>>>>>> can't
>>>>>>>>>>>>>>>>>> remove/replace it without breaking changes.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> So my proposal if things won't change in the near
>>>>>>>>>>>>>>>>>> future
>>>>>>>>>>> then
>>>>>>>>>>>> we
>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> start removing the pendulum from the core and replace
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>> with
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> native
>>>>>>>>>>>>>>>>>> datetime / zoneinfo / tzinfo. But maybe we have
>>>>>>>>>>>>>>>>>> another
>>>>>>>>> way
>>>>>>>>>>> to
>>>>>>>>>>>>>>> resolve
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> without breaking changes? Because for me it would be
>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>> little
>>>>>>>>>>>>> weird
>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>> removal pendulum would be a main driver for release
>>>>>>>>> Airflow
>>>>>>>>>>> 3
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ----
>>>>>>>>>>>>>>>>>> Best Wishes
>>>>>>>>>>>>>>>>>> *Andrey Anshin*
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thu, 28 Sept 2023 at 13:01, Andrey Anshin <
>>>>>>>>>>>>>>> andrey.anshin@taragol.is
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> This discussion is more about the known problem of
>>>>>>>>>>> pendulum
>>>>>>>>>>>> and
>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>> could deal with it and maybe how we (as Community)
>>>>>>>> might
>>>>>>>>>>> help
>>>>>>>>>>>>>>> autor.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The library is mostly supported by a single author
>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>> Eustace
>>>>>>>>>>>>>>>> (
>>>>>>>>>>>>>>>>>>> https://github.com/sdispater) and it seems like we
>>>>>>>> bump
>>>>>>>>>>> into
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> situation which is described in xkcd #2347 (
>>>>>>>>>>>>>>>>>>> https://imgs.xkcd.com/comics/dependency.png). To be
>>>>>>>>>>> honest
>>>>>>>>>>>> it
>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>> something new when library mainly supported by one
>>>>>>>>> author
>>>>>>>>>>> so
>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> always a risk that the library will no longer be
>>>>>>>>>>> supported /
>>>>>>>>>>>>>>>> abandoned
>>>>>>>>>>>>>>>>>>> And if takes in account that pendulum provides core
>>>>>>>>>>>>> functionality
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>> Airflow it could have dramatical impact in the future.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Pendulum is a really nice library which helps a lot
>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>> developers
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>>>>>> with dates/datetimes. However there is one major
>>>>>>>>> problem,
>>>>>>>>>>> the
>>>>>>>>>>>>>> last
>>>>>>>>>>>>>>>>>> release
>>>>>>>>>>>>>>>>>>> of this library happened more than 3 years ago (
>>>>>>>>>>>>>>>>>>> https://pypi.org/project/pendulum/#history) in the
>>>>>>>> time
>>>>>>>>>>> when
>>>>>>>>>>>>>>> Airflow
>>>>>>>>>>>>>>>>>>> 1.10.11 was released
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Fortunately, the project is not abandoned and on a
>>>>>>>>> regular
>>>>>>>>>>>>> basis
>>>>>>>>>>>>>>>>> commits
>>>>>>>>>>>>>>>>>>> add into the master branch. However these commits
>>>>>>>>>>>>>>>>>>> are
>>>>>>>>> not
>>>>>>>>>>>>>> included
>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>> final release and that's why some things related to
>>>>>>>>>>> datetime
>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>> expected in Airflow. There are list of known (for
>>>>>>>>>>>>>>>>>>> me)
>>>>>>>>>>> issues
>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>> affect Airflow
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> *Memory Leak on parse*:
>>>>>>>>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/720,
>>>>>>>>> this
>>>>>>>>>>> one
>>>>>>>>>>>>>> fixed
>>>>>>>>>>>>>>>> 2
>>>>>>>>>>>>>>>>>>> years ago but not available yet (
>>>>>>>>>>>>>>>>>>> https://github.com/sdispater/pendulum/pull/563).
>>>>>>>> Since
>>>>>>>>> we
>>>>>>>>>>>> use
>>>>>>>>>>>>>>> parse
>>>>>>>>>>>>>>>>>> dates
>>>>>>>>>>>>>>>>>>> in airflow codebase: datetime parameters and
>>>>>>>>>>>>>>>>>>> datetime
>>>>>>>> in
>>>>>>>>>>> logs
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>>>>> could be a reason for memory leakage in Airflow:
>>>>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>> https://github.com/apache/airflow/discussions/24694
>>>>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>> https://github.com/apache/airflow/discussions/28597
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> *Incorrect time zones*, known issues and should be
>>>>>>>>> already
>>>>>>>>>>>>> fixed
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>> master branch
>>>>>>>>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/700,
>>>>>>>>>>> Mexico
>>>>>>>>>>>> do
>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>> DST
>>>>>>>>>>>>>>>>>>> anymore
>>>>>>>>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/706,
>>>>>>>>> Egypt
>>>>>>>>>>>>>>> reinstate
>>>>>>>>>>>>>>>>> DST
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> We add clarification in
>>>>>>>>>>>>>>> https://github.com/apache/airflow/pull/30467
>>>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>>>>>>> however it seems like there is no other way rather
>>>>>>>> than
>>>>>>>>>>>>> patching
>>>>>>>>>>>>>>>>> Pendulum
>>>>>>>>>>>>>>>>>>> right now.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> All these issues should be solved as soon as
>>>>>>>>>>>>>>>>>>> pendulum
>>>>>>>> 3
>>>>>>>>> is
>>>>>>>>>>>>>>> released.
>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>> current announced estimation is end of september/
>>>>>>>>>>> beginning
>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> October:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>> https://github.com/sdispater/pendulum/issues/600#issuecomment-17112
>>>> 99677
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> So in theory we would have a fixed version of
>>>>>>>>>>>>>>>>>>> pendulum
>>>>>>>>>>> soon,
>>>>>>>>>>>>> and
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> might
>>>>>>>>>>>>>>>>>>> break something in Airflow but from my point of view
>>>>>>>> it
>>>>>>>>> is
>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>> current status.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> However there might be a situation where the release
>>>>>>>> of
>>>>>>>>>>> the
>>>>>>>>>>>>>>> pendulum
>>>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>>> be postponed, so maybe better to have a backup plan.
>>>>>>>>> What
>>>>>>>>>>>> could
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>> this case?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Maybe we should start to use zoneinfo.ZoneInfo
>>>>>>>>>>>>>>>>>>> instead
>>>>>>>>> of
>>>>>>>>>>>>>> pendulum
>>>>>>>>>>>>>>>>>>> datetime?
>>>>>>>>> https://github.com/apache/airflow/issues/19450
>>>>>>>>>>>>>>>>>>> Pros:
>>>>>>>>>>>>>>>>>>> - stdlib (python 3.9+)
>>>>>>>>>>>>>>>>>>> - In pendulum 3.0 Timezone based on
>>>>>>>>>>>>>>>>>>> zoneinfo.Zoneinfo
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Cons:
>>>>>>>>>>>>>>>>>>> - Current serialization model can't deal with
>>>>>>>>>>>>>>>>>>> backport
>>>>>>>>>>>>> packages.
>>>>>>>>>>>>>>> E.g.
>>>>>>>>>>>>>>>>>>> timezone which are serialized in backport_zoneinfo
>>>>>>>> can't
>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> deserialized
>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>> zoneinfo
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Maybe we should replace parse datetime with another
>>>>>>>>>>> solution.
>>>>>>>>>>>>>> Does
>>>>>>>>>>>>>>>>> anyone
>>>>>>>>>>>>>>>>>>> know a good replacement?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Maybe someone from Airflow Community could propose
>>>>>>>> their
>>>>>>>>>>> help
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>> maintenance of library:
>>>>>>>>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/590
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Maybe we should get rid of the pendulum at all, as a
>>>>>>>>> last
>>>>>>>>>>>>> resort
>>>>>>>>>>>>>>>>>> solution.
>>>>>>>>>>>>>>>>>>> I can't imagine how we could do that, because a lot
>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>> stuff
>>>>>>>>>>>>>>> depends
>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> pendulum and removing it would be a breaking change.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ----
>>>>>>>>>>>>>>>>>>> Best Wishes
>>>>>>>>>>>>>>>>>>> *Andrey Anshin*
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Bolke de Bruin
>>>>>>>>>>>>>>>>> bdbruin@gmail.com
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>> -----------------------------------------------------------------
>>>>>> ---- To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> --
>>> Bolke de Bruin
>>> bdbruin@gmail.com
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
________________________________
Strike Technologies, LLC (“Strike”) is part of the GTS family of companies. Strike is a technology solutions provider, and is not a broker or dealer and does not transact any securities related business directly whatsoever. This communication is the property of Strike and its affiliates, and does not constitute an offer to sell or the solicitation of an offer to buy any security in any jurisdiction. It is intended only for the person to whom it is addressed and may contain information that is privileged, confidential, or otherwise protected from disclosure. Distribution or copying of this communication, or the information contained herein, by anyone other than the intended recipient is prohibited. If you have received this communication in error, please immediately notify Strike at info@striketechnologies.com, and delete and destroy any copies hereof.
________________________________
CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any attachments are intended solely for the addressee. This transmission is covered by the Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The information contained in this transmission is confidential in nature and protected from further use or disclosure under U.S. Pub. L. 106-102, 113 U.S. Stat. 1338 (1999), and may be subject to attorney-client or other legal privilege. Your use or disclosure of this information for any purpose other than that intended by its transmittal is strictly prohibited, and may subject you to fines and/or penalties under federal and state law. If you are not the intended recipient of this transmission, please DESTROY ALL COPIES RECEIVED and confirm destruction to the sender via return transmittal.
Re: [DISCUSS] Future of Pendulum in Airflow
Posted by Wei Lee <we...@gmail.com>.
I’d vote for 2-a. Removing it in the upcoming minor release may not give enough time for airflow users to adjust. On the other hand, we would also want to decrease the amount of time we spend supporting various versions of Pendulum. 2.9 might be a reasonable compromise.
Best,
Wei
> On Dec 19, 2023, at 9:52 PM, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> I am for a) with deprecation warning when Pendulum 2 is installed in
> Airflow 2.8.1+.
>
> I understand we have a long way before we can get rid of Pendulum
> dependency in general as it will be a major pain for our users for a
> number of reasons, so I think getting rid of Pendulum is likely not
> easy / feasible in the short/medium term (though it would likely be
> best).
> Pendulum 3 solves the biggest pain (one of the things that blocks us
> from officially having Python 3.12 support) and when we consider the
> scope of the features we use from Pendulum, it's not very likely it
> will cause us similar pains even if we keep it for quite some time. So
> supporting Pendulum 3 while giving our users (and 3rd-party libraries)
> a chance to upgrade to Pendulum 3 seems a good balance between cost
> and risk - at least for a year or so until Python 3.13 will be out
> (with possibly some more breaking changes) or until some security
> issues will be discovered (and not fixed for some time) in Pendulum.
>
> J,
>
> On Tue, Dec 19, 2023 at 2:43 PM Andrey Anshin <an...@taragol.is> wrote:
>>
>> Greetings everyone!
>>
>> Because pendulum 3 was released we probably need to discuss about support
>> for the previous version of pendulum in Airflow core.
>>
>> There is not much options here
>>
>> Option 1: Drop support of next patch version of Airflow, e.g. 2.8.1. It
>> might (or might not) break someone's pipelines. In general we do not count
>> change upstream library version as breaking changes, however this could be
>> an exclusion, because `pendulum 2` first time released in May 9, 2018 and
>> there is potentially could be a lot of user code which do not work properly
>> with `pendulum 3`
>>
>>
>> Option 2: Keep support of both pendulum 2 and pendulum 3 for a while. There
>> are different suboptions
>> a). Support pendulum 2 until Airflow 2.9
>> b). Support pendulum 2 until the min Airflow version in providers reaches
>> Airflow 2.9, a bit more than 1 year. In Airflow 2.9 add user warning about
>> limited time of support
>> c). Support pendulum 2 until it is not painful
>> d). Support pendulum 2 until Airflow 3 (Forever?)
>>
>>
>> On Mon, 18 Dec 2023 at 16:53, Bolke de Bruin <bd...@gmail.com> wrote:
>>
>>> Pendulum 3 was released.
>>>
>>> https://pypi.org/project/pendulum/
>>>
>>> Bolke
>>>
>>> On Fri, 1 Dec 2023 at 18:38, Aritra Basu <ar...@gmail.com> wrote:
>>>
>>>> I agree with wanting to stay away from having to deal with dst but I am
>>> on
>>>> the side of biting the bullet and transitioning to datetime and testing
>>>> thoroughly, especially considering our current predicament with pendulum
>>>> and the potential of us being in the same place again potentially in the
>>>> future.
>>>>
>>>> --
>>>> Regards,
>>>> Aritra Basu
>>>>
>>>> On Fri, Dec 1, 2023, 10:27 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>
>>>>> Yeah. Agree it's a challenge but I think it's also an opportunity to
>>>>> fix those - because whether we admit it, or not - we already have those
>>>>> problems - regardless of Pendulum's use.
>>>>>
>>>>> If you look at the issues (especially ones created around DST time) -
>>>> there
>>>>> are always one or two issues (even in latest versions of Airflow) where
>>>> DST
>>>>> transition is causing people problems. And they are rarely
>>>>> solved/addressed.
>>>>>
>>>>> Example from last October:
>>>>>
>>>>> https://github.com/apache/airflow/issues/35272
>>>>>
>>>>> Example from May 2022:
>>>>>
>>>>> https://github.com/apache/airflow/issues/23400
>>>>>
>>>>> And Example from March 2020 ( which might or might not be fixed by the
>>>> open
>>>>> PR):
>>>>>
>>>>> https://github.com/apache/airflow/issues/7999
>>>>>
>>>>> J.
>>>>>
>>>>>
>>>>> On Fri, Dec 1, 2023 at 5:41 PM Bolke de Bruin <bd...@gmail.com>
>>> wrote:
>>>>>
>>>>>> The alternative is to vendor in or fork pendulum as part of the
>>> Airflow
>>>>>> project. Also a major undertaking but maybe less effort.
>>>>>>
>>>>>> We shouldn't just address this as a technical challenge. The question
>>>>>> becomes: does that combination fornally do DST transitions correctly,
>>>>> does
>>>>>> it do leap seconds correctly, does it add dates correctly, does it do
>>>>>> timezone traversal correctly and many many other things.
>>>>>>
>>>>>> So if we switch to the datetime + zone info + timezone we need to
>>> have
>>>> a
>>>>>> test suite that verifies this. Otherwise we potentially expose our
>>>> users
>>>>> to
>>>>>> different behavior IN their data. Not just a schedule by in their SQL
>>>> or
>>>>> in
>>>>>> some scripting where logical date / execution date is used or where
>>>> date
>>>>>> arithmetic is applied.
>>>>>>
>>>>>> So it is not changing one car for another. It is changing the means
>>> of
>>>>>> transportation and not knowing if you can get to all the places you
>>>> want
>>>>> /
>>>>>> need to go as before.
>>>>>>
>>>>>> Honestly, I'm a bit scared (I do see the lack of responsiveness). I
>>>> hope
>>>>>> someone can take that away.
>>>>>>
>>>>>> Bolke
>>>>>>
>>>>>>
>>>>>> Sent from my iPhone
>>>>>>
>>>>>>> On 1 Dec 2023, at 13:39, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>>>
>>>>>>> Agree. I think this is the best way forward.- we have to bite the
>>>>>> bullet of
>>>>>>> potential backwards compatibility issues and separating out a
>>> compat
>>>>>> layer
>>>>>>> where we could optionally use pendulum instead would be the most
>>>> "easy"
>>>>>>> approach for our users. Most of them would not even notice if we
>>> do a
>>>>>> good
>>>>>>> job, but those who somehow depend on pendulum objects would get a
>>>>>>> possibility to switch back.
>>>>>>>
>>>>>>>> On Fri, Dec 1, 2023 at 1:09 PM Andrey Anshin <
>>>>> andrey.anshin@taragol.is>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> IMHO, if we would like to replace pendulum to alternatives, the
>>>> better
>>>>>>>> alternative would be combination of datetime.datetime + zoneinfo +
>>>>>>>> datetime.timezone
>>>>>>>>
>>>>>>>> We still do hacks around pendulum functional, which makes hacks
>>>> around
>>>>>>>> datetime.datetime, so in my perspective better remove redundant
>>>> layers
>>>>>> and
>>>>>>>> use native objects directly.
>>>>>>>>
>>>>>>>> What confuses me the most is how we get around type changes in
>>> Tasks
>>>>>>>> Context. All dates return as pendulum.DateTime for a very long
>>> time
>>>>>>>> (literally for ages) and it is probably could be classified as
>>>>> breaking
>>>>>>>> changes, because it might break quite a bit users pipelines. It
>>>> still
>>>>>> could
>>>>>>>> be resolved as optional pendulum dependency + config option for
>>>>> datetime
>>>>>>>> types in context, which might help to get familiar types for a
>>>> while.
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Wed, 29 Nov 2023 at 05:58, Jarek Potiuk <ja...@potiuk.com>
>>>> wrote:
>>>>>>>>>
>>>>>>>>> "infamous fork of " should be "fork of infamous Akka" - just to
>>> be
>>>>>> clear
>>>>>>>> :)
>>>>>>>>>
>>>>>>>>>> On Wed, Nov 29, 2023 at 2:57 AM Jarek Potiuk <ja...@potiuk.com>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> It's a very rare occurrence that ASF accepts forks. Usually it
>>> has
>>>>> to
>>>>>>>> be
>>>>>>>>>> willingly donated by those who own the IP rights, - mainly
>>> because
>>>>> of
>>>>>>>>>> trademark issues. Licence is one thing, but carrying things like
>>>> the
>>>>>>>> name
>>>>>>>>>> of the project is not possible without clearing IP and donating
>>>> the
>>>>>>>>>> project.
>>>>>>>>>>
>>>>>>>>>> It happened last time last year - with Pekko project - infamous
>>>> fork
>>>>>> or
>>>>>>>>>> Akka after Lightbend changed licensing for Akka -
>>>>>>>>>>
>>>>>>>>
>>>>>
>>> https://www.lightbend.com/blog/why-we-are-changing-the-license-for-akka
>>>>>>>>>> making it impossible to be used in ASF projects with the new
>>>>> licence.
>>>>>>>> But
>>>>>>>>>> it was a really "strong" case and the "we do not accept forks
>>>>> usually"
>>>>>>>>> was
>>>>>>>>>> quite a contentious issue (I followed the discussion - it was
>>>>>>>> fascinating
>>>>>>>>>> actually). It tool a LOT of time and effort for those who led
>>> the
>>>>>>>> effort
>>>>>>>>>> eventually - and also mostly because things had to be
>>>> crystal-clear
>>>>>> and
>>>>>>>>> not
>>>>>>>>>> "litigable" because the other side had some business reasons to
>>>> make
>>>>>>>> the
>>>>>>>>>> licensing changes and they did not understood why it is
>>> impossible
>>>>> to
>>>>>>>> use
>>>>>>>>>> Akka as-is even with all the exclusions they offered to ASF -
>>> and
>>>>>>>>>> effectively it was a "hostile" move.
>>>>>>>>>>
>>>>>>>>>> I don't think it's "worth" it in this case to be honest -
>>>> especially
>>>>>> if
>>>>>>>>>> the maintainers are effectively "silent" and seem to ignore
>>>> problems
>>>>>> of
>>>>>>>>> one
>>>>>>>>>> of the really serious users they have. If they were willing to
>>>>>>>> cooperate,
>>>>>>>>>> we could do a lot, but when we are left alone, I think it will
>>> be
>>>>> far
>>>>>>>>> more
>>>>>>>>>> convenient for us to look for alternatives - to be honest.
>>>>>>>>>>
>>>>>>>>>> J.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Nov 29, 2023 at 2:46 AM Austin Bennett <
>>> austin@apache.org
>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> An option would be to fork Pendulum? It is MIT Licensed, I
>>> don't
>>>>>> know
>>>>>>>>>>> whether that poses problems to get in ASF?
>>>>>>>>>>>
>>>>>>>>>>> If forking (?) which is somewhat non-ideal, would we want that
>>>> 'in'
>>>>>>>>>>> airflow?
>>>>>>>>>>>
>>>>>>>>>>> If not 'in' airflow, I wonder if ASF incubator would accept a
>>>>> forked
>>>>>>>>>>> project? [ anecdotally Linux Foundation has incubated
>>> Starrocks
>>>>>> which
>>>>>>>>> is
>>>>>>>>>>> a
>>>>>>>>>>> fork of Apache Doris, so there's precedence for this to be OK
>>> in
>>>>> the
>>>>>>>>> wider
>>>>>>>>>>> OSS ecosystem ].
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Nov 28, 2023 at 6:41 PM Jarek Potiuk <jarek@potiuk.com
>>>>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I have not heard back from maintainers, just a comment from
>>>>> someone
>>>>>>>>> else
>>>>>>>>>>>> who suggested donating a pendulum to the ASF (which is kinda
>>>>>>>>> interesting
>>>>>>>>>>>> idea). I followed up, let's see. I think if we do not hear
>>> back
>>>>> and
>>>>>>>>>>> there
>>>>>>>>>>>> will be another week or two where "Pendulum 3 supporting 3.12
>>>>>>>> support
>>>>>>>>>>> "is
>>>>>>>>>>>> coming" but no-one knows when, effectively means that we
>>> should
>>>>> look
>>>>>>>>>>> for an
>>>>>>>>>>>> alternative like NOW.
>>>>>>>>>>>>
>>>>>>>>>>>> J.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Nov 22, 2023 at 3:27 AM Daniel Standish
>>>>>>>>>>>> <da...@astronomer.io.invalid> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks Jarek
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Nov 21, 2023, 4:34 PM Jarek Potiuk <jarek@potiuk.com
>>>>
>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think we miss important insight - straight from the
>>> source.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I believe it's time to be candid and simply ask questions
>>> for
>>>>>>>> the
>>>>>>>>>>>> future
>>>>>>>>>>>>> of
>>>>>>>>>>>>>> Pendulum directly where we should - ie. we should just ask
>>>>>>>>>>>> maintainers.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've just started a very candid and open discussion there -
>>>>>>>>>>>>>> https://github.com/sdispater/pendulum/discussions/771 .
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I linked back to this discussion and explained where we are
>>>>>>>> coming
>>>>>>>>>>>> from.
>>>>>>>>>>>>> I
>>>>>>>>>>>>>> even offered an option, that maybe - if they accept some
>>> help
>>>>>>>> and
>>>>>>>>>>> would
>>>>>>>>>>>>> be
>>>>>>>>>>>>>> open to make a few of us maintainers of Pendulum, that I
>>> will
>>>>>>>> ask
>>>>>>>>> if
>>>>>>>>>>>> some
>>>>>>>>>>>>>> of the maintainers here in Airflow would like to step up (I
>>>>>>>> would
>>>>>>>>> be
>>>>>>>>>>>>>> willing to - for one) and help to at least move Pendulum
>>>> through
>>>>>>>>>>> 3.12
>>>>>>>>>>>> and
>>>>>>>>>>>>>> maybe keep it running for as long as it will be needed to
>>>>>>>> maintain
>>>>>>>>>>> it.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think if they would be open to some of us helping them -
>>>> this
>>>>>>>>>>> might
>>>>>>>>>>>> be
>>>>>>>>>>>>>> actually the "cheapest" option for us to be honest at least
>>>> in a
>>>>>>>>>>>> mid-term
>>>>>>>>>>>>>> if we could gain influence and even take part and speed up
>>>>>>>>> releases
>>>>>>>>>>> and
>>>>>>>>>>>>>> merging of issues that are blocking us (or would be blocking
>>>> us
>>>>>>>> in
>>>>>>>>>>> the
>>>>>>>>>>>>>> future).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I tried to be very friendly but candid and direct on what
>>> kind
>>>>>>>> of
>>>>>>>>>>>>> problems
>>>>>>>>>>>>>> it creates us and also expressed our thanks for supporting
>>> us
>>>>>>>> for
>>>>>>>>> so
>>>>>>>>>>>> long
>>>>>>>>>>>>>> (I do believe they deserve it). I think the sole (pretty
>>>> much)
>>>>>>>>>>>>> maintainer
>>>>>>>>>>>>>> of Pendulum might really have a hard time maintaining and
>>>>>>>> keeping
>>>>>>>>>>> it up
>>>>>>>>>>>>> for
>>>>>>>>>>>>>> years - without being paid for and thanked. I hope - maybe
>>> all
>>>>>>>>> that
>>>>>>>>>>>> they
>>>>>>>>>>>>>> need are a few words of encouragement and thanks and
>>> realising
>>>>>>>>> that
>>>>>>>>>>>>> others
>>>>>>>>>>>>>> depend on their work - but also seeing that they are
>>>> understood,
>>>>>>>>>>>>> empathised
>>>>>>>>>>>>>> with and that there are viable alternatives they might
>>> follow
>>>>>>>>> might
>>>>>>>>>>> be
>>>>>>>>>>>>>> helpful for them to be also candid and respond. I hope my
>>>>>>>>> intentions
>>>>>>>>>>>> will
>>>>>>>>>>>>>> be understood - that I am not complaining, but even trying
>>> to
>>>>>>>>> help.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Let\s see where it gets us.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> J.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Nov 17, 2023 at 5:32 PM Andrey Anshin <
>>>>>>>>>>>> andrey.anshin@taragol.is>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Just to clarify I'd like us to consider the possibility
>>> that
>>>>>>>> no
>>>>>>>>>>> new
>>>>>>>>>>>>>>> pendulum would be released or released at the end of 2024,
>>>>>>>> like
>>>>>>>>> a
>>>>>>>>>>>>>>> pessimistic scenario:
>>>>>>>>>>>>>>> - What should we do in this case?
>>>>>>>>>>>>>>> - Work out a backup plan.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ----
>>>>>>>>>>>>>>> Best Wishes
>>>>>>>>>>>>>>> *Andrey Anshin*
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, 17 Nov 2023 at 16:33, Jarek Potiuk <
>>> jarek@potiuk.com
>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Also I think TP - had a document in the past (years ago)
>>>>>>>>>>>>> describing a
>>>>>>>>>>>>>>>> draft of a more complete alternative we can take to
>>> approach
>>>>>>>>>>>> datetime
>>>>>>>>>>>>>> vs.
>>>>>>>>>>>>>>>> pendulum dichotomy. I cannot easily find the document and
>>>>>>>>>>>> discussion
>>>>>>>>>>>>> -
>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>> I do remember it was proposing some interesting changes in
>>>>>>>> the
>>>>>>>>>>>>> approach
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> Airflow to have an abstraction layer on top of it (as far
>>>>>>>> as I
>>>>>>>>>>>>>> remember).
>>>>>>>>>>>>>>>> Maybe we can resurrect that idea if TP might find the
>>>>>>>>> proposal ?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Nov 17, 2023 at 1:06 PM Bolke de Bruin <
>>>>>>>>>>> bdbruin@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I agree that the current speed of development of Pendulum
>>>>>>>>>>> leaves
>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>> to be desired. However, I think we should not
>>>>>>>> underestimate
>>>>>>>>>>> the
>>>>>>>>>>>>>> effort
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> replacing it. It is not just a matter of
>>>>>>>>>>> %s/pendulum/datetime/g
>>>>>>>>>>>> so
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> say.
>>>>>>>>>>>>>>>>> If we are *truly* thinking about moving to native
>>>>>>>> datetime /
>>>>>>>>>>>>> zoneinfo
>>>>>>>>>>>>>>> etc
>>>>>>>>>>>>>>>>> we need *extensive* tests, basically copying what
>>> pendulum
>>>>>>>>>>> does
>>>>>>>>>>>> to
>>>>>>>>>>>>>>> check
>>>>>>>>>>>>>>>>> its behavior. The reason is that native implementations
>>> in
>>>>>>>>> the
>>>>>>>>>>>> past
>>>>>>>>>>>>>>> made
>>>>>>>>>>>>>>>>> serious mistakes and I do not put a lot of trust in them.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> An abstraction or vendoring in could be alternatives, but
>>>>>>>>> they
>>>>>>>>>>>>> bring
>>>>>>>>>>>>>>>> their
>>>>>>>>>>>>>>>>> own significant challenges.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I have re-raised the issue here:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> https://github.com/sdispater/pendulum/issues/753
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The upside is that it seems the amount of issues with the
>>>>>>>>>>> beta is
>>>>>>>>>>>>>>>> limited,
>>>>>>>>>>>>>>>>> so hopefully the maintainers can spend a couple of cycles
>>>>>>>> to
>>>>>>>>>>>>> address
>>>>>>>>>>>>>>>> them.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, 17 Nov 2023 at 10:28, Andrey Anshin <
>>>>>>>>>>>>>> andrey.anshin@taragol.is>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> There is no changes in stable pendulum so let's try to
>>>>>>>>>>> continue
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> discussion and start think about "Plan B"
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Just a reminder:
>>>>>>>>>>>>>>>>>> - pendulum 2.1.2 released 3 years ago (at the time
>>>>>>>> Airflow
>>>>>>>>>>>>> 1.10.x)
>>>>>>>>>>>>>>>>>> - pendulum 2 doesn't work well in Python 3.12, this is a
>>>>>>>>>>>>>> showstopper
>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>> the support Python 3.12
>>>>>>>>>>>>>>>>>> - pendulum 2 have memory leaks (
>>>>>>>>>>>>>>>>>> https://github.com/sdispater/pendulum/issues/720), and
>>>>>>>>>>> Airflow
>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>> approach
>>>>>>>>>>>>>>>>>> to achieve this leaks, especially in K8S executor
>>>>>>>>>>>>>>>>>> - pendulum 2 doesn't use system tzdata by default, but
>>>>>>>> we
>>>>>>>>>>> have
>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>> workaround, thanks Bolke for the documentation
>>>>>>>>>>>>>>>>>> - pendulum it is a core dependency of Airflow, and I
>>>>>>>> guess
>>>>>>>>>>> we
>>>>>>>>>>>>> can't
>>>>>>>>>>>>>>>>>> remove/replace it without breaking changes.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> So my proposal if things won't change in the near future
>>>>>>>>>>> then
>>>>>>>>>>>> we
>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> start removing the pendulum from the core and replace it
>>>>>>>>>>> with
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> native
>>>>>>>>>>>>>>>>>> datetime / zoneinfo / tzinfo. But maybe we have another
>>>>>>>>> way
>>>>>>>>>>> to
>>>>>>>>>>>>>>> resolve
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> without breaking changes? Because for me it would be a
>>>>>>>>>>> little
>>>>>>>>>>>>> weird
>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>> removal pendulum would be a main driver for release
>>>>>>>>> Airflow
>>>>>>>>>>> 3
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ----
>>>>>>>>>>>>>>>>>> Best Wishes
>>>>>>>>>>>>>>>>>> *Andrey Anshin*
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thu, 28 Sept 2023 at 13:01, Andrey Anshin <
>>>>>>>>>>>>>>> andrey.anshin@taragol.is
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> This discussion is more about the known problem of
>>>>>>>>>>> pendulum
>>>>>>>>>>>> and
>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>> could deal with it and maybe how we (as Community)
>>>>>>>> might
>>>>>>>>>>> help
>>>>>>>>>>>>>>> autor.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The library is mostly supported by a single author
>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>> Eustace
>>>>>>>>>>>>>>>> (
>>>>>>>>>>>>>>>>>>> https://github.com/sdispater) and it seems like we
>>>>>>>> bump
>>>>>>>>>>> into
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> situation which is described in xkcd #2347 (
>>>>>>>>>>>>>>>>>>> https://imgs.xkcd.com/comics/dependency.png). To be
>>>>>>>>>>> honest
>>>>>>>>>>>> it
>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>> something new when library mainly supported by one
>>>>>>>>> author
>>>>>>>>>>> so
>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> always a risk that the library will no longer be
>>>>>>>>>>> supported /
>>>>>>>>>>>>>>>> abandoned
>>>>>>>>>>>>>>>>>>> And if takes in account that pendulum provides core
>>>>>>>>>>>>> functionality
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>> Airflow it could have dramatical impact in the future.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Pendulum is a really nice library which helps a lot of
>>>>>>>>>>>>> developers
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>>>>>> with dates/datetimes. However there is one major
>>>>>>>>> problem,
>>>>>>>>>>> the
>>>>>>>>>>>>>> last
>>>>>>>>>>>>>>>>>> release
>>>>>>>>>>>>>>>>>>> of this library happened more than 3 years ago (
>>>>>>>>>>>>>>>>>>> https://pypi.org/project/pendulum/#history) in the
>>>>>>>> time
>>>>>>>>>>> when
>>>>>>>>>>>>>>> Airflow
>>>>>>>>>>>>>>>>>>> 1.10.11 was released
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Fortunately, the project is not abandoned and on a
>>>>>>>>> regular
>>>>>>>>>>>>> basis
>>>>>>>>>>>>>>>>> commits
>>>>>>>>>>>>>>>>>>> add into the master branch. However these commits are
>>>>>>>>> not
>>>>>>>>>>>>>> included
>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>> final release and that's why some things related to
>>>>>>>>>>> datetime
>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>> expected in Airflow. There are list of known (for me)
>>>>>>>>>>> issues
>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>> affect Airflow
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> *Memory Leak on parse*:
>>>>>>>>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/720,
>>>>>>>>> this
>>>>>>>>>>> one
>>>>>>>>>>>>>> fixed
>>>>>>>>>>>>>>>> 2
>>>>>>>>>>>>>>>>>>> years ago but not available yet (
>>>>>>>>>>>>>>>>>>> https://github.com/sdispater/pendulum/pull/563).
>>>>>>>> Since
>>>>>>>>> we
>>>>>>>>>>>> use
>>>>>>>>>>>>>>> parse
>>>>>>>>>>>>>>>>>> dates
>>>>>>>>>>>>>>>>>>> in airflow codebase: datetime parameters and datetime
>>>>>>>> in
>>>>>>>>>>> logs
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>>>>> could be a reason for memory leakage in Airflow:
>>>>>>>>>>>>>>>>>>> - https://github.com/apache/airflow/discussions/24694
>>>>>>>>>>>>>>>>>>> - https://github.com/apache/airflow/discussions/28597
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> *Incorrect time zones*, known issues and should be
>>>>>>>>> already
>>>>>>>>>>>>> fixed
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>> master branch
>>>>>>>>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/700,
>>>>>>>>>>> Mexico
>>>>>>>>>>>> do
>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>> DST
>>>>>>>>>>>>>>>>>>> anymore
>>>>>>>>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/706,
>>>>>>>>> Egypt
>>>>>>>>>>>>>>> reinstate
>>>>>>>>>>>>>>>>> DST
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> We add clarification in
>>>>>>>>>>>>>>> https://github.com/apache/airflow/pull/30467
>>>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>>>>>>> however it seems like there is no other way rather
>>>>>>>> than
>>>>>>>>>>>>> patching
>>>>>>>>>>>>>>>>> Pendulum
>>>>>>>>>>>>>>>>>>> right now.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> All these issues should be solved as soon as pendulum
>>>>>>>> 3
>>>>>>>>> is
>>>>>>>>>>>>>>> released.
>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>> current announced estimation is end of september/
>>>>>>>>>>> beginning
>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> October:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>> https://github.com/sdispater/pendulum/issues/600#issuecomment-1711299677
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> So in theory we would have a fixed version of pendulum
>>>>>>>>>>> soon,
>>>>>>>>>>>>> and
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> might
>>>>>>>>>>>>>>>>>>> break something in Airflow but from my point of view
>>>>>>>> it
>>>>>>>>> is
>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>> current status.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> However there might be a situation where the release
>>>>>>>> of
>>>>>>>>>>> the
>>>>>>>>>>>>>>> pendulum
>>>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>>> be postponed, so maybe better to have a backup plan.
>>>>>>>>> What
>>>>>>>>>>>> could
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>> this case?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Maybe we should start to use zoneinfo.ZoneInfo instead
>>>>>>>>> of
>>>>>>>>>>>>>> pendulum
>>>>>>>>>>>>>>>>>>> datetime?
>>>>>>>>> https://github.com/apache/airflow/issues/19450
>>>>>>>>>>>>>>>>>>> Pros:
>>>>>>>>>>>>>>>>>>> - stdlib (python 3.9+)
>>>>>>>>>>>>>>>>>>> - In pendulum 3.0 Timezone based on zoneinfo.Zoneinfo
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Cons:
>>>>>>>>>>>>>>>>>>> - Current serialization model can't deal with backport
>>>>>>>>>>>>> packages.
>>>>>>>>>>>>>>> E.g.
>>>>>>>>>>>>>>>>>>> timezone which are serialized in backport_zoneinfo
>>>>>>>> can't
>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> deserialized
>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>> zoneinfo
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Maybe we should replace parse datetime with another
>>>>>>>>>>> solution.
>>>>>>>>>>>>>> Does
>>>>>>>>>>>>>>>>> anyone
>>>>>>>>>>>>>>>>>>> know a good replacement?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Maybe someone from Airflow Community could propose
>>>>>>>> their
>>>>>>>>>>> help
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>> maintenance of library:
>>>>>>>>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/590
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Maybe we should get rid of the pendulum at all, as a
>>>>>>>>> last
>>>>>>>>>>>>> resort
>>>>>>>>>>>>>>>>>> solution.
>>>>>>>>>>>>>>>>>>> I can't imagine how we could do that, because a lot of
>>>>>>>>>>> stuff
>>>>>>>>>>>>>>> depends
>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> pendulum and removing it would be a breaking change.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ----
>>>>>>>>>>>>>>>>>>> Best Wishes
>>>>>>>>>>>>>>>>>>> *Andrey Anshin*
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Bolke de Bruin
>>>>>>>>>>>>>>>>> bdbruin@gmail.com
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> --
>>> Bolke de Bruin
>>> bdbruin@gmail.com
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
Re: [DISCUSS] Future of Pendulum in Airflow
Posted by Jarek Potiuk <ja...@potiuk.com>.
I am for a) with deprecation warning when Pendulum 2 is installed in
Airflow 2.8.1+.
I understand we have a long way before we can get rid of Pendulum
dependency in general as it will be a major pain for our users for a
number of reasons, so I think getting rid of Pendulum is likely not
easy / feasible in the short/medium term (though it would likely be
best).
Pendulum 3 solves the biggest pain (one of the things that blocks us
from officially having Python 3.12 support) and when we consider the
scope of the features we use from Pendulum, it's not very likely it
will cause us similar pains even if we keep it for quite some time. So
supporting Pendulum 3 while giving our users (and 3rd-party libraries)
a chance to upgrade to Pendulum 3 seems a good balance between cost
and risk - at least for a year or so until Python 3.13 will be out
(with possibly some more breaking changes) or until some security
issues will be discovered (and not fixed for some time) in Pendulum.
J,
On Tue, Dec 19, 2023 at 2:43 PM Andrey Anshin <an...@taragol.is> wrote:
>
> Greetings everyone!
>
> Because pendulum 3 was released we probably need to discuss about support
> for the previous version of pendulum in Airflow core.
>
> There is not much options here
>
> Option 1: Drop support of next patch version of Airflow, e.g. 2.8.1. It
> might (or might not) break someone's pipelines. In general we do not count
> change upstream library version as breaking changes, however this could be
> an exclusion, because `pendulum 2` first time released in May 9, 2018 and
> there is potentially could be a lot of user code which do not work properly
> with `pendulum 3`
>
>
> Option 2: Keep support of both pendulum 2 and pendulum 3 for a while. There
> are different suboptions
> a). Support pendulum 2 until Airflow 2.9
> b). Support pendulum 2 until the min Airflow version in providers reaches
> Airflow 2.9, a bit more than 1 year. In Airflow 2.9 add user warning about
> limited time of support
> c). Support pendulum 2 until it is not painful
> d). Support pendulum 2 until Airflow 3 (Forever?)
>
>
> On Mon, 18 Dec 2023 at 16:53, Bolke de Bruin <bd...@gmail.com> wrote:
>
> > Pendulum 3 was released.
> >
> > https://pypi.org/project/pendulum/
> >
> > Bolke
> >
> > On Fri, 1 Dec 2023 at 18:38, Aritra Basu <ar...@gmail.com> wrote:
> >
> > > I agree with wanting to stay away from having to deal with dst but I am
> > on
> > > the side of biting the bullet and transitioning to datetime and testing
> > > thoroughly, especially considering our current predicament with pendulum
> > > and the potential of us being in the same place again potentially in the
> > > future.
> > >
> > > --
> > > Regards,
> > > Aritra Basu
> > >
> > > On Fri, Dec 1, 2023, 10:27 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > > Yeah. Agree it's a challenge but I think it's also an opportunity to
> > > > fix those - because whether we admit it, or not - we already have those
> > > > problems - regardless of Pendulum's use.
> > > >
> > > > If you look at the issues (especially ones created around DST time) -
> > > there
> > > > are always one or two issues (even in latest versions of Airflow) where
> > > DST
> > > > transition is causing people problems. And they are rarely
> > > > solved/addressed.
> > > >
> > > > Example from last October:
> > > >
> > > > https://github.com/apache/airflow/issues/35272
> > > >
> > > > Example from May 2022:
> > > >
> > > > https://github.com/apache/airflow/issues/23400
> > > >
> > > > And Example from March 2020 ( which might or might not be fixed by the
> > > open
> > > > PR):
> > > >
> > > > https://github.com/apache/airflow/issues/7999
> > > >
> > > > J.
> > > >
> > > >
> > > > On Fri, Dec 1, 2023 at 5:41 PM Bolke de Bruin <bd...@gmail.com>
> > wrote:
> > > >
> > > > > The alternative is to vendor in or fork pendulum as part of the
> > Airflow
> > > > > project. Also a major undertaking but maybe less effort.
> > > > >
> > > > > We shouldn't just address this as a technical challenge. The question
> > > > > becomes: does that combination fornally do DST transitions correctly,
> > > > does
> > > > > it do leap seconds correctly, does it add dates correctly, does it do
> > > > > timezone traversal correctly and many many other things.
> > > > >
> > > > > So if we switch to the datetime + zone info + timezone we need to
> > have
> > > a
> > > > > test suite that verifies this. Otherwise we potentially expose our
> > > users
> > > > to
> > > > > different behavior IN their data. Not just a schedule by in their SQL
> > > or
> > > > in
> > > > > some scripting where logical date / execution date is used or where
> > > date
> > > > > arithmetic is applied.
> > > > >
> > > > > So it is not changing one car for another. It is changing the means
> > of
> > > > > transportation and not knowing if you can get to all the places you
> > > want
> > > > /
> > > > > need to go as before.
> > > > >
> > > > > Honestly, I'm a bit scared (I do see the lack of responsiveness). I
> > > hope
> > > > > someone can take that away.
> > > > >
> > > > > Bolke
> > > > >
> > > > >
> > > > > Sent from my iPhone
> > > > >
> > > > > > On 1 Dec 2023, at 13:39, Jarek Potiuk <ja...@potiuk.com> wrote:
> > > > > >
> > > > > > Agree. I think this is the best way forward.- we have to bite the
> > > > > bullet of
> > > > > > potential backwards compatibility issues and separating out a
> > compat
> > > > > layer
> > > > > > where we could optionally use pendulum instead would be the most
> > > "easy"
> > > > > > approach for our users. Most of them would not even notice if we
> > do a
> > > > > good
> > > > > > job, but those who somehow depend on pendulum objects would get a
> > > > > > possibility to switch back.
> > > > > >
> > > > > >> On Fri, Dec 1, 2023 at 1:09 PM Andrey Anshin <
> > > > andrey.anshin@taragol.is>
> > > > > >> wrote:
> > > > > >>
> > > > > >> IMHO, if we would like to replace pendulum to alternatives, the
> > > better
> > > > > >> alternative would be combination of datetime.datetime + zoneinfo +
> > > > > >> datetime.timezone
> > > > > >>
> > > > > >> We still do hacks around pendulum functional, which makes hacks
> > > around
> > > > > >> datetime.datetime, so in my perspective better remove redundant
> > > layers
> > > > > and
> > > > > >> use native objects directly.
> > > > > >>
> > > > > >> What confuses me the most is how we get around type changes in
> > Tasks
> > > > > >> Context. All dates return as pendulum.DateTime for a very long
> > time
> > > > > >> (literally for ages) and it is probably could be classified as
> > > > breaking
> > > > > >> changes, because it might break quite a bit users pipelines. It
> > > still
> > > > > could
> > > > > >> be resolved as optional pendulum dependency + config option for
> > > > datetime
> > > > > >> types in context, which might help to get familiar types for a
> > > while.
> > > > > >>
> > > > > >>
> > > > > >>> On Wed, 29 Nov 2023 at 05:58, Jarek Potiuk <ja...@potiuk.com>
> > > wrote:
> > > > > >>>
> > > > > >>> "infamous fork of " should be "fork of infamous Akka" - just to
> > be
> > > > > clear
> > > > > >> :)
> > > > > >>>
> > > > > >>>> On Wed, Nov 29, 2023 at 2:57 AM Jarek Potiuk <ja...@potiuk.com>
> > > > > wrote:
> > > > > >>>
> > > > > >>>> It's a very rare occurrence that ASF accepts forks. Usually it
> > has
> > > > to
> > > > > >> be
> > > > > >>>> willingly donated by those who own the IP rights, - mainly
> > because
> > > > of
> > > > > >>>> trademark issues. Licence is one thing, but carrying things like
> > > the
> > > > > >> name
> > > > > >>>> of the project is not possible without clearing IP and donating
> > > the
> > > > > >>>> project.
> > > > > >>>>
> > > > > >>>> It happened last time last year - with Pekko project - infamous
> > > fork
> > > > > or
> > > > > >>>> Akka after Lightbend changed licensing for Akka -
> > > > > >>>>
> > > > > >>
> > > >
> > https://www.lightbend.com/blog/why-we-are-changing-the-license-for-akka
> > > > > >>>> making it impossible to be used in ASF projects with the new
> > > > licence.
> > > > > >> But
> > > > > >>>> it was a really "strong" case and the "we do not accept forks
> > > > usually"
> > > > > >>> was
> > > > > >>>> quite a contentious issue (I followed the discussion - it was
> > > > > >> fascinating
> > > > > >>>> actually). It tool a LOT of time and effort for those who led
> > the
> > > > > >> effort
> > > > > >>>> eventually - and also mostly because things had to be
> > > crystal-clear
> > > > > and
> > > > > >>> not
> > > > > >>>> "litigable" because the other side had some business reasons to
> > > make
> > > > > >> the
> > > > > >>>> licensing changes and they did not understood why it is
> > impossible
> > > > to
> > > > > >> use
> > > > > >>>> Akka as-is even with all the exclusions they offered to ASF -
> > and
> > > > > >>>> effectively it was a "hostile" move.
> > > > > >>>>
> > > > > >>>> I don't think it's "worth" it in this case to be honest -
> > > especially
> > > > > if
> > > > > >>>> the maintainers are effectively "silent" and seem to ignore
> > > problems
> > > > > of
> > > > > >>> one
> > > > > >>>> of the really serious users they have. If they were willing to
> > > > > >> cooperate,
> > > > > >>>> we could do a lot, but when we are left alone, I think it will
> > be
> > > > far
> > > > > >>> more
> > > > > >>>> convenient for us to look for alternatives - to be honest.
> > > > > >>>>
> > > > > >>>> J.
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> On Wed, Nov 29, 2023 at 2:46 AM Austin Bennett <
> > austin@apache.org
> > > >
> > > > > >>> wrote:
> > > > > >>>>
> > > > > >>>>> An option would be to fork Pendulum? It is MIT Licensed, I
> > don't
> > > > > know
> > > > > >>>>> whether that poses problems to get in ASF?
> > > > > >>>>>
> > > > > >>>>> If forking (?) which is somewhat non-ideal, would we want that
> > > 'in'
> > > > > >>>>> airflow?
> > > > > >>>>>
> > > > > >>>>> If not 'in' airflow, I wonder if ASF incubator would accept a
> > > > forked
> > > > > >>>>> project? [ anecdotally Linux Foundation has incubated
> > Starrocks
> > > > > which
> > > > > >>> is
> > > > > >>>>> a
> > > > > >>>>> fork of Apache Doris, so there's precedence for this to be OK
> > in
> > > > the
> > > > > >>> wider
> > > > > >>>>> OSS ecosystem ].
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> On Tue, Nov 28, 2023 at 6:41 PM Jarek Potiuk <jarek@potiuk.com
> > >
> > > > > >> wrote:
> > > > > >>>>>
> > > > > >>>>>> I have not heard back from maintainers, just a comment from
> > > > someone
> > > > > >>> else
> > > > > >>>>>> who suggested donating a pendulum to the ASF (which is kinda
> > > > > >>> interesting
> > > > > >>>>>> idea). I followed up, let's see. I think if we do not hear
> > back
> > > > and
> > > > > >>>>> there
> > > > > >>>>>> will be another week or two where "Pendulum 3 supporting 3.12
> > > > > >> support
> > > > > >>>>> "is
> > > > > >>>>>> coming" but no-one knows when, effectively means that we
> > should
> > > > look
> > > > > >>>>> for an
> > > > > >>>>>> alternative like NOW.
> > > > > >>>>>>
> > > > > >>>>>> J.
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> On Wed, Nov 22, 2023 at 3:27 AM Daniel Standish
> > > > > >>>>>> <da...@astronomer.io.invalid> wrote:
> > > > > >>>>>>
> > > > > >>>>>>> Thanks Jarek
> > > > > >>>>>>>
> > > > > >>>>>>> On Tue, Nov 21, 2023, 4:34 PM Jarek Potiuk <jarek@potiuk.com
> > >
> > > > > >>> wrote:
> > > > > >>>>>>>
> > > > > >>>>>>>> I think we miss important insight - straight from the
> > source.
> > > > > >>>>>>>>
> > > > > >>>>>>>> I believe it's time to be candid and simply ask questions
> > for
> > > > > >> the
> > > > > >>>>>> future
> > > > > >>>>>>> of
> > > > > >>>>>>>> Pendulum directly where we should - ie. we should just ask
> > > > > >>>>>> maintainers.
> > > > > >>>>>>>>
> > > > > >>>>>>>> I've just started a very candid and open discussion there -
> > > > > >>>>>>>> https://github.com/sdispater/pendulum/discussions/771 .
> > > > > >>>>>>>>
> > > > > >>>>>>>> I linked back to this discussion and explained where we are
> > > > > >> coming
> > > > > >>>>>> from.
> > > > > >>>>>>> I
> > > > > >>>>>>>> even offered an option, that maybe - if they accept some
> > help
> > > > > >> and
> > > > > >>>>> would
> > > > > >>>>>>> be
> > > > > >>>>>>>> open to make a few of us maintainers of Pendulum, that I
> > will
> > > > > >> ask
> > > > > >>> if
> > > > > >>>>>> some
> > > > > >>>>>>>> of the maintainers here in Airflow would like to step up (I
> > > > > >> would
> > > > > >>> be
> > > > > >>>>>>>> willing to - for one) and help to at least move Pendulum
> > > through
> > > > > >>>>> 3.12
> > > > > >>>>>> and
> > > > > >>>>>>>> maybe keep it running for as long as it will be needed to
> > > > > >> maintain
> > > > > >>>>> it.
> > > > > >>>>>>>>
> > > > > >>>>>>>> I think if they would be open to some of us helping them -
> > > this
> > > > > >>>>> might
> > > > > >>>>>> be
> > > > > >>>>>>>> actually the "cheapest" option for us to be honest at least
> > > in a
> > > > > >>>>>> mid-term
> > > > > >>>>>>>> if we could gain influence and even take part and speed up
> > > > > >>> releases
> > > > > >>>>> and
> > > > > >>>>>>>> merging of issues that are blocking us (or would be blocking
> > > us
> > > > > >> in
> > > > > >>>>> the
> > > > > >>>>>>>> future).
> > > > > >>>>>>>>
> > > > > >>>>>>>> I tried to be very friendly but candid and direct on what
> > kind
> > > > > >> of
> > > > > >>>>>>> problems
> > > > > >>>>>>>> it creates us and also expressed our thanks for supporting
> > us
> > > > > >> for
> > > > > >>> so
> > > > > >>>>>> long
> > > > > >>>>>>>> (I do believe they deserve it). I think the sole (pretty
> > > much)
> > > > > >>>>>>> maintainer
> > > > > >>>>>>>> of Pendulum might really have a hard time maintaining and
> > > > > >> keeping
> > > > > >>>>> it up
> > > > > >>>>>>> for
> > > > > >>>>>>>> years - without being paid for and thanked. I hope - maybe
> > all
> > > > > >>> that
> > > > > >>>>>> they
> > > > > >>>>>>>> need are a few words of encouragement and thanks and
> > realising
> > > > > >>> that
> > > > > >>>>>>> others
> > > > > >>>>>>>> depend on their work - but also seeing that they are
> > > understood,
> > > > > >>>>>>> empathised
> > > > > >>>>>>>> with and that there are viable alternatives they might
> > follow
> > > > > >>> might
> > > > > >>>>> be
> > > > > >>>>>>>> helpful for them to be also candid and respond. I hope my
> > > > > >>> intentions
> > > > > >>>>>> will
> > > > > >>>>>>>> be understood - that I am not complaining, but even trying
> > to
> > > > > >>> help.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Let\s see where it gets us.
> > > > > >>>>>>>>
> > > > > >>>>>>>> J.
> > > > > >>>>>>>>
> > > > > >>>>>>>> On Fri, Nov 17, 2023 at 5:32 PM Andrey Anshin <
> > > > > >>>>>> andrey.anshin@taragol.is>
> > > > > >>>>>>>> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>>> Just to clarify I'd like us to consider the possibility
> > that
> > > > > >> no
> > > > > >>>>> new
> > > > > >>>>>>>>> pendulum would be released or released at the end of 2024,
> > > > > >> like
> > > > > >>> a
> > > > > >>>>>>>>> pessimistic scenario:
> > > > > >>>>>>>>> - What should we do in this case?
> > > > > >>>>>>>>> - Work out a backup plan.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> ----
> > > > > >>>>>>>>> Best Wishes
> > > > > >>>>>>>>> *Andrey Anshin*
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> On Fri, 17 Nov 2023 at 16:33, Jarek Potiuk <
> > jarek@potiuk.com
> > > >
> > > > > >>>>> wrote:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> Also I think TP - had a document in the past (years ago)
> > > > > >>>>>>> describing a
> > > > > >>>>>>>>>> draft of a more complete alternative we can take to
> > approach
> > > > > >>>>>> datetime
> > > > > >>>>>>>> vs.
> > > > > >>>>>>>>>> pendulum dichotomy. I cannot easily find the document and
> > > > > >>>>>> discussion
> > > > > >>>>>>> -
> > > > > >>>>>>>>> but
> > > > > >>>>>>>>>> I do remember it was proposing some interesting changes in
> > > > > >> the
> > > > > >>>>>>> approach
> > > > > >>>>>>>>> of
> > > > > >>>>>>>>>> Airflow to have an abstraction layer on top of it (as far
> > > > > >> as I
> > > > > >>>>>>>> remember).
> > > > > >>>>>>>>>> Maybe we can resurrect that idea if TP might find the
> > > > > >>> proposal ?
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> On Fri, Nov 17, 2023 at 1:06 PM Bolke de Bruin <
> > > > > >>>>> bdbruin@gmail.com>
> > > > > >>>>>>>>> wrote:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>> Hi,
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> I agree that the current speed of development of Pendulum
> > > > > >>>>> leaves
> > > > > >>>>>>>>>> something
> > > > > >>>>>>>>>>> to be desired. However, I think we should not
> > > > > >> underestimate
> > > > > >>>>> the
> > > > > >>>>>>>> effort
> > > > > >>>>>>>>> of
> > > > > >>>>>>>>>>> replacing it. It is not just a matter of
> > > > > >>>>> %s/pendulum/datetime/g
> > > > > >>>>>> so
> > > > > >>>>>>> to
> > > > > >>>>>>>>>> say.
> > > > > >>>>>>>>>>> If we are *truly* thinking about moving to native
> > > > > >> datetime /
> > > > > >>>>>>> zoneinfo
> > > > > >>>>>>>>> etc
> > > > > >>>>>>>>>>> we need *extensive* tests, basically copying what
> > pendulum
> > > > > >>>>> does
> > > > > >>>>>> to
> > > > > >>>>>>>>> check
> > > > > >>>>>>>>>>> its behavior. The reason is that native implementations
> > in
> > > > > >>> the
> > > > > >>>>>> past
> > > > > >>>>>>>>> made
> > > > > >>>>>>>>>>> serious mistakes and I do not put a lot of trust in them.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> An abstraction or vendoring in could be alternatives, but
> > > > > >>> they
> > > > > >>>>>>> bring
> > > > > >>>>>>>>>> their
> > > > > >>>>>>>>>>> own significant challenges.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> I have re-raised the issue here:
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> https://github.com/sdispater/pendulum/issues/753
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> The upside is that it seems the amount of issues with the
> > > > > >>>>> beta is
> > > > > >>>>>>>>>> limited,
> > > > > >>>>>>>>>>> so hopefully the maintainers can spend a couple of cycles
> > > > > >> to
> > > > > >>>>>>> address
> > > > > >>>>>>>>>> them.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Bolke
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> On Fri, 17 Nov 2023 at 10:28, Andrey Anshin <
> > > > > >>>>>>>> andrey.anshin@taragol.is>
> > > > > >>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>> There is no changes in stable pendulum so let's try to
> > > > > >>>>> continue
> > > > > >>>>>>>> this
> > > > > >>>>>>>>>>>> discussion and start think about "Plan B"
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Just a reminder:
> > > > > >>>>>>>>>>>> - pendulum 2.1.2 released 3 years ago (at the time
> > > > > >> Airflow
> > > > > >>>>>>> 1.10.x)
> > > > > >>>>>>>>>>>> - pendulum 2 doesn't work well in Python 3.12, this is a
> > > > > >>>>>>>> showstopper
> > > > > >>>>>>>>>> for
> > > > > >>>>>>>>>>>> the support Python 3.12
> > > > > >>>>>>>>>>>> - pendulum 2 have memory leaks (
> > > > > >>>>>>>>>>>> https://github.com/sdispater/pendulum/issues/720), and
> > > > > >>>>> Airflow
> > > > > >>>>>>> use
> > > > > >>>>>>>>>>>> approach
> > > > > >>>>>>>>>>>> to achieve this leaks, especially in K8S executor
> > > > > >>>>>>>>>>>> - pendulum 2 doesn't use system tzdata by default, but
> > > > > >> we
> > > > > >>>>> have
> > > > > >>>>>> a
> > > > > >>>>>>>>>>>> workaround, thanks Bolke for the documentation
> > > > > >>>>>>>>>>>> - pendulum it is a core dependency of Airflow, and I
> > > > > >> guess
> > > > > >>>>> we
> > > > > >>>>>>> can't
> > > > > >>>>>>>>>>>> remove/replace it without breaking changes.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> So my proposal if things won't change in the near future
> > > > > >>>>> then
> > > > > >>>>>> we
> > > > > >>>>>>>> need
> > > > > >>>>>>>>>> to
> > > > > >>>>>>>>>>>> start removing the pendulum from the core and replace it
> > > > > >>>>> with
> > > > > >>>>>> the
> > > > > >>>>>>>>>> native
> > > > > >>>>>>>>>>>> datetime / zoneinfo / tzinfo. But maybe we have another
> > > > > >>> way
> > > > > >>>>> to
> > > > > >>>>>>>>> resolve
> > > > > >>>>>>>>>>> this
> > > > > >>>>>>>>>>>> without breaking changes? Because for me it would be a
> > > > > >>>>> little
> > > > > >>>>>>> weird
> > > > > >>>>>>>>> if
> > > > > >>>>>>>>>>>> removal pendulum would be a main driver for release
> > > > > >>> Airflow
> > > > > >>>>> 3
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> ----
> > > > > >>>>>>>>>>>> Best Wishes
> > > > > >>>>>>>>>>>> *Andrey Anshin*
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> On Thu, 28 Sept 2023 at 13:01, Andrey Anshin <
> > > > > >>>>>>>>> andrey.anshin@taragol.is
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> This discussion is more about the known problem of
> > > > > >>>>> pendulum
> > > > > >>>>>> and
> > > > > >>>>>>>> how
> > > > > >>>>>>>>>> we
> > > > > >>>>>>>>>>>>> could deal with it and maybe how we (as Community)
> > > > > >> might
> > > > > >>>>> help
> > > > > >>>>>>>>> autor.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> The library is mostly supported by a single author
> > > > > >>>>> Sébastien
> > > > > >>>>>>>>> Eustace
> > > > > >>>>>>>>>> (
> > > > > >>>>>>>>>>>>> https://github.com/sdispater) and it seems like we
> > > > > >> bump
> > > > > >>>>> into
> > > > > >>>>>>> the
> > > > > >>>>>>>>>>>>> situation which is described in xkcd #2347 (
> > > > > >>>>>>>>>>>>> https://imgs.xkcd.com/comics/dependency.png). To be
> > > > > >>>>> honest
> > > > > >>>>>> it
> > > > > >>>>>>> is
> > > > > >>>>>>>>> not
> > > > > >>>>>>>>>>>>> something new when library mainly supported by one
> > > > > >>> author
> > > > > >>>>> so
> > > > > >>>>>>>> there
> > > > > >>>>>>>>> is
> > > > > >>>>>>>>>>>>> always a risk that the library will no longer be
> > > > > >>>>> supported /
> > > > > >>>>>>>>>> abandoned
> > > > > >>>>>>>>>>>>> And if takes in account that pendulum provides core
> > > > > >>>>>>> functionality
> > > > > >>>>>>>>> in
> > > > > >>>>>>>>>>>>> Airflow it could have dramatical impact in the future.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Pendulum is a really nice library which helps a lot of
> > > > > >>>>>>> developers
> > > > > >>>>>>>>> to
> > > > > >>>>>>>>>>> work
> > > > > >>>>>>>>>>>>> with dates/datetimes. However there is one major
> > > > > >>> problem,
> > > > > >>>>> the
> > > > > >>>>>>>> last
> > > > > >>>>>>>>>>>> release
> > > > > >>>>>>>>>>>>> of this library happened more than 3 years ago (
> > > > > >>>>>>>>>>>>> https://pypi.org/project/pendulum/#history) in the
> > > > > >> time
> > > > > >>>>> when
> > > > > >>>>>>>>> Airflow
> > > > > >>>>>>>>>>>>> 1.10.11 was released
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Fortunately, the project is not abandoned and on a
> > > > > >>> regular
> > > > > >>>>>>> basis
> > > > > >>>>>>>>>>> commits
> > > > > >>>>>>>>>>>>> add into the master branch. However these commits are
> > > > > >>> not
> > > > > >>>>>>>> included
> > > > > >>>>>>>>>> into
> > > > > >>>>>>>>>>>> any
> > > > > >>>>>>>>>>>>> final release and that's why some things related to
> > > > > >>>>> datetime
> > > > > >>>>>>>> don't
> > > > > >>>>>>>>>> work
> > > > > >>>>>>>>>>>> as
> > > > > >>>>>>>>>>>>> expected in Airflow. There are list of known (for me)
> > > > > >>>>> issues
> > > > > >>>>>>>> which
> > > > > >>>>>>>>>> are
> > > > > >>>>>>>>>>>>> affect Airflow
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> *Memory Leak on parse*:
> > > > > >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/720,
> > > > > >>> this
> > > > > >>>>> one
> > > > > >>>>>>>> fixed
> > > > > >>>>>>>>>> 2
> > > > > >>>>>>>>>>>>> years ago but not available yet (
> > > > > >>>>>>>>>>>>> https://github.com/sdispater/pendulum/pull/563).
> > > > > >> Since
> > > > > >>> we
> > > > > >>>>>> use
> > > > > >>>>>>>>> parse
> > > > > >>>>>>>>>>>> dates
> > > > > >>>>>>>>>>>>> in airflow codebase: datetime parameters and datetime
> > > > > >> in
> > > > > >>>>> logs
> > > > > >>>>>>>> this
> > > > > >>>>>>>>>> one
> > > > > >>>>>>>>>>>>> could be a reason for memory leakage in Airflow:
> > > > > >>>>>>>>>>>>> - https://github.com/apache/airflow/discussions/24694
> > > > > >>>>>>>>>>>>> - https://github.com/apache/airflow/discussions/28597
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> *Incorrect time zones*, known issues and should be
> > > > > >>> already
> > > > > >>>>>>> fixed
> > > > > >>>>>>>> in
> > > > > >>>>>>>>>>>>> master branch
> > > > > >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/700,
> > > > > >>>>> Mexico
> > > > > >>>>>> do
> > > > > >>>>>>>> not
> > > > > >>>>>>>>>> use
> > > > > >>>>>>>>>>>> DST
> > > > > >>>>>>>>>>>>> anymore
> > > > > >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/706,
> > > > > >>> Egypt
> > > > > >>>>>>>>> reinstate
> > > > > >>>>>>>>>>> DST
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> We add clarification in
> > > > > >>>>>>>>> https://github.com/apache/airflow/pull/30467
> > > > > >>>>>>>>>> ,
> > > > > >>>>>>>>>>>>> however it seems like there is no other way rather
> > > > > >> than
> > > > > >>>>>>> patching
> > > > > >>>>>>>>>>> Pendulum
> > > > > >>>>>>>>>>>>> right now.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> All these issues should be solved as soon as pendulum
> > > > > >> 3
> > > > > >>> is
> > > > > >>>>>>>>> released.
> > > > > >>>>>>>>>>> The
> > > > > >>>>>>>>>>>>> current announced estimation is end of september/
> > > > > >>>>> beginning
> > > > > >>>>>> of
> > > > > >>>>>>>>>> October:
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>
> > > > > >>>
> > > > >
> > > https://github.com/sdispater/pendulum/issues/600#issuecomment-1711299677
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> So in theory we would have a fixed version of pendulum
> > > > > >>>>> soon,
> > > > > >>>>>>> and
> > > > > >>>>>>>> it
> > > > > >>>>>>>>>>> might
> > > > > >>>>>>>>>>>>> break something in Airflow but from my point of view
> > > > > >> it
> > > > > >>> is
> > > > > >>>>>>> better
> > > > > >>>>>>>>>> than
> > > > > >>>>>>>>>>>>> current status.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> However there might be a situation where the release
> > > > > >> of
> > > > > >>>>> the
> > > > > >>>>>>>>> pendulum
> > > > > >>>>>>>>>>>> would
> > > > > >>>>>>>>>>>>> be postponed, so maybe better to have a backup plan.
> > > > > >>> What
> > > > > >>>>>> could
> > > > > >>>>>>>> we
> > > > > >>>>>>>>> do
> > > > > >>>>>>>>>>> in
> > > > > >>>>>>>>>>>>> this case?
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Maybe we should start to use zoneinfo.ZoneInfo instead
> > > > > >>> of
> > > > > >>>>>>>> pendulum
> > > > > >>>>>>>>>>>>> datetime?
> > > > > >>> https://github.com/apache/airflow/issues/19450
> > > > > >>>>>>>>>>>>> Pros:
> > > > > >>>>>>>>>>>>> - stdlib (python 3.9+)
> > > > > >>>>>>>>>>>>> - In pendulum 3.0 Timezone based on zoneinfo.Zoneinfo
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Cons:
> > > > > >>>>>>>>>>>>> - Current serialization model can't deal with backport
> > > > > >>>>>>> packages.
> > > > > >>>>>>>>> E.g.
> > > > > >>>>>>>>>>>>> timezone which are serialized in backport_zoneinfo
> > > > > >> can't
> > > > > >>>>> be
> > > > > >>>>>>>>>>> deserialized
> > > > > >>>>>>>>>>>> in
> > > > > >>>>>>>>>>>>> zoneinfo
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Maybe we should replace parse datetime with another
> > > > > >>>>> solution.
> > > > > >>>>>>>> Does
> > > > > >>>>>>>>>>> anyone
> > > > > >>>>>>>>>>>>> know a good replacement?
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Maybe someone from Airflow Community could propose
> > > > > >> their
> > > > > >>>>> help
> > > > > >>>>>>>> with
> > > > > >>>>>>>>>>>>> maintenance of library:
> > > > > >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/590
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Maybe we should get rid of the pendulum at all, as a
> > > > > >>> last
> > > > > >>>>>>> resort
> > > > > >>>>>>>>>>>> solution.
> > > > > >>>>>>>>>>>>> I can't imagine how we could do that, because a lot of
> > > > > >>>>> stuff
> > > > > >>>>>>>>> depends
> > > > > >>>>>>>>>> on
> > > > > >>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>> pendulum and removing it would be a breaking change.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> ----
> > > > > >>>>>>>>>>>>> Best Wishes
> > > > > >>>>>>>>>>>>> *Andrey Anshin*
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> --
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> --
> > > > > >>>>>>>>>>> Bolke de Bruin
> > > > > >>>>>>>>>>> bdbruin@gmail.com
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>
> > > > >
> > > > > ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> >
> > --
> > Bolke de Bruin
> > bdbruin@gmail.com
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] Future of Pendulum in Airflow
Posted by Andrey Anshin <an...@taragol.is>.
Greetings everyone!
Because pendulum 3 was released we probably need to discuss about support
for the previous version of pendulum in Airflow core.
There is not much options here
Option 1: Drop support of next patch version of Airflow, e.g. 2.8.1. It
might (or might not) break someone's pipelines. In general we do not count
change upstream library version as breaking changes, however this could be
an exclusion, because `pendulum 2` first time released in May 9, 2018 and
there is potentially could be a lot of user code which do not work properly
with `pendulum 3`
Option 2: Keep support of both pendulum 2 and pendulum 3 for a while. There
are different suboptions
a). Support pendulum 2 until Airflow 2.9
b). Support pendulum 2 until the min Airflow version in providers reaches
Airflow 2.9, a bit more than 1 year. In Airflow 2.9 add user warning about
limited time of support
c). Support pendulum 2 until it is not painful
d). Support pendulum 2 until Airflow 3 (Forever?)
On Mon, 18 Dec 2023 at 16:53, Bolke de Bruin <bd...@gmail.com> wrote:
> Pendulum 3 was released.
>
> https://pypi.org/project/pendulum/
>
> Bolke
>
> On Fri, 1 Dec 2023 at 18:38, Aritra Basu <ar...@gmail.com> wrote:
>
> > I agree with wanting to stay away from having to deal with dst but I am
> on
> > the side of biting the bullet and transitioning to datetime and testing
> > thoroughly, especially considering our current predicament with pendulum
> > and the potential of us being in the same place again potentially in the
> > future.
> >
> > --
> > Regards,
> > Aritra Basu
> >
> > On Fri, Dec 1, 2023, 10:27 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > Yeah. Agree it's a challenge but I think it's also an opportunity to
> > > fix those - because whether we admit it, or not - we already have those
> > > problems - regardless of Pendulum's use.
> > >
> > > If you look at the issues (especially ones created around DST time) -
> > there
> > > are always one or two issues (even in latest versions of Airflow) where
> > DST
> > > transition is causing people problems. And they are rarely
> > > solved/addressed.
> > >
> > > Example from last October:
> > >
> > > https://github.com/apache/airflow/issues/35272
> > >
> > > Example from May 2022:
> > >
> > > https://github.com/apache/airflow/issues/23400
> > >
> > > And Example from March 2020 ( which might or might not be fixed by the
> > open
> > > PR):
> > >
> > > https://github.com/apache/airflow/issues/7999
> > >
> > > J.
> > >
> > >
> > > On Fri, Dec 1, 2023 at 5:41 PM Bolke de Bruin <bd...@gmail.com>
> wrote:
> > >
> > > > The alternative is to vendor in or fork pendulum as part of the
> Airflow
> > > > project. Also a major undertaking but maybe less effort.
> > > >
> > > > We shouldn't just address this as a technical challenge. The question
> > > > becomes: does that combination fornally do DST transitions correctly,
> > > does
> > > > it do leap seconds correctly, does it add dates correctly, does it do
> > > > timezone traversal correctly and many many other things.
> > > >
> > > > So if we switch to the datetime + zone info + timezone we need to
> have
> > a
> > > > test suite that verifies this. Otherwise we potentially expose our
> > users
> > > to
> > > > different behavior IN their data. Not just a schedule by in their SQL
> > or
> > > in
> > > > some scripting where logical date / execution date is used or where
> > date
> > > > arithmetic is applied.
> > > >
> > > > So it is not changing one car for another. It is changing the means
> of
> > > > transportation and not knowing if you can get to all the places you
> > want
> > > /
> > > > need to go as before.
> > > >
> > > > Honestly, I'm a bit scared (I do see the lack of responsiveness). I
> > hope
> > > > someone can take that away.
> > > >
> > > > Bolke
> > > >
> > > >
> > > > Sent from my iPhone
> > > >
> > > > > On 1 Dec 2023, at 13:39, Jarek Potiuk <ja...@potiuk.com> wrote:
> > > > >
> > > > > Agree. I think this is the best way forward.- we have to bite the
> > > > bullet of
> > > > > potential backwards compatibility issues and separating out a
> compat
> > > > layer
> > > > > where we could optionally use pendulum instead would be the most
> > "easy"
> > > > > approach for our users. Most of them would not even notice if we
> do a
> > > > good
> > > > > job, but those who somehow depend on pendulum objects would get a
> > > > > possibility to switch back.
> > > > >
> > > > >> On Fri, Dec 1, 2023 at 1:09 PM Andrey Anshin <
> > > andrey.anshin@taragol.is>
> > > > >> wrote:
> > > > >>
> > > > >> IMHO, if we would like to replace pendulum to alternatives, the
> > better
> > > > >> alternative would be combination of datetime.datetime + zoneinfo +
> > > > >> datetime.timezone
> > > > >>
> > > > >> We still do hacks around pendulum functional, which makes hacks
> > around
> > > > >> datetime.datetime, so in my perspective better remove redundant
> > layers
> > > > and
> > > > >> use native objects directly.
> > > > >>
> > > > >> What confuses me the most is how we get around type changes in
> Tasks
> > > > >> Context. All dates return as pendulum.DateTime for a very long
> time
> > > > >> (literally for ages) and it is probably could be classified as
> > > breaking
> > > > >> changes, because it might break quite a bit users pipelines. It
> > still
> > > > could
> > > > >> be resolved as optional pendulum dependency + config option for
> > > datetime
> > > > >> types in context, which might help to get familiar types for a
> > while.
> > > > >>
> > > > >>
> > > > >>> On Wed, 29 Nov 2023 at 05:58, Jarek Potiuk <ja...@potiuk.com>
> > wrote:
> > > > >>>
> > > > >>> "infamous fork of " should be "fork of infamous Akka" - just to
> be
> > > > clear
> > > > >> :)
> > > > >>>
> > > > >>>> On Wed, Nov 29, 2023 at 2:57 AM Jarek Potiuk <ja...@potiuk.com>
> > > > wrote:
> > > > >>>
> > > > >>>> It's a very rare occurrence that ASF accepts forks. Usually it
> has
> > > to
> > > > >> be
> > > > >>>> willingly donated by those who own the IP rights, - mainly
> because
> > > of
> > > > >>>> trademark issues. Licence is one thing, but carrying things like
> > the
> > > > >> name
> > > > >>>> of the project is not possible without clearing IP and donating
> > the
> > > > >>>> project.
> > > > >>>>
> > > > >>>> It happened last time last year - with Pekko project - infamous
> > fork
> > > > or
> > > > >>>> Akka after Lightbend changed licensing for Akka -
> > > > >>>>
> > > > >>
> > >
> https://www.lightbend.com/blog/why-we-are-changing-the-license-for-akka
> > > > >>>> making it impossible to be used in ASF projects with the new
> > > licence.
> > > > >> But
> > > > >>>> it was a really "strong" case and the "we do not accept forks
> > > usually"
> > > > >>> was
> > > > >>>> quite a contentious issue (I followed the discussion - it was
> > > > >> fascinating
> > > > >>>> actually). It tool a LOT of time and effort for those who led
> the
> > > > >> effort
> > > > >>>> eventually - and also mostly because things had to be
> > crystal-clear
> > > > and
> > > > >>> not
> > > > >>>> "litigable" because the other side had some business reasons to
> > make
> > > > >> the
> > > > >>>> licensing changes and they did not understood why it is
> impossible
> > > to
> > > > >> use
> > > > >>>> Akka as-is even with all the exclusions they offered to ASF -
> and
> > > > >>>> effectively it was a "hostile" move.
> > > > >>>>
> > > > >>>> I don't think it's "worth" it in this case to be honest -
> > especially
> > > > if
> > > > >>>> the maintainers are effectively "silent" and seem to ignore
> > problems
> > > > of
> > > > >>> one
> > > > >>>> of the really serious users they have. If they were willing to
> > > > >> cooperate,
> > > > >>>> we could do a lot, but when we are left alone, I think it will
> be
> > > far
> > > > >>> more
> > > > >>>> convenient for us to look for alternatives - to be honest.
> > > > >>>>
> > > > >>>> J.
> > > > >>>>
> > > > >>>>
> > > > >>>> On Wed, Nov 29, 2023 at 2:46 AM Austin Bennett <
> austin@apache.org
> > >
> > > > >>> wrote:
> > > > >>>>
> > > > >>>>> An option would be to fork Pendulum? It is MIT Licensed, I
> don't
> > > > know
> > > > >>>>> whether that poses problems to get in ASF?
> > > > >>>>>
> > > > >>>>> If forking (?) which is somewhat non-ideal, would we want that
> > 'in'
> > > > >>>>> airflow?
> > > > >>>>>
> > > > >>>>> If not 'in' airflow, I wonder if ASF incubator would accept a
> > > forked
> > > > >>>>> project? [ anecdotally Linux Foundation has incubated
> Starrocks
> > > > which
> > > > >>> is
> > > > >>>>> a
> > > > >>>>> fork of Apache Doris, so there's precedence for this to be OK
> in
> > > the
> > > > >>> wider
> > > > >>>>> OSS ecosystem ].
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> On Tue, Nov 28, 2023 at 6:41 PM Jarek Potiuk <jarek@potiuk.com
> >
> > > > >> wrote:
> > > > >>>>>
> > > > >>>>>> I have not heard back from maintainers, just a comment from
> > > someone
> > > > >>> else
> > > > >>>>>> who suggested donating a pendulum to the ASF (which is kinda
> > > > >>> interesting
> > > > >>>>>> idea). I followed up, let's see. I think if we do not hear
> back
> > > and
> > > > >>>>> there
> > > > >>>>>> will be another week or two where "Pendulum 3 supporting 3.12
> > > > >> support
> > > > >>>>> "is
> > > > >>>>>> coming" but no-one knows when, effectively means that we
> should
> > > look
> > > > >>>>> for an
> > > > >>>>>> alternative like NOW.
> > > > >>>>>>
> > > > >>>>>> J.
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> On Wed, Nov 22, 2023 at 3:27 AM Daniel Standish
> > > > >>>>>> <da...@astronomer.io.invalid> wrote:
> > > > >>>>>>
> > > > >>>>>>> Thanks Jarek
> > > > >>>>>>>
> > > > >>>>>>> On Tue, Nov 21, 2023, 4:34 PM Jarek Potiuk <jarek@potiuk.com
> >
> > > > >>> wrote:
> > > > >>>>>>>
> > > > >>>>>>>> I think we miss important insight - straight from the
> source.
> > > > >>>>>>>>
> > > > >>>>>>>> I believe it's time to be candid and simply ask questions
> for
> > > > >> the
> > > > >>>>>> future
> > > > >>>>>>> of
> > > > >>>>>>>> Pendulum directly where we should - ie. we should just ask
> > > > >>>>>> maintainers.
> > > > >>>>>>>>
> > > > >>>>>>>> I've just started a very candid and open discussion there -
> > > > >>>>>>>> https://github.com/sdispater/pendulum/discussions/771 .
> > > > >>>>>>>>
> > > > >>>>>>>> I linked back to this discussion and explained where we are
> > > > >> coming
> > > > >>>>>> from.
> > > > >>>>>>> I
> > > > >>>>>>>> even offered an option, that maybe - if they accept some
> help
> > > > >> and
> > > > >>>>> would
> > > > >>>>>>> be
> > > > >>>>>>>> open to make a few of us maintainers of Pendulum, that I
> will
> > > > >> ask
> > > > >>> if
> > > > >>>>>> some
> > > > >>>>>>>> of the maintainers here in Airflow would like to step up (I
> > > > >> would
> > > > >>> be
> > > > >>>>>>>> willing to - for one) and help to at least move Pendulum
> > through
> > > > >>>>> 3.12
> > > > >>>>>> and
> > > > >>>>>>>> maybe keep it running for as long as it will be needed to
> > > > >> maintain
> > > > >>>>> it.
> > > > >>>>>>>>
> > > > >>>>>>>> I think if they would be open to some of us helping them -
> > this
> > > > >>>>> might
> > > > >>>>>> be
> > > > >>>>>>>> actually the "cheapest" option for us to be honest at least
> > in a
> > > > >>>>>> mid-term
> > > > >>>>>>>> if we could gain influence and even take part and speed up
> > > > >>> releases
> > > > >>>>> and
> > > > >>>>>>>> merging of issues that are blocking us (or would be blocking
> > us
> > > > >> in
> > > > >>>>> the
> > > > >>>>>>>> future).
> > > > >>>>>>>>
> > > > >>>>>>>> I tried to be very friendly but candid and direct on what
> kind
> > > > >> of
> > > > >>>>>>> problems
> > > > >>>>>>>> it creates us and also expressed our thanks for supporting
> us
> > > > >> for
> > > > >>> so
> > > > >>>>>> long
> > > > >>>>>>>> (I do believe they deserve it). I think the sole (pretty
> > much)
> > > > >>>>>>> maintainer
> > > > >>>>>>>> of Pendulum might really have a hard time maintaining and
> > > > >> keeping
> > > > >>>>> it up
> > > > >>>>>>> for
> > > > >>>>>>>> years - without being paid for and thanked. I hope - maybe
> all
> > > > >>> that
> > > > >>>>>> they
> > > > >>>>>>>> need are a few words of encouragement and thanks and
> realising
> > > > >>> that
> > > > >>>>>>> others
> > > > >>>>>>>> depend on their work - but also seeing that they are
> > understood,
> > > > >>>>>>> empathised
> > > > >>>>>>>> with and that there are viable alternatives they might
> follow
> > > > >>> might
> > > > >>>>> be
> > > > >>>>>>>> helpful for them to be also candid and respond. I hope my
> > > > >>> intentions
> > > > >>>>>> will
> > > > >>>>>>>> be understood - that I am not complaining, but even trying
> to
> > > > >>> help.
> > > > >>>>>>>>
> > > > >>>>>>>> Let\s see where it gets us.
> > > > >>>>>>>>
> > > > >>>>>>>> J.
> > > > >>>>>>>>
> > > > >>>>>>>> On Fri, Nov 17, 2023 at 5:32 PM Andrey Anshin <
> > > > >>>>>> andrey.anshin@taragol.is>
> > > > >>>>>>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> Just to clarify I'd like us to consider the possibility
> that
> > > > >> no
> > > > >>>>> new
> > > > >>>>>>>>> pendulum would be released or released at the end of 2024,
> > > > >> like
> > > > >>> a
> > > > >>>>>>>>> pessimistic scenario:
> > > > >>>>>>>>> - What should we do in this case?
> > > > >>>>>>>>> - Work out a backup plan.
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>> ----
> > > > >>>>>>>>> Best Wishes
> > > > >>>>>>>>> *Andrey Anshin*
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Fri, 17 Nov 2023 at 16:33, Jarek Potiuk <
> jarek@potiuk.com
> > >
> > > > >>>>> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> Also I think TP - had a document in the past (years ago)
> > > > >>>>>>> describing a
> > > > >>>>>>>>>> draft of a more complete alternative we can take to
> approach
> > > > >>>>>> datetime
> > > > >>>>>>>> vs.
> > > > >>>>>>>>>> pendulum dichotomy. I cannot easily find the document and
> > > > >>>>>> discussion
> > > > >>>>>>> -
> > > > >>>>>>>>> but
> > > > >>>>>>>>>> I do remember it was proposing some interesting changes in
> > > > >> the
> > > > >>>>>>> approach
> > > > >>>>>>>>> of
> > > > >>>>>>>>>> Airflow to have an abstraction layer on top of it (as far
> > > > >> as I
> > > > >>>>>>>> remember).
> > > > >>>>>>>>>> Maybe we can resurrect that idea if TP might find the
> > > > >>> proposal ?
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Fri, Nov 17, 2023 at 1:06 PM Bolke de Bruin <
> > > > >>>>> bdbruin@gmail.com>
> > > > >>>>>>>>> wrote:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> Hi,
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> I agree that the current speed of development of Pendulum
> > > > >>>>> leaves
> > > > >>>>>>>>>> something
> > > > >>>>>>>>>>> to be desired. However, I think we should not
> > > > >> underestimate
> > > > >>>>> the
> > > > >>>>>>>> effort
> > > > >>>>>>>>> of
> > > > >>>>>>>>>>> replacing it. It is not just a matter of
> > > > >>>>> %s/pendulum/datetime/g
> > > > >>>>>> so
> > > > >>>>>>> to
> > > > >>>>>>>>>> say.
> > > > >>>>>>>>>>> If we are *truly* thinking about moving to native
> > > > >> datetime /
> > > > >>>>>>> zoneinfo
> > > > >>>>>>>>> etc
> > > > >>>>>>>>>>> we need *extensive* tests, basically copying what
> pendulum
> > > > >>>>> does
> > > > >>>>>> to
> > > > >>>>>>>>> check
> > > > >>>>>>>>>>> its behavior. The reason is that native implementations
> in
> > > > >>> the
> > > > >>>>>> past
> > > > >>>>>>>>> made
> > > > >>>>>>>>>>> serious mistakes and I do not put a lot of trust in them.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> An abstraction or vendoring in could be alternatives, but
> > > > >>> they
> > > > >>>>>>> bring
> > > > >>>>>>>>>> their
> > > > >>>>>>>>>>> own significant challenges.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> I have re-raised the issue here:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> https://github.com/sdispater/pendulum/issues/753
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> The upside is that it seems the amount of issues with the
> > > > >>>>> beta is
> > > > >>>>>>>>>> limited,
> > > > >>>>>>>>>>> so hopefully the maintainers can spend a couple of cycles
> > > > >> to
> > > > >>>>>>> address
> > > > >>>>>>>>>> them.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Bolke
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> On Fri, 17 Nov 2023 at 10:28, Andrey Anshin <
> > > > >>>>>>>> andrey.anshin@taragol.is>
> > > > >>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> There is no changes in stable pendulum so let's try to
> > > > >>>>> continue
> > > > >>>>>>>> this
> > > > >>>>>>>>>>>> discussion and start think about "Plan B"
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Just a reminder:
> > > > >>>>>>>>>>>> - pendulum 2.1.2 released 3 years ago (at the time
> > > > >> Airflow
> > > > >>>>>>> 1.10.x)
> > > > >>>>>>>>>>>> - pendulum 2 doesn't work well in Python 3.12, this is a
> > > > >>>>>>>> showstopper
> > > > >>>>>>>>>> for
> > > > >>>>>>>>>>>> the support Python 3.12
> > > > >>>>>>>>>>>> - pendulum 2 have memory leaks (
> > > > >>>>>>>>>>>> https://github.com/sdispater/pendulum/issues/720), and
> > > > >>>>> Airflow
> > > > >>>>>>> use
> > > > >>>>>>>>>>>> approach
> > > > >>>>>>>>>>>> to achieve this leaks, especially in K8S executor
> > > > >>>>>>>>>>>> - pendulum 2 doesn't use system tzdata by default, but
> > > > >> we
> > > > >>>>> have
> > > > >>>>>> a
> > > > >>>>>>>>>>>> workaround, thanks Bolke for the documentation
> > > > >>>>>>>>>>>> - pendulum it is a core dependency of Airflow, and I
> > > > >> guess
> > > > >>>>> we
> > > > >>>>>>> can't
> > > > >>>>>>>>>>>> remove/replace it without breaking changes.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> So my proposal if things won't change in the near future
> > > > >>>>> then
> > > > >>>>>> we
> > > > >>>>>>>> need
> > > > >>>>>>>>>> to
> > > > >>>>>>>>>>>> start removing the pendulum from the core and replace it
> > > > >>>>> with
> > > > >>>>>> the
> > > > >>>>>>>>>> native
> > > > >>>>>>>>>>>> datetime / zoneinfo / tzinfo. But maybe we have another
> > > > >>> way
> > > > >>>>> to
> > > > >>>>>>>>> resolve
> > > > >>>>>>>>>>> this
> > > > >>>>>>>>>>>> without breaking changes? Because for me it would be a
> > > > >>>>> little
> > > > >>>>>>> weird
> > > > >>>>>>>>> if
> > > > >>>>>>>>>>>> removal pendulum would be a main driver for release
> > > > >>> Airflow
> > > > >>>>> 3
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> ----
> > > > >>>>>>>>>>>> Best Wishes
> > > > >>>>>>>>>>>> *Andrey Anshin*
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> On Thu, 28 Sept 2023 at 13:01, Andrey Anshin <
> > > > >>>>>>>>> andrey.anshin@taragol.is
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> This discussion is more about the known problem of
> > > > >>>>> pendulum
> > > > >>>>>> and
> > > > >>>>>>>> how
> > > > >>>>>>>>>> we
> > > > >>>>>>>>>>>>> could deal with it and maybe how we (as Community)
> > > > >> might
> > > > >>>>> help
> > > > >>>>>>>>> autor.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> The library is mostly supported by a single author
> > > > >>>>> Sébastien
> > > > >>>>>>>>> Eustace
> > > > >>>>>>>>>> (
> > > > >>>>>>>>>>>>> https://github.com/sdispater) and it seems like we
> > > > >> bump
> > > > >>>>> into
> > > > >>>>>>> the
> > > > >>>>>>>>>>>>> situation which is described in xkcd #2347 (
> > > > >>>>>>>>>>>>> https://imgs.xkcd.com/comics/dependency.png). To be
> > > > >>>>> honest
> > > > >>>>>> it
> > > > >>>>>>> is
> > > > >>>>>>>>> not
> > > > >>>>>>>>>>>>> something new when library mainly supported by one
> > > > >>> author
> > > > >>>>> so
> > > > >>>>>>>> there
> > > > >>>>>>>>> is
> > > > >>>>>>>>>>>>> always a risk that the library will no longer be
> > > > >>>>> supported /
> > > > >>>>>>>>>> abandoned
> > > > >>>>>>>>>>>>> And if takes in account that pendulum provides core
> > > > >>>>>>> functionality
> > > > >>>>>>>>> in
> > > > >>>>>>>>>>>>> Airflow it could have dramatical impact in the future.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Pendulum is a really nice library which helps a lot of
> > > > >>>>>>> developers
> > > > >>>>>>>>> to
> > > > >>>>>>>>>>> work
> > > > >>>>>>>>>>>>> with dates/datetimes. However there is one major
> > > > >>> problem,
> > > > >>>>> the
> > > > >>>>>>>> last
> > > > >>>>>>>>>>>> release
> > > > >>>>>>>>>>>>> of this library happened more than 3 years ago (
> > > > >>>>>>>>>>>>> https://pypi.org/project/pendulum/#history) in the
> > > > >> time
> > > > >>>>> when
> > > > >>>>>>>>> Airflow
> > > > >>>>>>>>>>>>> 1.10.11 was released
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Fortunately, the project is not abandoned and on a
> > > > >>> regular
> > > > >>>>>>> basis
> > > > >>>>>>>>>>> commits
> > > > >>>>>>>>>>>>> add into the master branch. However these commits are
> > > > >>> not
> > > > >>>>>>>> included
> > > > >>>>>>>>>> into
> > > > >>>>>>>>>>>> any
> > > > >>>>>>>>>>>>> final release and that's why some things related to
> > > > >>>>> datetime
> > > > >>>>>>>> don't
> > > > >>>>>>>>>> work
> > > > >>>>>>>>>>>> as
> > > > >>>>>>>>>>>>> expected in Airflow. There are list of known (for me)
> > > > >>>>> issues
> > > > >>>>>>>> which
> > > > >>>>>>>>>> are
> > > > >>>>>>>>>>>>> affect Airflow
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> *Memory Leak on parse*:
> > > > >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/720,
> > > > >>> this
> > > > >>>>> one
> > > > >>>>>>>> fixed
> > > > >>>>>>>>>> 2
> > > > >>>>>>>>>>>>> years ago but not available yet (
> > > > >>>>>>>>>>>>> https://github.com/sdispater/pendulum/pull/563).
> > > > >> Since
> > > > >>> we
> > > > >>>>>> use
> > > > >>>>>>>>> parse
> > > > >>>>>>>>>>>> dates
> > > > >>>>>>>>>>>>> in airflow codebase: datetime parameters and datetime
> > > > >> in
> > > > >>>>> logs
> > > > >>>>>>>> this
> > > > >>>>>>>>>> one
> > > > >>>>>>>>>>>>> could be a reason for memory leakage in Airflow:
> > > > >>>>>>>>>>>>> - https://github.com/apache/airflow/discussions/24694
> > > > >>>>>>>>>>>>> - https://github.com/apache/airflow/discussions/28597
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> *Incorrect time zones*, known issues and should be
> > > > >>> already
> > > > >>>>>>> fixed
> > > > >>>>>>>> in
> > > > >>>>>>>>>>>>> master branch
> > > > >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/700,
> > > > >>>>> Mexico
> > > > >>>>>> do
> > > > >>>>>>>> not
> > > > >>>>>>>>>> use
> > > > >>>>>>>>>>>> DST
> > > > >>>>>>>>>>>>> anymore
> > > > >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/706,
> > > > >>> Egypt
> > > > >>>>>>>>> reinstate
> > > > >>>>>>>>>>> DST
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> We add clarification in
> > > > >>>>>>>>> https://github.com/apache/airflow/pull/30467
> > > > >>>>>>>>>> ,
> > > > >>>>>>>>>>>>> however it seems like there is no other way rather
> > > > >> than
> > > > >>>>>>> patching
> > > > >>>>>>>>>>> Pendulum
> > > > >>>>>>>>>>>>> right now.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> All these issues should be solved as soon as pendulum
> > > > >> 3
> > > > >>> is
> > > > >>>>>>>>> released.
> > > > >>>>>>>>>>> The
> > > > >>>>>>>>>>>>> current announced estimation is end of september/
> > > > >>>>> beginning
> > > > >>>>>> of
> > > > >>>>>>>>>> October:
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>
> > > > >>>>>
> > > > >>>
> > > >
> > https://github.com/sdispater/pendulum/issues/600#issuecomment-1711299677
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> So in theory we would have a fixed version of pendulum
> > > > >>>>> soon,
> > > > >>>>>>> and
> > > > >>>>>>>> it
> > > > >>>>>>>>>>> might
> > > > >>>>>>>>>>>>> break something in Airflow but from my point of view
> > > > >> it
> > > > >>> is
> > > > >>>>>>> better
> > > > >>>>>>>>>> than
> > > > >>>>>>>>>>>>> current status.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> However there might be a situation where the release
> > > > >> of
> > > > >>>>> the
> > > > >>>>>>>>> pendulum
> > > > >>>>>>>>>>>> would
> > > > >>>>>>>>>>>>> be postponed, so maybe better to have a backup plan.
> > > > >>> What
> > > > >>>>>> could
> > > > >>>>>>>> we
> > > > >>>>>>>>> do
> > > > >>>>>>>>>>> in
> > > > >>>>>>>>>>>>> this case?
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Maybe we should start to use zoneinfo.ZoneInfo instead
> > > > >>> of
> > > > >>>>>>>> pendulum
> > > > >>>>>>>>>>>>> datetime?
> > > > >>> https://github.com/apache/airflow/issues/19450
> > > > >>>>>>>>>>>>> Pros:
> > > > >>>>>>>>>>>>> - stdlib (python 3.9+)
> > > > >>>>>>>>>>>>> - In pendulum 3.0 Timezone based on zoneinfo.Zoneinfo
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Cons:
> > > > >>>>>>>>>>>>> - Current serialization model can't deal with backport
> > > > >>>>>>> packages.
> > > > >>>>>>>>> E.g.
> > > > >>>>>>>>>>>>> timezone which are serialized in backport_zoneinfo
> > > > >> can't
> > > > >>>>> be
> > > > >>>>>>>>>>> deserialized
> > > > >>>>>>>>>>>> in
> > > > >>>>>>>>>>>>> zoneinfo
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Maybe we should replace parse datetime with another
> > > > >>>>> solution.
> > > > >>>>>>>> Does
> > > > >>>>>>>>>>> anyone
> > > > >>>>>>>>>>>>> know a good replacement?
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Maybe someone from Airflow Community could propose
> > > > >> their
> > > > >>>>> help
> > > > >>>>>>>> with
> > > > >>>>>>>>>>>>> maintenance of library:
> > > > >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/590
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Maybe we should get rid of the pendulum at all, as a
> > > > >>> last
> > > > >>>>>>> resort
> > > > >>>>>>>>>>>> solution.
> > > > >>>>>>>>>>>>> I can't imagine how we could do that, because a lot of
> > > > >>>>> stuff
> > > > >>>>>>>>> depends
> > > > >>>>>>>>>> on
> > > > >>>>>>>>>>>> the
> > > > >>>>>>>>>>>>> pendulum and removing it would be a breaking change.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> ----
> > > > >>>>>>>>>>>>> Best Wishes
> > > > >>>>>>>>>>>>> *Andrey Anshin*
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> --
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> --
> > > > >>>>>>>>>>> Bolke de Bruin
> > > > >>>>>>>>>>> bdbruin@gmail.com
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > >
> > > >
> > >
> >
>
>
> --
>
> --
> Bolke de Bruin
> bdbruin@gmail.com
>
Re: [DISCUSS] Future of Pendulum in Airflow
Posted by Bolke de Bruin <bd...@gmail.com>.
Pendulum 3 was released.
https://pypi.org/project/pendulum/
Bolke
On Fri, 1 Dec 2023 at 18:38, Aritra Basu <ar...@gmail.com> wrote:
> I agree with wanting to stay away from having to deal with dst but I am on
> the side of biting the bullet and transitioning to datetime and testing
> thoroughly, especially considering our current predicament with pendulum
> and the potential of us being in the same place again potentially in the
> future.
>
> --
> Regards,
> Aritra Basu
>
> On Fri, Dec 1, 2023, 10:27 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Yeah. Agree it's a challenge but I think it's also an opportunity to
> > fix those - because whether we admit it, or not - we already have those
> > problems - regardless of Pendulum's use.
> >
> > If you look at the issues (especially ones created around DST time) -
> there
> > are always one or two issues (even in latest versions of Airflow) where
> DST
> > transition is causing people problems. And they are rarely
> > solved/addressed.
> >
> > Example from last October:
> >
> > https://github.com/apache/airflow/issues/35272
> >
> > Example from May 2022:
> >
> > https://github.com/apache/airflow/issues/23400
> >
> > And Example from March 2020 ( which might or might not be fixed by the
> open
> > PR):
> >
> > https://github.com/apache/airflow/issues/7999
> >
> > J.
> >
> >
> > On Fri, Dec 1, 2023 at 5:41 PM Bolke de Bruin <bd...@gmail.com> wrote:
> >
> > > The alternative is to vendor in or fork pendulum as part of the Airflow
> > > project. Also a major undertaking but maybe less effort.
> > >
> > > We shouldn't just address this as a technical challenge. The question
> > > becomes: does that combination fornally do DST transitions correctly,
> > does
> > > it do leap seconds correctly, does it add dates correctly, does it do
> > > timezone traversal correctly and many many other things.
> > >
> > > So if we switch to the datetime + zone info + timezone we need to have
> a
> > > test suite that verifies this. Otherwise we potentially expose our
> users
> > to
> > > different behavior IN their data. Not just a schedule by in their SQL
> or
> > in
> > > some scripting where logical date / execution date is used or where
> date
> > > arithmetic is applied.
> > >
> > > So it is not changing one car for another. It is changing the means of
> > > transportation and not knowing if you can get to all the places you
> want
> > /
> > > need to go as before.
> > >
> > > Honestly, I'm a bit scared (I do see the lack of responsiveness). I
> hope
> > > someone can take that away.
> > >
> > > Bolke
> > >
> > >
> > > Sent from my iPhone
> > >
> > > > On 1 Dec 2023, at 13:39, Jarek Potiuk <ja...@potiuk.com> wrote:
> > > >
> > > > Agree. I think this is the best way forward.- we have to bite the
> > > bullet of
> > > > potential backwards compatibility issues and separating out a compat
> > > layer
> > > > where we could optionally use pendulum instead would be the most
> "easy"
> > > > approach for our users. Most of them would not even notice if we do a
> > > good
> > > > job, but those who somehow depend on pendulum objects would get a
> > > > possibility to switch back.
> > > >
> > > >> On Fri, Dec 1, 2023 at 1:09 PM Andrey Anshin <
> > andrey.anshin@taragol.is>
> > > >> wrote:
> > > >>
> > > >> IMHO, if we would like to replace pendulum to alternatives, the
> better
> > > >> alternative would be combination of datetime.datetime + zoneinfo +
> > > >> datetime.timezone
> > > >>
> > > >> We still do hacks around pendulum functional, which makes hacks
> around
> > > >> datetime.datetime, so in my perspective better remove redundant
> layers
> > > and
> > > >> use native objects directly.
> > > >>
> > > >> What confuses me the most is how we get around type changes in Tasks
> > > >> Context. All dates return as pendulum.DateTime for a very long time
> > > >> (literally for ages) and it is probably could be classified as
> > breaking
> > > >> changes, because it might break quite a bit users pipelines. It
> still
> > > could
> > > >> be resolved as optional pendulum dependency + config option for
> > datetime
> > > >> types in context, which might help to get familiar types for a
> while.
> > > >>
> > > >>
> > > >>> On Wed, 29 Nov 2023 at 05:58, Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > > >>>
> > > >>> "infamous fork of " should be "fork of infamous Akka" - just to be
> > > clear
> > > >> :)
> > > >>>
> > > >>>> On Wed, Nov 29, 2023 at 2:57 AM Jarek Potiuk <ja...@potiuk.com>
> > > wrote:
> > > >>>
> > > >>>> It's a very rare occurrence that ASF accepts forks. Usually it has
> > to
> > > >> be
> > > >>>> willingly donated by those who own the IP rights, - mainly because
> > of
> > > >>>> trademark issues. Licence is one thing, but carrying things like
> the
> > > >> name
> > > >>>> of the project is not possible without clearing IP and donating
> the
> > > >>>> project.
> > > >>>>
> > > >>>> It happened last time last year - with Pekko project - infamous
> fork
> > > or
> > > >>>> Akka after Lightbend changed licensing for Akka -
> > > >>>>
> > > >>
> > https://www.lightbend.com/blog/why-we-are-changing-the-license-for-akka
> > > >>>> making it impossible to be used in ASF projects with the new
> > licence.
> > > >> But
> > > >>>> it was a really "strong" case and the "we do not accept forks
> > usually"
> > > >>> was
> > > >>>> quite a contentious issue (I followed the discussion - it was
> > > >> fascinating
> > > >>>> actually). It tool a LOT of time and effort for those who led the
> > > >> effort
> > > >>>> eventually - and also mostly because things had to be
> crystal-clear
> > > and
> > > >>> not
> > > >>>> "litigable" because the other side had some business reasons to
> make
> > > >> the
> > > >>>> licensing changes and they did not understood why it is impossible
> > to
> > > >> use
> > > >>>> Akka as-is even with all the exclusions they offered to ASF - and
> > > >>>> effectively it was a "hostile" move.
> > > >>>>
> > > >>>> I don't think it's "worth" it in this case to be honest -
> especially
> > > if
> > > >>>> the maintainers are effectively "silent" and seem to ignore
> problems
> > > of
> > > >>> one
> > > >>>> of the really serious users they have. If they were willing to
> > > >> cooperate,
> > > >>>> we could do a lot, but when we are left alone, I think it will be
> > far
> > > >>> more
> > > >>>> convenient for us to look for alternatives - to be honest.
> > > >>>>
> > > >>>> J.
> > > >>>>
> > > >>>>
> > > >>>> On Wed, Nov 29, 2023 at 2:46 AM Austin Bennett <austin@apache.org
> >
> > > >>> wrote:
> > > >>>>
> > > >>>>> An option would be to fork Pendulum? It is MIT Licensed, I don't
> > > know
> > > >>>>> whether that poses problems to get in ASF?
> > > >>>>>
> > > >>>>> If forking (?) which is somewhat non-ideal, would we want that
> 'in'
> > > >>>>> airflow?
> > > >>>>>
> > > >>>>> If not 'in' airflow, I wonder if ASF incubator would accept a
> > forked
> > > >>>>> project? [ anecdotally Linux Foundation has incubated Starrocks
> > > which
> > > >>> is
> > > >>>>> a
> > > >>>>> fork of Apache Doris, so there's precedence for this to be OK in
> > the
> > > >>> wider
> > > >>>>> OSS ecosystem ].
> > > >>>>>
> > > >>>>>
> > > >>>>> On Tue, Nov 28, 2023 at 6:41 PM Jarek Potiuk <ja...@potiuk.com>
> > > >> wrote:
> > > >>>>>
> > > >>>>>> I have not heard back from maintainers, just a comment from
> > someone
> > > >>> else
> > > >>>>>> who suggested donating a pendulum to the ASF (which is kinda
> > > >>> interesting
> > > >>>>>> idea). I followed up, let's see. I think if we do not hear back
> > and
> > > >>>>> there
> > > >>>>>> will be another week or two where "Pendulum 3 supporting 3.12
> > > >> support
> > > >>>>> "is
> > > >>>>>> coming" but no-one knows when, effectively means that we should
> > look
> > > >>>>> for an
> > > >>>>>> alternative like NOW.
> > > >>>>>>
> > > >>>>>> J.
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> On Wed, Nov 22, 2023 at 3:27 AM Daniel Standish
> > > >>>>>> <da...@astronomer.io.invalid> wrote:
> > > >>>>>>
> > > >>>>>>> Thanks Jarek
> > > >>>>>>>
> > > >>>>>>> On Tue, Nov 21, 2023, 4:34 PM Jarek Potiuk <ja...@potiuk.com>
> > > >>> wrote:
> > > >>>>>>>
> > > >>>>>>>> I think we miss important insight - straight from the source.
> > > >>>>>>>>
> > > >>>>>>>> I believe it's time to be candid and simply ask questions for
> > > >> the
> > > >>>>>> future
> > > >>>>>>> of
> > > >>>>>>>> Pendulum directly where we should - ie. we should just ask
> > > >>>>>> maintainers.
> > > >>>>>>>>
> > > >>>>>>>> I've just started a very candid and open discussion there -
> > > >>>>>>>> https://github.com/sdispater/pendulum/discussions/771 .
> > > >>>>>>>>
> > > >>>>>>>> I linked back to this discussion and explained where we are
> > > >> coming
> > > >>>>>> from.
> > > >>>>>>> I
> > > >>>>>>>> even offered an option, that maybe - if they accept some help
> > > >> and
> > > >>>>> would
> > > >>>>>>> be
> > > >>>>>>>> open to make a few of us maintainers of Pendulum, that I will
> > > >> ask
> > > >>> if
> > > >>>>>> some
> > > >>>>>>>> of the maintainers here in Airflow would like to step up (I
> > > >> would
> > > >>> be
> > > >>>>>>>> willing to - for one) and help to at least move Pendulum
> through
> > > >>>>> 3.12
> > > >>>>>> and
> > > >>>>>>>> maybe keep it running for as long as it will be needed to
> > > >> maintain
> > > >>>>> it.
> > > >>>>>>>>
> > > >>>>>>>> I think if they would be open to some of us helping them -
> this
> > > >>>>> might
> > > >>>>>> be
> > > >>>>>>>> actually the "cheapest" option for us to be honest at least
> in a
> > > >>>>>> mid-term
> > > >>>>>>>> if we could gain influence and even take part and speed up
> > > >>> releases
> > > >>>>> and
> > > >>>>>>>> merging of issues that are blocking us (or would be blocking
> us
> > > >> in
> > > >>>>> the
> > > >>>>>>>> future).
> > > >>>>>>>>
> > > >>>>>>>> I tried to be very friendly but candid and direct on what kind
> > > >> of
> > > >>>>>>> problems
> > > >>>>>>>> it creates us and also expressed our thanks for supporting us
> > > >> for
> > > >>> so
> > > >>>>>> long
> > > >>>>>>>> (I do believe they deserve it). I think the sole (pretty
> much)
> > > >>>>>>> maintainer
> > > >>>>>>>> of Pendulum might really have a hard time maintaining and
> > > >> keeping
> > > >>>>> it up
> > > >>>>>>> for
> > > >>>>>>>> years - without being paid for and thanked. I hope - maybe all
> > > >>> that
> > > >>>>>> they
> > > >>>>>>>> need are a few words of encouragement and thanks and realising
> > > >>> that
> > > >>>>>>> others
> > > >>>>>>>> depend on their work - but also seeing that they are
> understood,
> > > >>>>>>> empathised
> > > >>>>>>>> with and that there are viable alternatives they might follow
> > > >>> might
> > > >>>>> be
> > > >>>>>>>> helpful for them to be also candid and respond. I hope my
> > > >>> intentions
> > > >>>>>> will
> > > >>>>>>>> be understood - that I am not complaining, but even trying to
> > > >>> help.
> > > >>>>>>>>
> > > >>>>>>>> Let\s see where it gets us.
> > > >>>>>>>>
> > > >>>>>>>> J.
> > > >>>>>>>>
> > > >>>>>>>> On Fri, Nov 17, 2023 at 5:32 PM Andrey Anshin <
> > > >>>>>> andrey.anshin@taragol.is>
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Just to clarify I'd like us to consider the possibility that
> > > >> no
> > > >>>>> new
> > > >>>>>>>>> pendulum would be released or released at the end of 2024,
> > > >> like
> > > >>> a
> > > >>>>>>>>> pessimistic scenario:
> > > >>>>>>>>> - What should we do in this case?
> > > >>>>>>>>> - Work out a backup plan.
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> ----
> > > >>>>>>>>> Best Wishes
> > > >>>>>>>>> *Andrey Anshin*
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> On Fri, 17 Nov 2023 at 16:33, Jarek Potiuk <jarek@potiuk.com
> >
> > > >>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Also I think TP - had a document in the past (years ago)
> > > >>>>>>> describing a
> > > >>>>>>>>>> draft of a more complete alternative we can take to approach
> > > >>>>>> datetime
> > > >>>>>>>> vs.
> > > >>>>>>>>>> pendulum dichotomy. I cannot easily find the document and
> > > >>>>>> discussion
> > > >>>>>>> -
> > > >>>>>>>>> but
> > > >>>>>>>>>> I do remember it was proposing some interesting changes in
> > > >> the
> > > >>>>>>> approach
> > > >>>>>>>>> of
> > > >>>>>>>>>> Airflow to have an abstraction layer on top of it (as far
> > > >> as I
> > > >>>>>>>> remember).
> > > >>>>>>>>>> Maybe we can resurrect that idea if TP might find the
> > > >>> proposal ?
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Fri, Nov 17, 2023 at 1:06 PM Bolke de Bruin <
> > > >>>>> bdbruin@gmail.com>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Hi,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I agree that the current speed of development of Pendulum
> > > >>>>> leaves
> > > >>>>>>>>>> something
> > > >>>>>>>>>>> to be desired. However, I think we should not
> > > >> underestimate
> > > >>>>> the
> > > >>>>>>>> effort
> > > >>>>>>>>> of
> > > >>>>>>>>>>> replacing it. It is not just a matter of
> > > >>>>> %s/pendulum/datetime/g
> > > >>>>>> so
> > > >>>>>>> to
> > > >>>>>>>>>> say.
> > > >>>>>>>>>>> If we are *truly* thinking about moving to native
> > > >> datetime /
> > > >>>>>>> zoneinfo
> > > >>>>>>>>> etc
> > > >>>>>>>>>>> we need *extensive* tests, basically copying what pendulum
> > > >>>>> does
> > > >>>>>> to
> > > >>>>>>>>> check
> > > >>>>>>>>>>> its behavior. The reason is that native implementations in
> > > >>> the
> > > >>>>>> past
> > > >>>>>>>>> made
> > > >>>>>>>>>>> serious mistakes and I do not put a lot of trust in them.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> An abstraction or vendoring in could be alternatives, but
> > > >>> they
> > > >>>>>>> bring
> > > >>>>>>>>>> their
> > > >>>>>>>>>>> own significant challenges.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I have re-raised the issue here:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> https://github.com/sdispater/pendulum/issues/753
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> The upside is that it seems the amount of issues with the
> > > >>>>> beta is
> > > >>>>>>>>>> limited,
> > > >>>>>>>>>>> so hopefully the maintainers can spend a couple of cycles
> > > >> to
> > > >>>>>>> address
> > > >>>>>>>>>> them.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Bolke
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Fri, 17 Nov 2023 at 10:28, Andrey Anshin <
> > > >>>>>>>> andrey.anshin@taragol.is>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> There is no changes in stable pendulum so let's try to
> > > >>>>> continue
> > > >>>>>>>> this
> > > >>>>>>>>>>>> discussion and start think about "Plan B"
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Just a reminder:
> > > >>>>>>>>>>>> - pendulum 2.1.2 released 3 years ago (at the time
> > > >> Airflow
> > > >>>>>>> 1.10.x)
> > > >>>>>>>>>>>> - pendulum 2 doesn't work well in Python 3.12, this is a
> > > >>>>>>>> showstopper
> > > >>>>>>>>>> for
> > > >>>>>>>>>>>> the support Python 3.12
> > > >>>>>>>>>>>> - pendulum 2 have memory leaks (
> > > >>>>>>>>>>>> https://github.com/sdispater/pendulum/issues/720), and
> > > >>>>> Airflow
> > > >>>>>>> use
> > > >>>>>>>>>>>> approach
> > > >>>>>>>>>>>> to achieve this leaks, especially in K8S executor
> > > >>>>>>>>>>>> - pendulum 2 doesn't use system tzdata by default, but
> > > >> we
> > > >>>>> have
> > > >>>>>> a
> > > >>>>>>>>>>>> workaround, thanks Bolke for the documentation
> > > >>>>>>>>>>>> - pendulum it is a core dependency of Airflow, and I
> > > >> guess
> > > >>>>> we
> > > >>>>>>> can't
> > > >>>>>>>>>>>> remove/replace it without breaking changes.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> So my proposal if things won't change in the near future
> > > >>>>> then
> > > >>>>>> we
> > > >>>>>>>> need
> > > >>>>>>>>>> to
> > > >>>>>>>>>>>> start removing the pendulum from the core and replace it
> > > >>>>> with
> > > >>>>>> the
> > > >>>>>>>>>> native
> > > >>>>>>>>>>>> datetime / zoneinfo / tzinfo. But maybe we have another
> > > >>> way
> > > >>>>> to
> > > >>>>>>>>> resolve
> > > >>>>>>>>>>> this
> > > >>>>>>>>>>>> without breaking changes? Because for me it would be a
> > > >>>>> little
> > > >>>>>>> weird
> > > >>>>>>>>> if
> > > >>>>>>>>>>>> removal pendulum would be a main driver for release
> > > >>> Airflow
> > > >>>>> 3
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> ----
> > > >>>>>>>>>>>> Best Wishes
> > > >>>>>>>>>>>> *Andrey Anshin*
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Thu, 28 Sept 2023 at 13:01, Andrey Anshin <
> > > >>>>>>>>> andrey.anshin@taragol.is
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> This discussion is more about the known problem of
> > > >>>>> pendulum
> > > >>>>>> and
> > > >>>>>>>> how
> > > >>>>>>>>>> we
> > > >>>>>>>>>>>>> could deal with it and maybe how we (as Community)
> > > >> might
> > > >>>>> help
> > > >>>>>>>>> autor.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> The library is mostly supported by a single author
> > > >>>>> Sébastien
> > > >>>>>>>>> Eustace
> > > >>>>>>>>>> (
> > > >>>>>>>>>>>>> https://github.com/sdispater) and it seems like we
> > > >> bump
> > > >>>>> into
> > > >>>>>>> the
> > > >>>>>>>>>>>>> situation which is described in xkcd #2347 (
> > > >>>>>>>>>>>>> https://imgs.xkcd.com/comics/dependency.png). To be
> > > >>>>> honest
> > > >>>>>> it
> > > >>>>>>> is
> > > >>>>>>>>> not
> > > >>>>>>>>>>>>> something new when library mainly supported by one
> > > >>> author
> > > >>>>> so
> > > >>>>>>>> there
> > > >>>>>>>>> is
> > > >>>>>>>>>>>>> always a risk that the library will no longer be
> > > >>>>> supported /
> > > >>>>>>>>>> abandoned
> > > >>>>>>>>>>>>> And if takes in account that pendulum provides core
> > > >>>>>>> functionality
> > > >>>>>>>>> in
> > > >>>>>>>>>>>>> Airflow it could have dramatical impact in the future.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Pendulum is a really nice library which helps a lot of
> > > >>>>>>> developers
> > > >>>>>>>>> to
> > > >>>>>>>>>>> work
> > > >>>>>>>>>>>>> with dates/datetimes. However there is one major
> > > >>> problem,
> > > >>>>> the
> > > >>>>>>>> last
> > > >>>>>>>>>>>> release
> > > >>>>>>>>>>>>> of this library happened more than 3 years ago (
> > > >>>>>>>>>>>>> https://pypi.org/project/pendulum/#history) in the
> > > >> time
> > > >>>>> when
> > > >>>>>>>>> Airflow
> > > >>>>>>>>>>>>> 1.10.11 was released
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Fortunately, the project is not abandoned and on a
> > > >>> regular
> > > >>>>>>> basis
> > > >>>>>>>>>>> commits
> > > >>>>>>>>>>>>> add into the master branch. However these commits are
> > > >>> not
> > > >>>>>>>> included
> > > >>>>>>>>>> into
> > > >>>>>>>>>>>> any
> > > >>>>>>>>>>>>> final release and that's why some things related to
> > > >>>>> datetime
> > > >>>>>>>> don't
> > > >>>>>>>>>> work
> > > >>>>>>>>>>>> as
> > > >>>>>>>>>>>>> expected in Airflow. There are list of known (for me)
> > > >>>>> issues
> > > >>>>>>>> which
> > > >>>>>>>>>> are
> > > >>>>>>>>>>>>> affect Airflow
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> *Memory Leak on parse*:
> > > >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/720,
> > > >>> this
> > > >>>>> one
> > > >>>>>>>> fixed
> > > >>>>>>>>>> 2
> > > >>>>>>>>>>>>> years ago but not available yet (
> > > >>>>>>>>>>>>> https://github.com/sdispater/pendulum/pull/563).
> > > >> Since
> > > >>> we
> > > >>>>>> use
> > > >>>>>>>>> parse
> > > >>>>>>>>>>>> dates
> > > >>>>>>>>>>>>> in airflow codebase: datetime parameters and datetime
> > > >> in
> > > >>>>> logs
> > > >>>>>>>> this
> > > >>>>>>>>>> one
> > > >>>>>>>>>>>>> could be a reason for memory leakage in Airflow:
> > > >>>>>>>>>>>>> - https://github.com/apache/airflow/discussions/24694
> > > >>>>>>>>>>>>> - https://github.com/apache/airflow/discussions/28597
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> *Incorrect time zones*, known issues and should be
> > > >>> already
> > > >>>>>>> fixed
> > > >>>>>>>> in
> > > >>>>>>>>>>>>> master branch
> > > >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/700,
> > > >>>>> Mexico
> > > >>>>>> do
> > > >>>>>>>> not
> > > >>>>>>>>>> use
> > > >>>>>>>>>>>> DST
> > > >>>>>>>>>>>>> anymore
> > > >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/706,
> > > >>> Egypt
> > > >>>>>>>>> reinstate
> > > >>>>>>>>>>> DST
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> We add clarification in
> > > >>>>>>>>> https://github.com/apache/airflow/pull/30467
> > > >>>>>>>>>> ,
> > > >>>>>>>>>>>>> however it seems like there is no other way rather
> > > >> than
> > > >>>>>>> patching
> > > >>>>>>>>>>> Pendulum
> > > >>>>>>>>>>>>> right now.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> All these issues should be solved as soon as pendulum
> > > >> 3
> > > >>> is
> > > >>>>>>>>> released.
> > > >>>>>>>>>>> The
> > > >>>>>>>>>>>>> current announced estimation is end of september/
> > > >>>>> beginning
> > > >>>>>> of
> > > >>>>>>>>>> October:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>>
> > > >>>
> > >
> https://github.com/sdispater/pendulum/issues/600#issuecomment-1711299677
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> So in theory we would have a fixed version of pendulum
> > > >>>>> soon,
> > > >>>>>>> and
> > > >>>>>>>> it
> > > >>>>>>>>>>> might
> > > >>>>>>>>>>>>> break something in Airflow but from my point of view
> > > >> it
> > > >>> is
> > > >>>>>>> better
> > > >>>>>>>>>> than
> > > >>>>>>>>>>>>> current status.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> However there might be a situation where the release
> > > >> of
> > > >>>>> the
> > > >>>>>>>>> pendulum
> > > >>>>>>>>>>>> would
> > > >>>>>>>>>>>>> be postponed, so maybe better to have a backup plan.
> > > >>> What
> > > >>>>>> could
> > > >>>>>>>> we
> > > >>>>>>>>> do
> > > >>>>>>>>>>> in
> > > >>>>>>>>>>>>> this case?
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Maybe we should start to use zoneinfo.ZoneInfo instead
> > > >>> of
> > > >>>>>>>> pendulum
> > > >>>>>>>>>>>>> datetime?
> > > >>> https://github.com/apache/airflow/issues/19450
> > > >>>>>>>>>>>>> Pros:
> > > >>>>>>>>>>>>> - stdlib (python 3.9+)
> > > >>>>>>>>>>>>> - In pendulum 3.0 Timezone based on zoneinfo.Zoneinfo
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Cons:
> > > >>>>>>>>>>>>> - Current serialization model can't deal with backport
> > > >>>>>>> packages.
> > > >>>>>>>>> E.g.
> > > >>>>>>>>>>>>> timezone which are serialized in backport_zoneinfo
> > > >> can't
> > > >>>>> be
> > > >>>>>>>>>>> deserialized
> > > >>>>>>>>>>>> in
> > > >>>>>>>>>>>>> zoneinfo
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Maybe we should replace parse datetime with another
> > > >>>>> solution.
> > > >>>>>>>> Does
> > > >>>>>>>>>>> anyone
> > > >>>>>>>>>>>>> know a good replacement?
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Maybe someone from Airflow Community could propose
> > > >> their
> > > >>>>> help
> > > >>>>>>>> with
> > > >>>>>>>>>>>>> maintenance of library:
> > > >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/590
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Maybe we should get rid of the pendulum at all, as a
> > > >>> last
> > > >>>>>>> resort
> > > >>>>>>>>>>>> solution.
> > > >>>>>>>>>>>>> I can't imagine how we could do that, because a lot of
> > > >>>>> stuff
> > > >>>>>>>>> depends
> > > >>>>>>>>>> on
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>> pendulum and removing it would be a breaking change.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> ----
> > > >>>>>>>>>>>>> Best Wishes
> > > >>>>>>>>>>>>> *Andrey Anshin*
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> --
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> --
> > > >>>>>>>>>>> Bolke de Bruin
> > > >>>>>>>>>>> bdbruin@gmail.com
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> > >
> >
>
--
--
Bolke de Bruin
bdbruin@gmail.com
Re: [DISCUSS] Future of Pendulum in Airflow
Posted by Aritra Basu <ar...@gmail.com>.
I agree with wanting to stay away from having to deal with dst but I am on
the side of biting the bullet and transitioning to datetime and testing
thoroughly, especially considering our current predicament with pendulum
and the potential of us being in the same place again potentially in the
future.
--
Regards,
Aritra Basu
On Fri, Dec 1, 2023, 10:27 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Yeah. Agree it's a challenge but I think it's also an opportunity to
> fix those - because whether we admit it, or not - we already have those
> problems - regardless of Pendulum's use.
>
> If you look at the issues (especially ones created around DST time) - there
> are always one or two issues (even in latest versions of Airflow) where DST
> transition is causing people problems. And they are rarely
> solved/addressed.
>
> Example from last October:
>
> https://github.com/apache/airflow/issues/35272
>
> Example from May 2022:
>
> https://github.com/apache/airflow/issues/23400
>
> And Example from March 2020 ( which might or might not be fixed by the open
> PR):
>
> https://github.com/apache/airflow/issues/7999
>
> J.
>
>
> On Fri, Dec 1, 2023 at 5:41 PM Bolke de Bruin <bd...@gmail.com> wrote:
>
> > The alternative is to vendor in or fork pendulum as part of the Airflow
> > project. Also a major undertaking but maybe less effort.
> >
> > We shouldn't just address this as a technical challenge. The question
> > becomes: does that combination fornally do DST transitions correctly,
> does
> > it do leap seconds correctly, does it add dates correctly, does it do
> > timezone traversal correctly and many many other things.
> >
> > So if we switch to the datetime + zone info + timezone we need to have a
> > test suite that verifies this. Otherwise we potentially expose our users
> to
> > different behavior IN their data. Not just a schedule by in their SQL or
> in
> > some scripting where logical date / execution date is used or where date
> > arithmetic is applied.
> >
> > So it is not changing one car for another. It is changing the means of
> > transportation and not knowing if you can get to all the places you want
> /
> > need to go as before.
> >
> > Honestly, I'm a bit scared (I do see the lack of responsiveness). I hope
> > someone can take that away.
> >
> > Bolke
> >
> >
> > Sent from my iPhone
> >
> > > On 1 Dec 2023, at 13:39, Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > Agree. I think this is the best way forward.- we have to bite the
> > bullet of
> > > potential backwards compatibility issues and separating out a compat
> > layer
> > > where we could optionally use pendulum instead would be the most "easy"
> > > approach for our users. Most of them would not even notice if we do a
> > good
> > > job, but those who somehow depend on pendulum objects would get a
> > > possibility to switch back.
> > >
> > >> On Fri, Dec 1, 2023 at 1:09 PM Andrey Anshin <
> andrey.anshin@taragol.is>
> > >> wrote:
> > >>
> > >> IMHO, if we would like to replace pendulum to alternatives, the better
> > >> alternative would be combination of datetime.datetime + zoneinfo +
> > >> datetime.timezone
> > >>
> > >> We still do hacks around pendulum functional, which makes hacks around
> > >> datetime.datetime, so in my perspective better remove redundant layers
> > and
> > >> use native objects directly.
> > >>
> > >> What confuses me the most is how we get around type changes in Tasks
> > >> Context. All dates return as pendulum.DateTime for a very long time
> > >> (literally for ages) and it is probably could be classified as
> breaking
> > >> changes, because it might break quite a bit users pipelines. It still
> > could
> > >> be resolved as optional pendulum dependency + config option for
> datetime
> > >> types in context, which might help to get familiar types for a while.
> > >>
> > >>
> > >>> On Wed, 29 Nov 2023 at 05:58, Jarek Potiuk <ja...@potiuk.com> wrote:
> > >>>
> > >>> "infamous fork of " should be "fork of infamous Akka" - just to be
> > clear
> > >> :)
> > >>>
> > >>>> On Wed, Nov 29, 2023 at 2:57 AM Jarek Potiuk <ja...@potiuk.com>
> > wrote:
> > >>>
> > >>>> It's a very rare occurrence that ASF accepts forks. Usually it has
> to
> > >> be
> > >>>> willingly donated by those who own the IP rights, - mainly because
> of
> > >>>> trademark issues. Licence is one thing, but carrying things like the
> > >> name
> > >>>> of the project is not possible without clearing IP and donating the
> > >>>> project.
> > >>>>
> > >>>> It happened last time last year - with Pekko project - infamous fork
> > or
> > >>>> Akka after Lightbend changed licensing for Akka -
> > >>>>
> > >>
> https://www.lightbend.com/blog/why-we-are-changing-the-license-for-akka
> > >>>> making it impossible to be used in ASF projects with the new
> licence.
> > >> But
> > >>>> it was a really "strong" case and the "we do not accept forks
> usually"
> > >>> was
> > >>>> quite a contentious issue (I followed the discussion - it was
> > >> fascinating
> > >>>> actually). It tool a LOT of time and effort for those who led the
> > >> effort
> > >>>> eventually - and also mostly because things had to be crystal-clear
> > and
> > >>> not
> > >>>> "litigable" because the other side had some business reasons to make
> > >> the
> > >>>> licensing changes and they did not understood why it is impossible
> to
> > >> use
> > >>>> Akka as-is even with all the exclusions they offered to ASF - and
> > >>>> effectively it was a "hostile" move.
> > >>>>
> > >>>> I don't think it's "worth" it in this case to be honest - especially
> > if
> > >>>> the maintainers are effectively "silent" and seem to ignore problems
> > of
> > >>> one
> > >>>> of the really serious users they have. If they were willing to
> > >> cooperate,
> > >>>> we could do a lot, but when we are left alone, I think it will be
> far
> > >>> more
> > >>>> convenient for us to look for alternatives - to be honest.
> > >>>>
> > >>>> J.
> > >>>>
> > >>>>
> > >>>> On Wed, Nov 29, 2023 at 2:46 AM Austin Bennett <au...@apache.org>
> > >>> wrote:
> > >>>>
> > >>>>> An option would be to fork Pendulum? It is MIT Licensed, I don't
> > know
> > >>>>> whether that poses problems to get in ASF?
> > >>>>>
> > >>>>> If forking (?) which is somewhat non-ideal, would we want that 'in'
> > >>>>> airflow?
> > >>>>>
> > >>>>> If not 'in' airflow, I wonder if ASF incubator would accept a
> forked
> > >>>>> project? [ anecdotally Linux Foundation has incubated Starrocks
> > which
> > >>> is
> > >>>>> a
> > >>>>> fork of Apache Doris, so there's precedence for this to be OK in
> the
> > >>> wider
> > >>>>> OSS ecosystem ].
> > >>>>>
> > >>>>>
> > >>>>> On Tue, Nov 28, 2023 at 6:41 PM Jarek Potiuk <ja...@potiuk.com>
> > >> wrote:
> > >>>>>
> > >>>>>> I have not heard back from maintainers, just a comment from
> someone
> > >>> else
> > >>>>>> who suggested donating a pendulum to the ASF (which is kinda
> > >>> interesting
> > >>>>>> idea). I followed up, let's see. I think if we do not hear back
> and
> > >>>>> there
> > >>>>>> will be another week or two where "Pendulum 3 supporting 3.12
> > >> support
> > >>>>> "is
> > >>>>>> coming" but no-one knows when, effectively means that we should
> look
> > >>>>> for an
> > >>>>>> alternative like NOW.
> > >>>>>>
> > >>>>>> J.
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> On Wed, Nov 22, 2023 at 3:27 AM Daniel Standish
> > >>>>>> <da...@astronomer.io.invalid> wrote:
> > >>>>>>
> > >>>>>>> Thanks Jarek
> > >>>>>>>
> > >>>>>>> On Tue, Nov 21, 2023, 4:34 PM Jarek Potiuk <ja...@potiuk.com>
> > >>> wrote:
> > >>>>>>>
> > >>>>>>>> I think we miss important insight - straight from the source.
> > >>>>>>>>
> > >>>>>>>> I believe it's time to be candid and simply ask questions for
> > >> the
> > >>>>>> future
> > >>>>>>> of
> > >>>>>>>> Pendulum directly where we should - ie. we should just ask
> > >>>>>> maintainers.
> > >>>>>>>>
> > >>>>>>>> I've just started a very candid and open discussion there -
> > >>>>>>>> https://github.com/sdispater/pendulum/discussions/771 .
> > >>>>>>>>
> > >>>>>>>> I linked back to this discussion and explained where we are
> > >> coming
> > >>>>>> from.
> > >>>>>>> I
> > >>>>>>>> even offered an option, that maybe - if they accept some help
> > >> and
> > >>>>> would
> > >>>>>>> be
> > >>>>>>>> open to make a few of us maintainers of Pendulum, that I will
> > >> ask
> > >>> if
> > >>>>>> some
> > >>>>>>>> of the maintainers here in Airflow would like to step up (I
> > >> would
> > >>> be
> > >>>>>>>> willing to - for one) and help to at least move Pendulum through
> > >>>>> 3.12
> > >>>>>> and
> > >>>>>>>> maybe keep it running for as long as it will be needed to
> > >> maintain
> > >>>>> it.
> > >>>>>>>>
> > >>>>>>>> I think if they would be open to some of us helping them - this
> > >>>>> might
> > >>>>>> be
> > >>>>>>>> actually the "cheapest" option for us to be honest at least in a
> > >>>>>> mid-term
> > >>>>>>>> if we could gain influence and even take part and speed up
> > >>> releases
> > >>>>> and
> > >>>>>>>> merging of issues that are blocking us (or would be blocking us
> > >> in
> > >>>>> the
> > >>>>>>>> future).
> > >>>>>>>>
> > >>>>>>>> I tried to be very friendly but candid and direct on what kind
> > >> of
> > >>>>>>> problems
> > >>>>>>>> it creates us and also expressed our thanks for supporting us
> > >> for
> > >>> so
> > >>>>>> long
> > >>>>>>>> (I do believe they deserve it). I think the sole (pretty much)
> > >>>>>>> maintainer
> > >>>>>>>> of Pendulum might really have a hard time maintaining and
> > >> keeping
> > >>>>> it up
> > >>>>>>> for
> > >>>>>>>> years - without being paid for and thanked. I hope - maybe all
> > >>> that
> > >>>>>> they
> > >>>>>>>> need are a few words of encouragement and thanks and realising
> > >>> that
> > >>>>>>> others
> > >>>>>>>> depend on their work - but also seeing that they are understood,
> > >>>>>>> empathised
> > >>>>>>>> with and that there are viable alternatives they might follow
> > >>> might
> > >>>>> be
> > >>>>>>>> helpful for them to be also candid and respond. I hope my
> > >>> intentions
> > >>>>>> will
> > >>>>>>>> be understood - that I am not complaining, but even trying to
> > >>> help.
> > >>>>>>>>
> > >>>>>>>> Let\s see where it gets us.
> > >>>>>>>>
> > >>>>>>>> J.
> > >>>>>>>>
> > >>>>>>>> On Fri, Nov 17, 2023 at 5:32 PM Andrey Anshin <
> > >>>>>> andrey.anshin@taragol.is>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Just to clarify I'd like us to consider the possibility that
> > >> no
> > >>>>> new
> > >>>>>>>>> pendulum would be released or released at the end of 2024,
> > >> like
> > >>> a
> > >>>>>>>>> pessimistic scenario:
> > >>>>>>>>> - What should we do in this case?
> > >>>>>>>>> - Work out a backup plan.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> ----
> > >>>>>>>>> Best Wishes
> > >>>>>>>>> *Andrey Anshin*
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On Fri, 17 Nov 2023 at 16:33, Jarek Potiuk <ja...@potiuk.com>
> > >>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Also I think TP - had a document in the past (years ago)
> > >>>>>>> describing a
> > >>>>>>>>>> draft of a more complete alternative we can take to approach
> > >>>>>> datetime
> > >>>>>>>> vs.
> > >>>>>>>>>> pendulum dichotomy. I cannot easily find the document and
> > >>>>>> discussion
> > >>>>>>> -
> > >>>>>>>>> but
> > >>>>>>>>>> I do remember it was proposing some interesting changes in
> > >> the
> > >>>>>>> approach
> > >>>>>>>>> of
> > >>>>>>>>>> Airflow to have an abstraction layer on top of it (as far
> > >> as I
> > >>>>>>>> remember).
> > >>>>>>>>>> Maybe we can resurrect that idea if TP might find the
> > >>> proposal ?
> > >>>>>>>>>>
> > >>>>>>>>>> On Fri, Nov 17, 2023 at 1:06 PM Bolke de Bruin <
> > >>>>> bdbruin@gmail.com>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hi,
> > >>>>>>>>>>>
> > >>>>>>>>>>> I agree that the current speed of development of Pendulum
> > >>>>> leaves
> > >>>>>>>>>> something
> > >>>>>>>>>>> to be desired. However, I think we should not
> > >> underestimate
> > >>>>> the
> > >>>>>>>> effort
> > >>>>>>>>> of
> > >>>>>>>>>>> replacing it. It is not just a matter of
> > >>>>> %s/pendulum/datetime/g
> > >>>>>> so
> > >>>>>>> to
> > >>>>>>>>>> say.
> > >>>>>>>>>>> If we are *truly* thinking about moving to native
> > >> datetime /
> > >>>>>>> zoneinfo
> > >>>>>>>>> etc
> > >>>>>>>>>>> we need *extensive* tests, basically copying what pendulum
> > >>>>> does
> > >>>>>> to
> > >>>>>>>>> check
> > >>>>>>>>>>> its behavior. The reason is that native implementations in
> > >>> the
> > >>>>>> past
> > >>>>>>>>> made
> > >>>>>>>>>>> serious mistakes and I do not put a lot of trust in them.
> > >>>>>>>>>>>
> > >>>>>>>>>>> An abstraction or vendoring in could be alternatives, but
> > >>> they
> > >>>>>>> bring
> > >>>>>>>>>> their
> > >>>>>>>>>>> own significant challenges.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I have re-raised the issue here:
> > >>>>>>>>>>>
> > >>>>>>>>>>> https://github.com/sdispater/pendulum/issues/753
> > >>>>>>>>>>>
> > >>>>>>>>>>> The upside is that it seems the amount of issues with the
> > >>>>> beta is
> > >>>>>>>>>> limited,
> > >>>>>>>>>>> so hopefully the maintainers can spend a couple of cycles
> > >> to
> > >>>>>>> address
> > >>>>>>>>>> them.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Bolke
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Fri, 17 Nov 2023 at 10:28, Andrey Anshin <
> > >>>>>>>> andrey.anshin@taragol.is>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> There is no changes in stable pendulum so let's try to
> > >>>>> continue
> > >>>>>>>> this
> > >>>>>>>>>>>> discussion and start think about "Plan B"
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Just a reminder:
> > >>>>>>>>>>>> - pendulum 2.1.2 released 3 years ago (at the time
> > >> Airflow
> > >>>>>>> 1.10.x)
> > >>>>>>>>>>>> - pendulum 2 doesn't work well in Python 3.12, this is a
> > >>>>>>>> showstopper
> > >>>>>>>>>> for
> > >>>>>>>>>>>> the support Python 3.12
> > >>>>>>>>>>>> - pendulum 2 have memory leaks (
> > >>>>>>>>>>>> https://github.com/sdispater/pendulum/issues/720), and
> > >>>>> Airflow
> > >>>>>>> use
> > >>>>>>>>>>>> approach
> > >>>>>>>>>>>> to achieve this leaks, especially in K8S executor
> > >>>>>>>>>>>> - pendulum 2 doesn't use system tzdata by default, but
> > >> we
> > >>>>> have
> > >>>>>> a
> > >>>>>>>>>>>> workaround, thanks Bolke for the documentation
> > >>>>>>>>>>>> - pendulum it is a core dependency of Airflow, and I
> > >> guess
> > >>>>> we
> > >>>>>>> can't
> > >>>>>>>>>>>> remove/replace it without breaking changes.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> So my proposal if things won't change in the near future
> > >>>>> then
> > >>>>>> we
> > >>>>>>>> need
> > >>>>>>>>>> to
> > >>>>>>>>>>>> start removing the pendulum from the core and replace it
> > >>>>> with
> > >>>>>> the
> > >>>>>>>>>> native
> > >>>>>>>>>>>> datetime / zoneinfo / tzinfo. But maybe we have another
> > >>> way
> > >>>>> to
> > >>>>>>>>> resolve
> > >>>>>>>>>>> this
> > >>>>>>>>>>>> without breaking changes? Because for me it would be a
> > >>>>> little
> > >>>>>>> weird
> > >>>>>>>>> if
> > >>>>>>>>>>>> removal pendulum would be a main driver for release
> > >>> Airflow
> > >>>>> 3
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> ----
> > >>>>>>>>>>>> Best Wishes
> > >>>>>>>>>>>> *Andrey Anshin*
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Thu, 28 Sept 2023 at 13:01, Andrey Anshin <
> > >>>>>>>>> andrey.anshin@taragol.is
> > >>>>>>>>>>>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> This discussion is more about the known problem of
> > >>>>> pendulum
> > >>>>>> and
> > >>>>>>>> how
> > >>>>>>>>>> we
> > >>>>>>>>>>>>> could deal with it and maybe how we (as Community)
> > >> might
> > >>>>> help
> > >>>>>>>>> autor.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> The library is mostly supported by a single author
> > >>>>> Sébastien
> > >>>>>>>>> Eustace
> > >>>>>>>>>> (
> > >>>>>>>>>>>>> https://github.com/sdispater) and it seems like we
> > >> bump
> > >>>>> into
> > >>>>>>> the
> > >>>>>>>>>>>>> situation which is described in xkcd #2347 (
> > >>>>>>>>>>>>> https://imgs.xkcd.com/comics/dependency.png). To be
> > >>>>> honest
> > >>>>>> it
> > >>>>>>> is
> > >>>>>>>>> not
> > >>>>>>>>>>>>> something new when library mainly supported by one
> > >>> author
> > >>>>> so
> > >>>>>>>> there
> > >>>>>>>>> is
> > >>>>>>>>>>>>> always a risk that the library will no longer be
> > >>>>> supported /
> > >>>>>>>>>> abandoned
> > >>>>>>>>>>>>> And if takes in account that pendulum provides core
> > >>>>>>> functionality
> > >>>>>>>>> in
> > >>>>>>>>>>>>> Airflow it could have dramatical impact in the future.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Pendulum is a really nice library which helps a lot of
> > >>>>>>> developers
> > >>>>>>>>> to
> > >>>>>>>>>>> work
> > >>>>>>>>>>>>> with dates/datetimes. However there is one major
> > >>> problem,
> > >>>>> the
> > >>>>>>>> last
> > >>>>>>>>>>>> release
> > >>>>>>>>>>>>> of this library happened more than 3 years ago (
> > >>>>>>>>>>>>> https://pypi.org/project/pendulum/#history) in the
> > >> time
> > >>>>> when
> > >>>>>>>>> Airflow
> > >>>>>>>>>>>>> 1.10.11 was released
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Fortunately, the project is not abandoned and on a
> > >>> regular
> > >>>>>>> basis
> > >>>>>>>>>>> commits
> > >>>>>>>>>>>>> add into the master branch. However these commits are
> > >>> not
> > >>>>>>>> included
> > >>>>>>>>>> into
> > >>>>>>>>>>>> any
> > >>>>>>>>>>>>> final release and that's why some things related to
> > >>>>> datetime
> > >>>>>>>> don't
> > >>>>>>>>>> work
> > >>>>>>>>>>>> as
> > >>>>>>>>>>>>> expected in Airflow. There are list of known (for me)
> > >>>>> issues
> > >>>>>>>> which
> > >>>>>>>>>> are
> > >>>>>>>>>>>>> affect Airflow
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> *Memory Leak on parse*:
> > >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/720,
> > >>> this
> > >>>>> one
> > >>>>>>>> fixed
> > >>>>>>>>>> 2
> > >>>>>>>>>>>>> years ago but not available yet (
> > >>>>>>>>>>>>> https://github.com/sdispater/pendulum/pull/563).
> > >> Since
> > >>> we
> > >>>>>> use
> > >>>>>>>>> parse
> > >>>>>>>>>>>> dates
> > >>>>>>>>>>>>> in airflow codebase: datetime parameters and datetime
> > >> in
> > >>>>> logs
> > >>>>>>>> this
> > >>>>>>>>>> one
> > >>>>>>>>>>>>> could be a reason for memory leakage in Airflow:
> > >>>>>>>>>>>>> - https://github.com/apache/airflow/discussions/24694
> > >>>>>>>>>>>>> - https://github.com/apache/airflow/discussions/28597
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> *Incorrect time zones*, known issues and should be
> > >>> already
> > >>>>>>> fixed
> > >>>>>>>> in
> > >>>>>>>>>>>>> master branch
> > >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/700,
> > >>>>> Mexico
> > >>>>>> do
> > >>>>>>>> not
> > >>>>>>>>>> use
> > >>>>>>>>>>>> DST
> > >>>>>>>>>>>>> anymore
> > >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/706,
> > >>> Egypt
> > >>>>>>>>> reinstate
> > >>>>>>>>>>> DST
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> We add clarification in
> > >>>>>>>>> https://github.com/apache/airflow/pull/30467
> > >>>>>>>>>> ,
> > >>>>>>>>>>>>> however it seems like there is no other way rather
> > >> than
> > >>>>>>> patching
> > >>>>>>>>>>> Pendulum
> > >>>>>>>>>>>>> right now.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> All these issues should be solved as soon as pendulum
> > >> 3
> > >>> is
> > >>>>>>>>> released.
> > >>>>>>>>>>> The
> > >>>>>>>>>>>>> current announced estimation is end of september/
> > >>>>> beginning
> > >>>>>> of
> > >>>>>>>>>> October:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>
> > https://github.com/sdispater/pendulum/issues/600#issuecomment-1711299677
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> So in theory we would have a fixed version of pendulum
> > >>>>> soon,
> > >>>>>>> and
> > >>>>>>>> it
> > >>>>>>>>>>> might
> > >>>>>>>>>>>>> break something in Airflow but from my point of view
> > >> it
> > >>> is
> > >>>>>>> better
> > >>>>>>>>>> than
> > >>>>>>>>>>>>> current status.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> However there might be a situation where the release
> > >> of
> > >>>>> the
> > >>>>>>>>> pendulum
> > >>>>>>>>>>>> would
> > >>>>>>>>>>>>> be postponed, so maybe better to have a backup plan.
> > >>> What
> > >>>>>> could
> > >>>>>>>> we
> > >>>>>>>>> do
> > >>>>>>>>>>> in
> > >>>>>>>>>>>>> this case?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Maybe we should start to use zoneinfo.ZoneInfo instead
> > >>> of
> > >>>>>>>> pendulum
> > >>>>>>>>>>>>> datetime?
> > >>> https://github.com/apache/airflow/issues/19450
> > >>>>>>>>>>>>> Pros:
> > >>>>>>>>>>>>> - stdlib (python 3.9+)
> > >>>>>>>>>>>>> - In pendulum 3.0 Timezone based on zoneinfo.Zoneinfo
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Cons:
> > >>>>>>>>>>>>> - Current serialization model can't deal with backport
> > >>>>>>> packages.
> > >>>>>>>>> E.g.
> > >>>>>>>>>>>>> timezone which are serialized in backport_zoneinfo
> > >> can't
> > >>>>> be
> > >>>>>>>>>>> deserialized
> > >>>>>>>>>>>> in
> > >>>>>>>>>>>>> zoneinfo
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Maybe we should replace parse datetime with another
> > >>>>> solution.
> > >>>>>>>> Does
> > >>>>>>>>>>> anyone
> > >>>>>>>>>>>>> know a good replacement?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Maybe someone from Airflow Community could propose
> > >> their
> > >>>>> help
> > >>>>>>>> with
> > >>>>>>>>>>>>> maintenance of library:
> > >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/590
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Maybe we should get rid of the pendulum at all, as a
> > >>> last
> > >>>>>>> resort
> > >>>>>>>>>>>> solution.
> > >>>>>>>>>>>>> I can't imagine how we could do that, because a lot of
> > >>>>> stuff
> > >>>>>>>>> depends
> > >>>>>>>>>> on
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>> pendulum and removing it would be a breaking change.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> ----
> > >>>>>>>>>>>>> Best Wishes
> > >>>>>>>>>>>>> *Andrey Anshin*
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> --
> > >>>>>>>>>>>
> > >>>>>>>>>>> --
> > >>>>>>>>>>> Bolke de Bruin
> > >>>>>>>>>>> bdbruin@gmail.com
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>
Re: [DISCUSS] Future of Pendulum in Airflow
Posted by Jarek Potiuk <ja...@potiuk.com>.
Yeah. Agree it's a challenge but I think it's also an opportunity to
fix those - because whether we admit it, or not - we already have those
problems - regardless of Pendulum's use.
If you look at the issues (especially ones created around DST time) - there
are always one or two issues (even in latest versions of Airflow) where DST
transition is causing people problems. And they are rarely solved/addressed.
Example from last October:
https://github.com/apache/airflow/issues/35272
Example from May 2022:
https://github.com/apache/airflow/issues/23400
And Example from March 2020 ( which might or might not be fixed by the open
PR):
https://github.com/apache/airflow/issues/7999
J.
On Fri, Dec 1, 2023 at 5:41 PM Bolke de Bruin <bd...@gmail.com> wrote:
> The alternative is to vendor in or fork pendulum as part of the Airflow
> project. Also a major undertaking but maybe less effort.
>
> We shouldn't just address this as a technical challenge. The question
> becomes: does that combination fornally do DST transitions correctly, does
> it do leap seconds correctly, does it add dates correctly, does it do
> timezone traversal correctly and many many other things.
>
> So if we switch to the datetime + zone info + timezone we need to have a
> test suite that verifies this. Otherwise we potentially expose our users to
> different behavior IN their data. Not just a schedule by in their SQL or in
> some scripting where logical date / execution date is used or where date
> arithmetic is applied.
>
> So it is not changing one car for another. It is changing the means of
> transportation and not knowing if you can get to all the places you want /
> need to go as before.
>
> Honestly, I'm a bit scared (I do see the lack of responsiveness). I hope
> someone can take that away.
>
> Bolke
>
>
> Sent from my iPhone
>
> > On 1 Dec 2023, at 13:39, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > Agree. I think this is the best way forward.- we have to bite the
> bullet of
> > potential backwards compatibility issues and separating out a compat
> layer
> > where we could optionally use pendulum instead would be the most "easy"
> > approach for our users. Most of them would not even notice if we do a
> good
> > job, but those who somehow depend on pendulum objects would get a
> > possibility to switch back.
> >
> >> On Fri, Dec 1, 2023 at 1:09 PM Andrey Anshin <an...@taragol.is>
> >> wrote:
> >>
> >> IMHO, if we would like to replace pendulum to alternatives, the better
> >> alternative would be combination of datetime.datetime + zoneinfo +
> >> datetime.timezone
> >>
> >> We still do hacks around pendulum functional, which makes hacks around
> >> datetime.datetime, so in my perspective better remove redundant layers
> and
> >> use native objects directly.
> >>
> >> What confuses me the most is how we get around type changes in Tasks
> >> Context. All dates return as pendulum.DateTime for a very long time
> >> (literally for ages) and it is probably could be classified as breaking
> >> changes, because it might break quite a bit users pipelines. It still
> could
> >> be resolved as optional pendulum dependency + config option for datetime
> >> types in context, which might help to get familiar types for a while.
> >>
> >>
> >>> On Wed, 29 Nov 2023 at 05:58, Jarek Potiuk <ja...@potiuk.com> wrote:
> >>>
> >>> "infamous fork of " should be "fork of infamous Akka" - just to be
> clear
> >> :)
> >>>
> >>>> On Wed, Nov 29, 2023 at 2:57 AM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> >>>
> >>>> It's a very rare occurrence that ASF accepts forks. Usually it has to
> >> be
> >>>> willingly donated by those who own the IP rights, - mainly because of
> >>>> trademark issues. Licence is one thing, but carrying things like the
> >> name
> >>>> of the project is not possible without clearing IP and donating the
> >>>> project.
> >>>>
> >>>> It happened last time last year - with Pekko project - infamous fork
> or
> >>>> Akka after Lightbend changed licensing for Akka -
> >>>>
> >> https://www.lightbend.com/blog/why-we-are-changing-the-license-for-akka
> >>>> making it impossible to be used in ASF projects with the new licence.
> >> But
> >>>> it was a really "strong" case and the "we do not accept forks usually"
> >>> was
> >>>> quite a contentious issue (I followed the discussion - it was
> >> fascinating
> >>>> actually). It tool a LOT of time and effort for those who led the
> >> effort
> >>>> eventually - and also mostly because things had to be crystal-clear
> and
> >>> not
> >>>> "litigable" because the other side had some business reasons to make
> >> the
> >>>> licensing changes and they did not understood why it is impossible to
> >> use
> >>>> Akka as-is even with all the exclusions they offered to ASF - and
> >>>> effectively it was a "hostile" move.
> >>>>
> >>>> I don't think it's "worth" it in this case to be honest - especially
> if
> >>>> the maintainers are effectively "silent" and seem to ignore problems
> of
> >>> one
> >>>> of the really serious users they have. If they were willing to
> >> cooperate,
> >>>> we could do a lot, but when we are left alone, I think it will be far
> >>> more
> >>>> convenient for us to look for alternatives - to be honest.
> >>>>
> >>>> J.
> >>>>
> >>>>
> >>>> On Wed, Nov 29, 2023 at 2:46 AM Austin Bennett <au...@apache.org>
> >>> wrote:
> >>>>
> >>>>> An option would be to fork Pendulum? It is MIT Licensed, I don't
> know
> >>>>> whether that poses problems to get in ASF?
> >>>>>
> >>>>> If forking (?) which is somewhat non-ideal, would we want that 'in'
> >>>>> airflow?
> >>>>>
> >>>>> If not 'in' airflow, I wonder if ASF incubator would accept a forked
> >>>>> project? [ anecdotally Linux Foundation has incubated Starrocks
> which
> >>> is
> >>>>> a
> >>>>> fork of Apache Doris, so there's precedence for this to be OK in the
> >>> wider
> >>>>> OSS ecosystem ].
> >>>>>
> >>>>>
> >>>>> On Tue, Nov 28, 2023 at 6:41 PM Jarek Potiuk <ja...@potiuk.com>
> >> wrote:
> >>>>>
> >>>>>> I have not heard back from maintainers, just a comment from someone
> >>> else
> >>>>>> who suggested donating a pendulum to the ASF (which is kinda
> >>> interesting
> >>>>>> idea). I followed up, let's see. I think if we do not hear back and
> >>>>> there
> >>>>>> will be another week or two where "Pendulum 3 supporting 3.12
> >> support
> >>>>> "is
> >>>>>> coming" but no-one knows when, effectively means that we should look
> >>>>> for an
> >>>>>> alternative like NOW.
> >>>>>>
> >>>>>> J.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Nov 22, 2023 at 3:27 AM Daniel Standish
> >>>>>> <da...@astronomer.io.invalid> wrote:
> >>>>>>
> >>>>>>> Thanks Jarek
> >>>>>>>
> >>>>>>> On Tue, Nov 21, 2023, 4:34 PM Jarek Potiuk <ja...@potiuk.com>
> >>> wrote:
> >>>>>>>
> >>>>>>>> I think we miss important insight - straight from the source.
> >>>>>>>>
> >>>>>>>> I believe it's time to be candid and simply ask questions for
> >> the
> >>>>>> future
> >>>>>>> of
> >>>>>>>> Pendulum directly where we should - ie. we should just ask
> >>>>>> maintainers.
> >>>>>>>>
> >>>>>>>> I've just started a very candid and open discussion there -
> >>>>>>>> https://github.com/sdispater/pendulum/discussions/771 .
> >>>>>>>>
> >>>>>>>> I linked back to this discussion and explained where we are
> >> coming
> >>>>>> from.
> >>>>>>> I
> >>>>>>>> even offered an option, that maybe - if they accept some help
> >> and
> >>>>> would
> >>>>>>> be
> >>>>>>>> open to make a few of us maintainers of Pendulum, that I will
> >> ask
> >>> if
> >>>>>> some
> >>>>>>>> of the maintainers here in Airflow would like to step up (I
> >> would
> >>> be
> >>>>>>>> willing to - for one) and help to at least move Pendulum through
> >>>>> 3.12
> >>>>>> and
> >>>>>>>> maybe keep it running for as long as it will be needed to
> >> maintain
> >>>>> it.
> >>>>>>>>
> >>>>>>>> I think if they would be open to some of us helping them - this
> >>>>> might
> >>>>>> be
> >>>>>>>> actually the "cheapest" option for us to be honest at least in a
> >>>>>> mid-term
> >>>>>>>> if we could gain influence and even take part and speed up
> >>> releases
> >>>>> and
> >>>>>>>> merging of issues that are blocking us (or would be blocking us
> >> in
> >>>>> the
> >>>>>>>> future).
> >>>>>>>>
> >>>>>>>> I tried to be very friendly but candid and direct on what kind
> >> of
> >>>>>>> problems
> >>>>>>>> it creates us and also expressed our thanks for supporting us
> >> for
> >>> so
> >>>>>> long
> >>>>>>>> (I do believe they deserve it). I think the sole (pretty much)
> >>>>>>> maintainer
> >>>>>>>> of Pendulum might really have a hard time maintaining and
> >> keeping
> >>>>> it up
> >>>>>>> for
> >>>>>>>> years - without being paid for and thanked. I hope - maybe all
> >>> that
> >>>>>> they
> >>>>>>>> need are a few words of encouragement and thanks and realising
> >>> that
> >>>>>>> others
> >>>>>>>> depend on their work - but also seeing that they are understood,
> >>>>>>> empathised
> >>>>>>>> with and that there are viable alternatives they might follow
> >>> might
> >>>>> be
> >>>>>>>> helpful for them to be also candid and respond. I hope my
> >>> intentions
> >>>>>> will
> >>>>>>>> be understood - that I am not complaining, but even trying to
> >>> help.
> >>>>>>>>
> >>>>>>>> Let\s see where it gets us.
> >>>>>>>>
> >>>>>>>> J.
> >>>>>>>>
> >>>>>>>> On Fri, Nov 17, 2023 at 5:32 PM Andrey Anshin <
> >>>>>> andrey.anshin@taragol.is>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Just to clarify I'd like us to consider the possibility that
> >> no
> >>>>> new
> >>>>>>>>> pendulum would be released or released at the end of 2024,
> >> like
> >>> a
> >>>>>>>>> pessimistic scenario:
> >>>>>>>>> - What should we do in this case?
> >>>>>>>>> - Work out a backup plan.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> ----
> >>>>>>>>> Best Wishes
> >>>>>>>>> *Andrey Anshin*
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Fri, 17 Nov 2023 at 16:33, Jarek Potiuk <ja...@potiuk.com>
> >>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Also I think TP - had a document in the past (years ago)
> >>>>>>> describing a
> >>>>>>>>>> draft of a more complete alternative we can take to approach
> >>>>>> datetime
> >>>>>>>> vs.
> >>>>>>>>>> pendulum dichotomy. I cannot easily find the document and
> >>>>>> discussion
> >>>>>>> -
> >>>>>>>>> but
> >>>>>>>>>> I do remember it was proposing some interesting changes in
> >> the
> >>>>>>> approach
> >>>>>>>>> of
> >>>>>>>>>> Airflow to have an abstraction layer on top of it (as far
> >> as I
> >>>>>>>> remember).
> >>>>>>>>>> Maybe we can resurrect that idea if TP might find the
> >>> proposal ?
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Nov 17, 2023 at 1:06 PM Bolke de Bruin <
> >>>>> bdbruin@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi,
> >>>>>>>>>>>
> >>>>>>>>>>> I agree that the current speed of development of Pendulum
> >>>>> leaves
> >>>>>>>>>> something
> >>>>>>>>>>> to be desired. However, I think we should not
> >> underestimate
> >>>>> the
> >>>>>>>> effort
> >>>>>>>>> of
> >>>>>>>>>>> replacing it. It is not just a matter of
> >>>>> %s/pendulum/datetime/g
> >>>>>> so
> >>>>>>> to
> >>>>>>>>>> say.
> >>>>>>>>>>> If we are *truly* thinking about moving to native
> >> datetime /
> >>>>>>> zoneinfo
> >>>>>>>>> etc
> >>>>>>>>>>> we need *extensive* tests, basically copying what pendulum
> >>>>> does
> >>>>>> to
> >>>>>>>>> check
> >>>>>>>>>>> its behavior. The reason is that native implementations in
> >>> the
> >>>>>> past
> >>>>>>>>> made
> >>>>>>>>>>> serious mistakes and I do not put a lot of trust in them.
> >>>>>>>>>>>
> >>>>>>>>>>> An abstraction or vendoring in could be alternatives, but
> >>> they
> >>>>>>> bring
> >>>>>>>>>> their
> >>>>>>>>>>> own significant challenges.
> >>>>>>>>>>>
> >>>>>>>>>>> I have re-raised the issue here:
> >>>>>>>>>>>
> >>>>>>>>>>> https://github.com/sdispater/pendulum/issues/753
> >>>>>>>>>>>
> >>>>>>>>>>> The upside is that it seems the amount of issues with the
> >>>>> beta is
> >>>>>>>>>> limited,
> >>>>>>>>>>> so hopefully the maintainers can spend a couple of cycles
> >> to
> >>>>>>> address
> >>>>>>>>>> them.
> >>>>>>>>>>>
> >>>>>>>>>>> Bolke
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, 17 Nov 2023 at 10:28, Andrey Anshin <
> >>>>>>>> andrey.anshin@taragol.is>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> There is no changes in stable pendulum so let's try to
> >>>>> continue
> >>>>>>>> this
> >>>>>>>>>>>> discussion and start think about "Plan B"
> >>>>>>>>>>>>
> >>>>>>>>>>>> Just a reminder:
> >>>>>>>>>>>> - pendulum 2.1.2 released 3 years ago (at the time
> >> Airflow
> >>>>>>> 1.10.x)
> >>>>>>>>>>>> - pendulum 2 doesn't work well in Python 3.12, this is a
> >>>>>>>> showstopper
> >>>>>>>>>> for
> >>>>>>>>>>>> the support Python 3.12
> >>>>>>>>>>>> - pendulum 2 have memory leaks (
> >>>>>>>>>>>> https://github.com/sdispater/pendulum/issues/720), and
> >>>>> Airflow
> >>>>>>> use
> >>>>>>>>>>>> approach
> >>>>>>>>>>>> to achieve this leaks, especially in K8S executor
> >>>>>>>>>>>> - pendulum 2 doesn't use system tzdata by default, but
> >> we
> >>>>> have
> >>>>>> a
> >>>>>>>>>>>> workaround, thanks Bolke for the documentation
> >>>>>>>>>>>> - pendulum it is a core dependency of Airflow, and I
> >> guess
> >>>>> we
> >>>>>>> can't
> >>>>>>>>>>>> remove/replace it without breaking changes.
> >>>>>>>>>>>>
> >>>>>>>>>>>> So my proposal if things won't change in the near future
> >>>>> then
> >>>>>> we
> >>>>>>>> need
> >>>>>>>>>> to
> >>>>>>>>>>>> start removing the pendulum from the core and replace it
> >>>>> with
> >>>>>> the
> >>>>>>>>>> native
> >>>>>>>>>>>> datetime / zoneinfo / tzinfo. But maybe we have another
> >>> way
> >>>>> to
> >>>>>>>>> resolve
> >>>>>>>>>>> this
> >>>>>>>>>>>> without breaking changes? Because for me it would be a
> >>>>> little
> >>>>>>> weird
> >>>>>>>>> if
> >>>>>>>>>>>> removal pendulum would be a main driver for release
> >>> Airflow
> >>>>> 3
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> ----
> >>>>>>>>>>>> Best Wishes
> >>>>>>>>>>>> *Andrey Anshin*
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, 28 Sept 2023 at 13:01, Andrey Anshin <
> >>>>>>>>> andrey.anshin@taragol.is
> >>>>>>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> This discussion is more about the known problem of
> >>>>> pendulum
> >>>>>> and
> >>>>>>>> how
> >>>>>>>>>> we
> >>>>>>>>>>>>> could deal with it and maybe how we (as Community)
> >> might
> >>>>> help
> >>>>>>>>> autor.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> The library is mostly supported by a single author
> >>>>> Sébastien
> >>>>>>>>> Eustace
> >>>>>>>>>> (
> >>>>>>>>>>>>> https://github.com/sdispater) and it seems like we
> >> bump
> >>>>> into
> >>>>>>> the
> >>>>>>>>>>>>> situation which is described in xkcd #2347 (
> >>>>>>>>>>>>> https://imgs.xkcd.com/comics/dependency.png). To be
> >>>>> honest
> >>>>>> it
> >>>>>>> is
> >>>>>>>>> not
> >>>>>>>>>>>>> something new when library mainly supported by one
> >>> author
> >>>>> so
> >>>>>>>> there
> >>>>>>>>> is
> >>>>>>>>>>>>> always a risk that the library will no longer be
> >>>>> supported /
> >>>>>>>>>> abandoned
> >>>>>>>>>>>>> And if takes in account that pendulum provides core
> >>>>>>> functionality
> >>>>>>>>> in
> >>>>>>>>>>>>> Airflow it could have dramatical impact in the future.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Pendulum is a really nice library which helps a lot of
> >>>>>>> developers
> >>>>>>>>> to
> >>>>>>>>>>> work
> >>>>>>>>>>>>> with dates/datetimes. However there is one major
> >>> problem,
> >>>>> the
> >>>>>>>> last
> >>>>>>>>>>>> release
> >>>>>>>>>>>>> of this library happened more than 3 years ago (
> >>>>>>>>>>>>> https://pypi.org/project/pendulum/#history) in the
> >> time
> >>>>> when
> >>>>>>>>> Airflow
> >>>>>>>>>>>>> 1.10.11 was released
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Fortunately, the project is not abandoned and on a
> >>> regular
> >>>>>>> basis
> >>>>>>>>>>> commits
> >>>>>>>>>>>>> add into the master branch. However these commits are
> >>> not
> >>>>>>>> included
> >>>>>>>>>> into
> >>>>>>>>>>>> any
> >>>>>>>>>>>>> final release and that's why some things related to
> >>>>> datetime
> >>>>>>>> don't
> >>>>>>>>>> work
> >>>>>>>>>>>> as
> >>>>>>>>>>>>> expected in Airflow. There are list of known (for me)
> >>>>> issues
> >>>>>>>> which
> >>>>>>>>>> are
> >>>>>>>>>>>>> affect Airflow
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> *Memory Leak on parse*:
> >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/720,
> >>> this
> >>>>> one
> >>>>>>>> fixed
> >>>>>>>>>> 2
> >>>>>>>>>>>>> years ago but not available yet (
> >>>>>>>>>>>>> https://github.com/sdispater/pendulum/pull/563).
> >> Since
> >>> we
> >>>>>> use
> >>>>>>>>> parse
> >>>>>>>>>>>> dates
> >>>>>>>>>>>>> in airflow codebase: datetime parameters and datetime
> >> in
> >>>>> logs
> >>>>>>>> this
> >>>>>>>>>> one
> >>>>>>>>>>>>> could be a reason for memory leakage in Airflow:
> >>>>>>>>>>>>> - https://github.com/apache/airflow/discussions/24694
> >>>>>>>>>>>>> - https://github.com/apache/airflow/discussions/28597
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> *Incorrect time zones*, known issues and should be
> >>> already
> >>>>>>> fixed
> >>>>>>>> in
> >>>>>>>>>>>>> master branch
> >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/700,
> >>>>> Mexico
> >>>>>> do
> >>>>>>>> not
> >>>>>>>>>> use
> >>>>>>>>>>>> DST
> >>>>>>>>>>>>> anymore
> >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/706,
> >>> Egypt
> >>>>>>>>> reinstate
> >>>>>>>>>>> DST
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> We add clarification in
> >>>>>>>>> https://github.com/apache/airflow/pull/30467
> >>>>>>>>>> ,
> >>>>>>>>>>>>> however it seems like there is no other way rather
> >> than
> >>>>>>> patching
> >>>>>>>>>>> Pendulum
> >>>>>>>>>>>>> right now.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> All these issues should be solved as soon as pendulum
> >> 3
> >>> is
> >>>>>>>>> released.
> >>>>>>>>>>> The
> >>>>>>>>>>>>> current announced estimation is end of september/
> >>>>> beginning
> >>>>>> of
> >>>>>>>>>> October:
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> https://github.com/sdispater/pendulum/issues/600#issuecomment-1711299677
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> So in theory we would have a fixed version of pendulum
> >>>>> soon,
> >>>>>>> and
> >>>>>>>> it
> >>>>>>>>>>> might
> >>>>>>>>>>>>> break something in Airflow but from my point of view
> >> it
> >>> is
> >>>>>>> better
> >>>>>>>>>> than
> >>>>>>>>>>>>> current status.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> However there might be a situation where the release
> >> of
> >>>>> the
> >>>>>>>>> pendulum
> >>>>>>>>>>>> would
> >>>>>>>>>>>>> be postponed, so maybe better to have a backup plan.
> >>> What
> >>>>>> could
> >>>>>>>> we
> >>>>>>>>> do
> >>>>>>>>>>> in
> >>>>>>>>>>>>> this case?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Maybe we should start to use zoneinfo.ZoneInfo instead
> >>> of
> >>>>>>>> pendulum
> >>>>>>>>>>>>> datetime?
> >>> https://github.com/apache/airflow/issues/19450
> >>>>>>>>>>>>> Pros:
> >>>>>>>>>>>>> - stdlib (python 3.9+)
> >>>>>>>>>>>>> - In pendulum 3.0 Timezone based on zoneinfo.Zoneinfo
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cons:
> >>>>>>>>>>>>> - Current serialization model can't deal with backport
> >>>>>>> packages.
> >>>>>>>>> E.g.
> >>>>>>>>>>>>> timezone which are serialized in backport_zoneinfo
> >> can't
> >>>>> be
> >>>>>>>>>>> deserialized
> >>>>>>>>>>>> in
> >>>>>>>>>>>>> zoneinfo
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Maybe we should replace parse datetime with another
> >>>>> solution.
> >>>>>>>> Does
> >>>>>>>>>>> anyone
> >>>>>>>>>>>>> know a good replacement?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Maybe someone from Airflow Community could propose
> >> their
> >>>>> help
> >>>>>>>> with
> >>>>>>>>>>>>> maintenance of library:
> >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/590
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Maybe we should get rid of the pendulum at all, as a
> >>> last
> >>>>>>> resort
> >>>>>>>>>>>> solution.
> >>>>>>>>>>>>> I can't imagine how we could do that, because a lot of
> >>>>> stuff
> >>>>>>>>> depends
> >>>>>>>>>> on
> >>>>>>>>>>>> the
> >>>>>>>>>>>>> pendulum and removing it would be a breaking change.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> ----
> >>>>>>>>>>>>> Best Wishes
> >>>>>>>>>>>>> *Andrey Anshin*
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>>> Bolke de Bruin
> >>>>>>>>>>> bdbruin@gmail.com
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [DISCUSS] Future of Pendulum in Airflow
Posted by Andrey Anshin <an...@taragol.is>.
> So if we switch to the datetime + zone info + timezone we need to have a
test suite that verifies this.
We need to have these test suites anyway. According to the issues and
attempts to fix DST transitions we still have that problem even with use
Pendulum, or maybe it is happen because we use pendulum rather than use
fold argument in datetime.datetime
On Fri, 1 Dec 2023 at 20:41, Bolke de Bruin <bd...@gmail.com> wrote:
> The alternative is to vendor in or fork pendulum as part of the Airflow
> project. Also a major undertaking but maybe less effort.
>
> We shouldn't just address this as a technical challenge. The question
> becomes: does that combination fornally do DST transitions correctly, does
> it do leap seconds correctly, does it add dates correctly, does it do
> timezone traversal correctly and many many other things.
>
> So if we switch to the datetime + zone info + timezone we need to have a
> test suite that verifies this. Otherwise we potentially expose our users to
> different behavior IN their data. Not just a schedule by in their SQL or in
> some scripting where logical date / execution date is used or where date
> arithmetic is applied.
>
> So it is not changing one car for another. It is changing the means of
> transportation and not knowing if you can get to all the places you want /
> need to go as before.
>
> Honestly, I'm a bit scared (I do see the lack of responsiveness). I hope
> someone can take that away.
>
> Bolke
>
>
> Sent from my iPhone
>
> > On 1 Dec 2023, at 13:39, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > Agree. I think this is the best way forward.- we have to bite the
> bullet of
> > potential backwards compatibility issues and separating out a compat
> layer
> > where we could optionally use pendulum instead would be the most "easy"
> > approach for our users. Most of them would not even notice if we do a
> good
> > job, but those who somehow depend on pendulum objects would get a
> > possibility to switch back.
> >
> >> On Fri, Dec 1, 2023 at 1:09 PM Andrey Anshin <an...@taragol.is>
> >> wrote:
> >>
> >> IMHO, if we would like to replace pendulum to alternatives, the better
> >> alternative would be combination of datetime.datetime + zoneinfo +
> >> datetime.timezone
> >>
> >> We still do hacks around pendulum functional, which makes hacks around
> >> datetime.datetime, so in my perspective better remove redundant layers
> and
> >> use native objects directly.
> >>
> >> What confuses me the most is how we get around type changes in Tasks
> >> Context. All dates return as pendulum.DateTime for a very long time
> >> (literally for ages) and it is probably could be classified as breaking
> >> changes, because it might break quite a bit users pipelines. It still
> could
> >> be resolved as optional pendulum dependency + config option for datetime
> >> types in context, which might help to get familiar types for a while.
> >>
> >>
> >>> On Wed, 29 Nov 2023 at 05:58, Jarek Potiuk <ja...@potiuk.com> wrote:
> >>>
> >>> "infamous fork of " should be "fork of infamous Akka" - just to be
> clear
> >> :)
> >>>
> >>>> On Wed, Nov 29, 2023 at 2:57 AM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> >>>
> >>>> It's a very rare occurrence that ASF accepts forks. Usually it has to
> >> be
> >>>> willingly donated by those who own the IP rights, - mainly because of
> >>>> trademark issues. Licence is one thing, but carrying things like the
> >> name
> >>>> of the project is not possible without clearing IP and donating the
> >>>> project.
> >>>>
> >>>> It happened last time last year - with Pekko project - infamous fork
> or
> >>>> Akka after Lightbend changed licensing for Akka -
> >>>>
> >> https://www.lightbend.com/blog/why-we-are-changing-the-license-for-akka
> >>>> making it impossible to be used in ASF projects with the new licence.
> >> But
> >>>> it was a really "strong" case and the "we do not accept forks usually"
> >>> was
> >>>> quite a contentious issue (I followed the discussion - it was
> >> fascinating
> >>>> actually). It tool a LOT of time and effort for those who led the
> >> effort
> >>>> eventually - and also mostly because things had to be crystal-clear
> and
> >>> not
> >>>> "litigable" because the other side had some business reasons to make
> >> the
> >>>> licensing changes and they did not understood why it is impossible to
> >> use
> >>>> Akka as-is even with all the exclusions they offered to ASF - and
> >>>> effectively it was a "hostile" move.
> >>>>
> >>>> I don't think it's "worth" it in this case to be honest - especially
> if
> >>>> the maintainers are effectively "silent" and seem to ignore problems
> of
> >>> one
> >>>> of the really serious users they have. If they were willing to
> >> cooperate,
> >>>> we could do a lot, but when we are left alone, I think it will be far
> >>> more
> >>>> convenient for us to look for alternatives - to be honest.
> >>>>
> >>>> J.
> >>>>
> >>>>
> >>>> On Wed, Nov 29, 2023 at 2:46 AM Austin Bennett <au...@apache.org>
> >>> wrote:
> >>>>
> >>>>> An option would be to fork Pendulum? It is MIT Licensed, I don't
> know
> >>>>> whether that poses problems to get in ASF?
> >>>>>
> >>>>> If forking (?) which is somewhat non-ideal, would we want that 'in'
> >>>>> airflow?
> >>>>>
> >>>>> If not 'in' airflow, I wonder if ASF incubator would accept a forked
> >>>>> project? [ anecdotally Linux Foundation has incubated Starrocks
> which
> >>> is
> >>>>> a
> >>>>> fork of Apache Doris, so there's precedence for this to be OK in the
> >>> wider
> >>>>> OSS ecosystem ].
> >>>>>
> >>>>>
> >>>>> On Tue, Nov 28, 2023 at 6:41 PM Jarek Potiuk <ja...@potiuk.com>
> >> wrote:
> >>>>>
> >>>>>> I have not heard back from maintainers, just a comment from someone
> >>> else
> >>>>>> who suggested donating a pendulum to the ASF (which is kinda
> >>> interesting
> >>>>>> idea). I followed up, let's see. I think if we do not hear back and
> >>>>> there
> >>>>>> will be another week or two where "Pendulum 3 supporting 3.12
> >> support
> >>>>> "is
> >>>>>> coming" but no-one knows when, effectively means that we should look
> >>>>> for an
> >>>>>> alternative like NOW.
> >>>>>>
> >>>>>> J.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Nov 22, 2023 at 3:27 AM Daniel Standish
> >>>>>> <da...@astronomer.io.invalid> wrote:
> >>>>>>
> >>>>>>> Thanks Jarek
> >>>>>>>
> >>>>>>> On Tue, Nov 21, 2023, 4:34 PM Jarek Potiuk <ja...@potiuk.com>
> >>> wrote:
> >>>>>>>
> >>>>>>>> I think we miss important insight - straight from the source.
> >>>>>>>>
> >>>>>>>> I believe it's time to be candid and simply ask questions for
> >> the
> >>>>>> future
> >>>>>>> of
> >>>>>>>> Pendulum directly where we should - ie. we should just ask
> >>>>>> maintainers.
> >>>>>>>>
> >>>>>>>> I've just started a very candid and open discussion there -
> >>>>>>>> https://github.com/sdispater/pendulum/discussions/771 .
> >>>>>>>>
> >>>>>>>> I linked back to this discussion and explained where we are
> >> coming
> >>>>>> from.
> >>>>>>> I
> >>>>>>>> even offered an option, that maybe - if they accept some help
> >> and
> >>>>> would
> >>>>>>> be
> >>>>>>>> open to make a few of us maintainers of Pendulum, that I will
> >> ask
> >>> if
> >>>>>> some
> >>>>>>>> of the maintainers here in Airflow would like to step up (I
> >> would
> >>> be
> >>>>>>>> willing to - for one) and help to at least move Pendulum through
> >>>>> 3.12
> >>>>>> and
> >>>>>>>> maybe keep it running for as long as it will be needed to
> >> maintain
> >>>>> it.
> >>>>>>>>
> >>>>>>>> I think if they would be open to some of us helping them - this
> >>>>> might
> >>>>>> be
> >>>>>>>> actually the "cheapest" option for us to be honest at least in a
> >>>>>> mid-term
> >>>>>>>> if we could gain influence and even take part and speed up
> >>> releases
> >>>>> and
> >>>>>>>> merging of issues that are blocking us (or would be blocking us
> >> in
> >>>>> the
> >>>>>>>> future).
> >>>>>>>>
> >>>>>>>> I tried to be very friendly but candid and direct on what kind
> >> of
> >>>>>>> problems
> >>>>>>>> it creates us and also expressed our thanks for supporting us
> >> for
> >>> so
> >>>>>> long
> >>>>>>>> (I do believe they deserve it). I think the sole (pretty much)
> >>>>>>> maintainer
> >>>>>>>> of Pendulum might really have a hard time maintaining and
> >> keeping
> >>>>> it up
> >>>>>>> for
> >>>>>>>> years - without being paid for and thanked. I hope - maybe all
> >>> that
> >>>>>> they
> >>>>>>>> need are a few words of encouragement and thanks and realising
> >>> that
> >>>>>>> others
> >>>>>>>> depend on their work - but also seeing that they are understood,
> >>>>>>> empathised
> >>>>>>>> with and that there are viable alternatives they might follow
> >>> might
> >>>>> be
> >>>>>>>> helpful for them to be also candid and respond. I hope my
> >>> intentions
> >>>>>> will
> >>>>>>>> be understood - that I am not complaining, but even trying to
> >>> help.
> >>>>>>>>
> >>>>>>>> Let\s see where it gets us.
> >>>>>>>>
> >>>>>>>> J.
> >>>>>>>>
> >>>>>>>> On Fri, Nov 17, 2023 at 5:32 PM Andrey Anshin <
> >>>>>> andrey.anshin@taragol.is>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Just to clarify I'd like us to consider the possibility that
> >> no
> >>>>> new
> >>>>>>>>> pendulum would be released or released at the end of 2024,
> >> like
> >>> a
> >>>>>>>>> pessimistic scenario:
> >>>>>>>>> - What should we do in this case?
> >>>>>>>>> - Work out a backup plan.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> ----
> >>>>>>>>> Best Wishes
> >>>>>>>>> *Andrey Anshin*
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Fri, 17 Nov 2023 at 16:33, Jarek Potiuk <ja...@potiuk.com>
> >>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Also I think TP - had a document in the past (years ago)
> >>>>>>> describing a
> >>>>>>>>>> draft of a more complete alternative we can take to approach
> >>>>>> datetime
> >>>>>>>> vs.
> >>>>>>>>>> pendulum dichotomy. I cannot easily find the document and
> >>>>>> discussion
> >>>>>>> -
> >>>>>>>>> but
> >>>>>>>>>> I do remember it was proposing some interesting changes in
> >> the
> >>>>>>> approach
> >>>>>>>>> of
> >>>>>>>>>> Airflow to have an abstraction layer on top of it (as far
> >> as I
> >>>>>>>> remember).
> >>>>>>>>>> Maybe we can resurrect that idea if TP might find the
> >>> proposal ?
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Nov 17, 2023 at 1:06 PM Bolke de Bruin <
> >>>>> bdbruin@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi,
> >>>>>>>>>>>
> >>>>>>>>>>> I agree that the current speed of development of Pendulum
> >>>>> leaves
> >>>>>>>>>> something
> >>>>>>>>>>> to be desired. However, I think we should not
> >> underestimate
> >>>>> the
> >>>>>>>> effort
> >>>>>>>>> of
> >>>>>>>>>>> replacing it. It is not just a matter of
> >>>>> %s/pendulum/datetime/g
> >>>>>> so
> >>>>>>> to
> >>>>>>>>>> say.
> >>>>>>>>>>> If we are *truly* thinking about moving to native
> >> datetime /
> >>>>>>> zoneinfo
> >>>>>>>>> etc
> >>>>>>>>>>> we need *extensive* tests, basically copying what pendulum
> >>>>> does
> >>>>>> to
> >>>>>>>>> check
> >>>>>>>>>>> its behavior. The reason is that native implementations in
> >>> the
> >>>>>> past
> >>>>>>>>> made
> >>>>>>>>>>> serious mistakes and I do not put a lot of trust in them.
> >>>>>>>>>>>
> >>>>>>>>>>> An abstraction or vendoring in could be alternatives, but
> >>> they
> >>>>>>> bring
> >>>>>>>>>> their
> >>>>>>>>>>> own significant challenges.
> >>>>>>>>>>>
> >>>>>>>>>>> I have re-raised the issue here:
> >>>>>>>>>>>
> >>>>>>>>>>> https://github.com/sdispater/pendulum/issues/753
> >>>>>>>>>>>
> >>>>>>>>>>> The upside is that it seems the amount of issues with the
> >>>>> beta is
> >>>>>>>>>> limited,
> >>>>>>>>>>> so hopefully the maintainers can spend a couple of cycles
> >> to
> >>>>>>> address
> >>>>>>>>>> them.
> >>>>>>>>>>>
> >>>>>>>>>>> Bolke
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, 17 Nov 2023 at 10:28, Andrey Anshin <
> >>>>>>>> andrey.anshin@taragol.is>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> There is no changes in stable pendulum so let's try to
> >>>>> continue
> >>>>>>>> this
> >>>>>>>>>>>> discussion and start think about "Plan B"
> >>>>>>>>>>>>
> >>>>>>>>>>>> Just a reminder:
> >>>>>>>>>>>> - pendulum 2.1.2 released 3 years ago (at the time
> >> Airflow
> >>>>>>> 1.10.x)
> >>>>>>>>>>>> - pendulum 2 doesn't work well in Python 3.12, this is a
> >>>>>>>> showstopper
> >>>>>>>>>> for
> >>>>>>>>>>>> the support Python 3.12
> >>>>>>>>>>>> - pendulum 2 have memory leaks (
> >>>>>>>>>>>> https://github.com/sdispater/pendulum/issues/720), and
> >>>>> Airflow
> >>>>>>> use
> >>>>>>>>>>>> approach
> >>>>>>>>>>>> to achieve this leaks, especially in K8S executor
> >>>>>>>>>>>> - pendulum 2 doesn't use system tzdata by default, but
> >> we
> >>>>> have
> >>>>>> a
> >>>>>>>>>>>> workaround, thanks Bolke for the documentation
> >>>>>>>>>>>> - pendulum it is a core dependency of Airflow, and I
> >> guess
> >>>>> we
> >>>>>>> can't
> >>>>>>>>>>>> remove/replace it without breaking changes.
> >>>>>>>>>>>>
> >>>>>>>>>>>> So my proposal if things won't change in the near future
> >>>>> then
> >>>>>> we
> >>>>>>>> need
> >>>>>>>>>> to
> >>>>>>>>>>>> start removing the pendulum from the core and replace it
> >>>>> with
> >>>>>> the
> >>>>>>>>>> native
> >>>>>>>>>>>> datetime / zoneinfo / tzinfo. But maybe we have another
> >>> way
> >>>>> to
> >>>>>>>>> resolve
> >>>>>>>>>>> this
> >>>>>>>>>>>> without breaking changes? Because for me it would be a
> >>>>> little
> >>>>>>> weird
> >>>>>>>>> if
> >>>>>>>>>>>> removal pendulum would be a main driver for release
> >>> Airflow
> >>>>> 3
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> ----
> >>>>>>>>>>>> Best Wishes
> >>>>>>>>>>>> *Andrey Anshin*
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, 28 Sept 2023 at 13:01, Andrey Anshin <
> >>>>>>>>> andrey.anshin@taragol.is
> >>>>>>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> This discussion is more about the known problem of
> >>>>> pendulum
> >>>>>> and
> >>>>>>>> how
> >>>>>>>>>> we
> >>>>>>>>>>>>> could deal with it and maybe how we (as Community)
> >> might
> >>>>> help
> >>>>>>>>> autor.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> The library is mostly supported by a single author
> >>>>> Sébastien
> >>>>>>>>> Eustace
> >>>>>>>>>> (
> >>>>>>>>>>>>> https://github.com/sdispater) and it seems like we
> >> bump
> >>>>> into
> >>>>>>> the
> >>>>>>>>>>>>> situation which is described in xkcd #2347 (
> >>>>>>>>>>>>> https://imgs.xkcd.com/comics/dependency.png). To be
> >>>>> honest
> >>>>>> it
> >>>>>>> is
> >>>>>>>>> not
> >>>>>>>>>>>>> something new when library mainly supported by one
> >>> author
> >>>>> so
> >>>>>>>> there
> >>>>>>>>> is
> >>>>>>>>>>>>> always a risk that the library will no longer be
> >>>>> supported /
> >>>>>>>>>> abandoned
> >>>>>>>>>>>>> And if takes in account that pendulum provides core
> >>>>>>> functionality
> >>>>>>>>> in
> >>>>>>>>>>>>> Airflow it could have dramatical impact in the future.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Pendulum is a really nice library which helps a lot of
> >>>>>>> developers
> >>>>>>>>> to
> >>>>>>>>>>> work
> >>>>>>>>>>>>> with dates/datetimes. However there is one major
> >>> problem,
> >>>>> the
> >>>>>>>> last
> >>>>>>>>>>>> release
> >>>>>>>>>>>>> of this library happened more than 3 years ago (
> >>>>>>>>>>>>> https://pypi.org/project/pendulum/#history) in the
> >> time
> >>>>> when
> >>>>>>>>> Airflow
> >>>>>>>>>>>>> 1.10.11 was released
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Fortunately, the project is not abandoned and on a
> >>> regular
> >>>>>>> basis
> >>>>>>>>>>> commits
> >>>>>>>>>>>>> add into the master branch. However these commits are
> >>> not
> >>>>>>>> included
> >>>>>>>>>> into
> >>>>>>>>>>>> any
> >>>>>>>>>>>>> final release and that's why some things related to
> >>>>> datetime
> >>>>>>>> don't
> >>>>>>>>>> work
> >>>>>>>>>>>> as
> >>>>>>>>>>>>> expected in Airflow. There are list of known (for me)
> >>>>> issues
> >>>>>>>> which
> >>>>>>>>>> are
> >>>>>>>>>>>>> affect Airflow
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> *Memory Leak on parse*:
> >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/720,
> >>> this
> >>>>> one
> >>>>>>>> fixed
> >>>>>>>>>> 2
> >>>>>>>>>>>>> years ago but not available yet (
> >>>>>>>>>>>>> https://github.com/sdispater/pendulum/pull/563).
> >> Since
> >>> we
> >>>>>> use
> >>>>>>>>> parse
> >>>>>>>>>>>> dates
> >>>>>>>>>>>>> in airflow codebase: datetime parameters and datetime
> >> in
> >>>>> logs
> >>>>>>>> this
> >>>>>>>>>> one
> >>>>>>>>>>>>> could be a reason for memory leakage in Airflow:
> >>>>>>>>>>>>> - https://github.com/apache/airflow/discussions/24694
> >>>>>>>>>>>>> - https://github.com/apache/airflow/discussions/28597
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> *Incorrect time zones*, known issues and should be
> >>> already
> >>>>>>> fixed
> >>>>>>>> in
> >>>>>>>>>>>>> master branch
> >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/700,
> >>>>> Mexico
> >>>>>> do
> >>>>>>>> not
> >>>>>>>>>> use
> >>>>>>>>>>>> DST
> >>>>>>>>>>>>> anymore
> >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/706,
> >>> Egypt
> >>>>>>>>> reinstate
> >>>>>>>>>>> DST
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> We add clarification in
> >>>>>>>>> https://github.com/apache/airflow/pull/30467
> >>>>>>>>>> ,
> >>>>>>>>>>>>> however it seems like there is no other way rather
> >> than
> >>>>>>> patching
> >>>>>>>>>>> Pendulum
> >>>>>>>>>>>>> right now.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> All these issues should be solved as soon as pendulum
> >> 3
> >>> is
> >>>>>>>>> released.
> >>>>>>>>>>> The
> >>>>>>>>>>>>> current announced estimation is end of september/
> >>>>> beginning
> >>>>>> of
> >>>>>>>>>> October:
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> https://github.com/sdispater/pendulum/issues/600#issuecomment-1711299677
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> So in theory we would have a fixed version of pendulum
> >>>>> soon,
> >>>>>>> and
> >>>>>>>> it
> >>>>>>>>>>> might
> >>>>>>>>>>>>> break something in Airflow but from my point of view
> >> it
> >>> is
> >>>>>>> better
> >>>>>>>>>> than
> >>>>>>>>>>>>> current status.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> However there might be a situation where the release
> >> of
> >>>>> the
> >>>>>>>>> pendulum
> >>>>>>>>>>>> would
> >>>>>>>>>>>>> be postponed, so maybe better to have a backup plan.
> >>> What
> >>>>>> could
> >>>>>>>> we
> >>>>>>>>> do
> >>>>>>>>>>> in
> >>>>>>>>>>>>> this case?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Maybe we should start to use zoneinfo.ZoneInfo instead
> >>> of
> >>>>>>>> pendulum
> >>>>>>>>>>>>> datetime?
> >>> https://github.com/apache/airflow/issues/19450
> >>>>>>>>>>>>> Pros:
> >>>>>>>>>>>>> - stdlib (python 3.9+)
> >>>>>>>>>>>>> - In pendulum 3.0 Timezone based on zoneinfo.Zoneinfo
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cons:
> >>>>>>>>>>>>> - Current serialization model can't deal with backport
> >>>>>>> packages.
> >>>>>>>>> E.g.
> >>>>>>>>>>>>> timezone which are serialized in backport_zoneinfo
> >> can't
> >>>>> be
> >>>>>>>>>>> deserialized
> >>>>>>>>>>>> in
> >>>>>>>>>>>>> zoneinfo
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Maybe we should replace parse datetime with another
> >>>>> solution.
> >>>>>>>> Does
> >>>>>>>>>>> anyone
> >>>>>>>>>>>>> know a good replacement?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Maybe someone from Airflow Community could propose
> >> their
> >>>>> help
> >>>>>>>> with
> >>>>>>>>>>>>> maintenance of library:
> >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/590
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Maybe we should get rid of the pendulum at all, as a
> >>> last
> >>>>>>> resort
> >>>>>>>>>>>> solution.
> >>>>>>>>>>>>> I can't imagine how we could do that, because a lot of
> >>>>> stuff
> >>>>>>>>> depends
> >>>>>>>>>> on
> >>>>>>>>>>>> the
> >>>>>>>>>>>>> pendulum and removing it would be a breaking change.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> ----
> >>>>>>>>>>>>> Best Wishes
> >>>>>>>>>>>>> *Andrey Anshin*
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>>> Bolke de Bruin
> >>>>>>>>>>> bdbruin@gmail.com
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [DISCUSS] Future of Pendulum in Airflow
Posted by Bolke de Bruin <bd...@gmail.com>.
The alternative is to vendor in or fork pendulum as part of the Airflow project. Also a major undertaking but maybe less effort.
We shouldn't just address this as a technical challenge. The question becomes: does that combination fornally do DST transitions correctly, does it do leap seconds correctly, does it add dates correctly, does it do timezone traversal correctly and many many other things.
So if we switch to the datetime + zone info + timezone we need to have a test suite that verifies this. Otherwise we potentially expose our users to different behavior IN their data. Not just a schedule by in their SQL or in some scripting where logical date / execution date is used or where date arithmetic is applied.
So it is not changing one car for another. It is changing the means of transportation and not knowing if you can get to all the places you want / need to go as before.
Honestly, I'm a bit scared (I do see the lack of responsiveness). I hope someone can take that away.
Bolke
Sent from my iPhone
> On 1 Dec 2023, at 13:39, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> Agree. I think this is the best way forward.- we have to bite the bullet of
> potential backwards compatibility issues and separating out a compat layer
> where we could optionally use pendulum instead would be the most "easy"
> approach for our users. Most of them would not even notice if we do a good
> job, but those who somehow depend on pendulum objects would get a
> possibility to switch back.
>
>> On Fri, Dec 1, 2023 at 1:09 PM Andrey Anshin <an...@taragol.is>
>> wrote:
>>
>> IMHO, if we would like to replace pendulum to alternatives, the better
>> alternative would be combination of datetime.datetime + zoneinfo +
>> datetime.timezone
>>
>> We still do hacks around pendulum functional, which makes hacks around
>> datetime.datetime, so in my perspective better remove redundant layers and
>> use native objects directly.
>>
>> What confuses me the most is how we get around type changes in Tasks
>> Context. All dates return as pendulum.DateTime for a very long time
>> (literally for ages) and it is probably could be classified as breaking
>> changes, because it might break quite a bit users pipelines. It still could
>> be resolved as optional pendulum dependency + config option for datetime
>> types in context, which might help to get familiar types for a while.
>>
>>
>>> On Wed, 29 Nov 2023 at 05:58, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>> "infamous fork of " should be "fork of infamous Akka" - just to be clear
>> :)
>>>
>>>> On Wed, Nov 29, 2023 at 2:57 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>>> It's a very rare occurrence that ASF accepts forks. Usually it has to
>> be
>>>> willingly donated by those who own the IP rights, - mainly because of
>>>> trademark issues. Licence is one thing, but carrying things like the
>> name
>>>> of the project is not possible without clearing IP and donating the
>>>> project.
>>>>
>>>> It happened last time last year - with Pekko project - infamous fork or
>>>> Akka after Lightbend changed licensing for Akka -
>>>>
>> https://www.lightbend.com/blog/why-we-are-changing-the-license-for-akka
>>>> making it impossible to be used in ASF projects with the new licence.
>> But
>>>> it was a really "strong" case and the "we do not accept forks usually"
>>> was
>>>> quite a contentious issue (I followed the discussion - it was
>> fascinating
>>>> actually). It tool a LOT of time and effort for those who led the
>> effort
>>>> eventually - and also mostly because things had to be crystal-clear and
>>> not
>>>> "litigable" because the other side had some business reasons to make
>> the
>>>> licensing changes and they did not understood why it is impossible to
>> use
>>>> Akka as-is even with all the exclusions they offered to ASF - and
>>>> effectively it was a "hostile" move.
>>>>
>>>> I don't think it's "worth" it in this case to be honest - especially if
>>>> the maintainers are effectively "silent" and seem to ignore problems of
>>> one
>>>> of the really serious users they have. If they were willing to
>> cooperate,
>>>> we could do a lot, but when we are left alone, I think it will be far
>>> more
>>>> convenient for us to look for alternatives - to be honest.
>>>>
>>>> J.
>>>>
>>>>
>>>> On Wed, Nov 29, 2023 at 2:46 AM Austin Bennett <au...@apache.org>
>>> wrote:
>>>>
>>>>> An option would be to fork Pendulum? It is MIT Licensed, I don't know
>>>>> whether that poses problems to get in ASF?
>>>>>
>>>>> If forking (?) which is somewhat non-ideal, would we want that 'in'
>>>>> airflow?
>>>>>
>>>>> If not 'in' airflow, I wonder if ASF incubator would accept a forked
>>>>> project? [ anecdotally Linux Foundation has incubated Starrocks which
>>> is
>>>>> a
>>>>> fork of Apache Doris, so there's precedence for this to be OK in the
>>> wider
>>>>> OSS ecosystem ].
>>>>>
>>>>>
>>>>> On Tue, Nov 28, 2023 at 6:41 PM Jarek Potiuk <ja...@potiuk.com>
>> wrote:
>>>>>
>>>>>> I have not heard back from maintainers, just a comment from someone
>>> else
>>>>>> who suggested donating a pendulum to the ASF (which is kinda
>>> interesting
>>>>>> idea). I followed up, let's see. I think if we do not hear back and
>>>>> there
>>>>>> will be another week or two where "Pendulum 3 supporting 3.12
>> support
>>>>> "is
>>>>>> coming" but no-one knows when, effectively means that we should look
>>>>> for an
>>>>>> alternative like NOW.
>>>>>>
>>>>>> J.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Nov 22, 2023 at 3:27 AM Daniel Standish
>>>>>> <da...@astronomer.io.invalid> wrote:
>>>>>>
>>>>>>> Thanks Jarek
>>>>>>>
>>>>>>> On Tue, Nov 21, 2023, 4:34 PM Jarek Potiuk <ja...@potiuk.com>
>>> wrote:
>>>>>>>
>>>>>>>> I think we miss important insight - straight from the source.
>>>>>>>>
>>>>>>>> I believe it's time to be candid and simply ask questions for
>> the
>>>>>> future
>>>>>>> of
>>>>>>>> Pendulum directly where we should - ie. we should just ask
>>>>>> maintainers.
>>>>>>>>
>>>>>>>> I've just started a very candid and open discussion there -
>>>>>>>> https://github.com/sdispater/pendulum/discussions/771 .
>>>>>>>>
>>>>>>>> I linked back to this discussion and explained where we are
>> coming
>>>>>> from.
>>>>>>> I
>>>>>>>> even offered an option, that maybe - if they accept some help
>> and
>>>>> would
>>>>>>> be
>>>>>>>> open to make a few of us maintainers of Pendulum, that I will
>> ask
>>> if
>>>>>> some
>>>>>>>> of the maintainers here in Airflow would like to step up (I
>> would
>>> be
>>>>>>>> willing to - for one) and help to at least move Pendulum through
>>>>> 3.12
>>>>>> and
>>>>>>>> maybe keep it running for as long as it will be needed to
>> maintain
>>>>> it.
>>>>>>>>
>>>>>>>> I think if they would be open to some of us helping them - this
>>>>> might
>>>>>> be
>>>>>>>> actually the "cheapest" option for us to be honest at least in a
>>>>>> mid-term
>>>>>>>> if we could gain influence and even take part and speed up
>>> releases
>>>>> and
>>>>>>>> merging of issues that are blocking us (or would be blocking us
>> in
>>>>> the
>>>>>>>> future).
>>>>>>>>
>>>>>>>> I tried to be very friendly but candid and direct on what kind
>> of
>>>>>>> problems
>>>>>>>> it creates us and also expressed our thanks for supporting us
>> for
>>> so
>>>>>> long
>>>>>>>> (I do believe they deserve it). I think the sole (pretty much)
>>>>>>> maintainer
>>>>>>>> of Pendulum might really have a hard time maintaining and
>> keeping
>>>>> it up
>>>>>>> for
>>>>>>>> years - without being paid for and thanked. I hope - maybe all
>>> that
>>>>>> they
>>>>>>>> need are a few words of encouragement and thanks and realising
>>> that
>>>>>>> others
>>>>>>>> depend on their work - but also seeing that they are understood,
>>>>>>> empathised
>>>>>>>> with and that there are viable alternatives they might follow
>>> might
>>>>> be
>>>>>>>> helpful for them to be also candid and respond. I hope my
>>> intentions
>>>>>> will
>>>>>>>> be understood - that I am not complaining, but even trying to
>>> help.
>>>>>>>>
>>>>>>>> Let\s see where it gets us.
>>>>>>>>
>>>>>>>> J.
>>>>>>>>
>>>>>>>> On Fri, Nov 17, 2023 at 5:32 PM Andrey Anshin <
>>>>>> andrey.anshin@taragol.is>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Just to clarify I'd like us to consider the possibility that
>> no
>>>>> new
>>>>>>>>> pendulum would be released or released at the end of 2024,
>> like
>>> a
>>>>>>>>> pessimistic scenario:
>>>>>>>>> - What should we do in this case?
>>>>>>>>> - Work out a backup plan.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ----
>>>>>>>>> Best Wishes
>>>>>>>>> *Andrey Anshin*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, 17 Nov 2023 at 16:33, Jarek Potiuk <ja...@potiuk.com>
>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Also I think TP - had a document in the past (years ago)
>>>>>>> describing a
>>>>>>>>>> draft of a more complete alternative we can take to approach
>>>>>> datetime
>>>>>>>> vs.
>>>>>>>>>> pendulum dichotomy. I cannot easily find the document and
>>>>>> discussion
>>>>>>> -
>>>>>>>>> but
>>>>>>>>>> I do remember it was proposing some interesting changes in
>> the
>>>>>>> approach
>>>>>>>>> of
>>>>>>>>>> Airflow to have an abstraction layer on top of it (as far
>> as I
>>>>>>>> remember).
>>>>>>>>>> Maybe we can resurrect that idea if TP might find the
>>> proposal ?
>>>>>>>>>>
>>>>>>>>>> On Fri, Nov 17, 2023 at 1:06 PM Bolke de Bruin <
>>>>> bdbruin@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I agree that the current speed of development of Pendulum
>>>>> leaves
>>>>>>>>>> something
>>>>>>>>>>> to be desired. However, I think we should not
>> underestimate
>>>>> the
>>>>>>>> effort
>>>>>>>>> of
>>>>>>>>>>> replacing it. It is not just a matter of
>>>>> %s/pendulum/datetime/g
>>>>>> so
>>>>>>> to
>>>>>>>>>> say.
>>>>>>>>>>> If we are *truly* thinking about moving to native
>> datetime /
>>>>>>> zoneinfo
>>>>>>>>> etc
>>>>>>>>>>> we need *extensive* tests, basically copying what pendulum
>>>>> does
>>>>>> to
>>>>>>>>> check
>>>>>>>>>>> its behavior. The reason is that native implementations in
>>> the
>>>>>> past
>>>>>>>>> made
>>>>>>>>>>> serious mistakes and I do not put a lot of trust in them.
>>>>>>>>>>>
>>>>>>>>>>> An abstraction or vendoring in could be alternatives, but
>>> they
>>>>>>> bring
>>>>>>>>>> their
>>>>>>>>>>> own significant challenges.
>>>>>>>>>>>
>>>>>>>>>>> I have re-raised the issue here:
>>>>>>>>>>>
>>>>>>>>>>> https://github.com/sdispater/pendulum/issues/753
>>>>>>>>>>>
>>>>>>>>>>> The upside is that it seems the amount of issues with the
>>>>> beta is
>>>>>>>>>> limited,
>>>>>>>>>>> so hopefully the maintainers can spend a couple of cycles
>> to
>>>>>>> address
>>>>>>>>>> them.
>>>>>>>>>>>
>>>>>>>>>>> Bolke
>>>>>>>>>>>
>>>>>>>>>>> On Fri, 17 Nov 2023 at 10:28, Andrey Anshin <
>>>>>>>> andrey.anshin@taragol.is>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> There is no changes in stable pendulum so let's try to
>>>>> continue
>>>>>>>> this
>>>>>>>>>>>> discussion and start think about "Plan B"
>>>>>>>>>>>>
>>>>>>>>>>>> Just a reminder:
>>>>>>>>>>>> - pendulum 2.1.2 released 3 years ago (at the time
>> Airflow
>>>>>>> 1.10.x)
>>>>>>>>>>>> - pendulum 2 doesn't work well in Python 3.12, this is a
>>>>>>>> showstopper
>>>>>>>>>> for
>>>>>>>>>>>> the support Python 3.12
>>>>>>>>>>>> - pendulum 2 have memory leaks (
>>>>>>>>>>>> https://github.com/sdispater/pendulum/issues/720), and
>>>>> Airflow
>>>>>>> use
>>>>>>>>>>>> approach
>>>>>>>>>>>> to achieve this leaks, especially in K8S executor
>>>>>>>>>>>> - pendulum 2 doesn't use system tzdata by default, but
>> we
>>>>> have
>>>>>> a
>>>>>>>>>>>> workaround, thanks Bolke for the documentation
>>>>>>>>>>>> - pendulum it is a core dependency of Airflow, and I
>> guess
>>>>> we
>>>>>>> can't
>>>>>>>>>>>> remove/replace it without breaking changes.
>>>>>>>>>>>>
>>>>>>>>>>>> So my proposal if things won't change in the near future
>>>>> then
>>>>>> we
>>>>>>>> need
>>>>>>>>>> to
>>>>>>>>>>>> start removing the pendulum from the core and replace it
>>>>> with
>>>>>> the
>>>>>>>>>> native
>>>>>>>>>>>> datetime / zoneinfo / tzinfo. But maybe we have another
>>> way
>>>>> to
>>>>>>>>> resolve
>>>>>>>>>>> this
>>>>>>>>>>>> without breaking changes? Because for me it would be a
>>>>> little
>>>>>>> weird
>>>>>>>>> if
>>>>>>>>>>>> removal pendulum would be a main driver for release
>>> Airflow
>>>>> 3
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ----
>>>>>>>>>>>> Best Wishes
>>>>>>>>>>>> *Andrey Anshin*
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, 28 Sept 2023 at 13:01, Andrey Anshin <
>>>>>>>>> andrey.anshin@taragol.is
>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> This discussion is more about the known problem of
>>>>> pendulum
>>>>>> and
>>>>>>>> how
>>>>>>>>>> we
>>>>>>>>>>>>> could deal with it and maybe how we (as Community)
>> might
>>>>> help
>>>>>>>>> autor.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The library is mostly supported by a single author
>>>>> Sébastien
>>>>>>>>> Eustace
>>>>>>>>>> (
>>>>>>>>>>>>> https://github.com/sdispater) and it seems like we
>> bump
>>>>> into
>>>>>>> the
>>>>>>>>>>>>> situation which is described in xkcd #2347 (
>>>>>>>>>>>>> https://imgs.xkcd.com/comics/dependency.png). To be
>>>>> honest
>>>>>> it
>>>>>>> is
>>>>>>>>> not
>>>>>>>>>>>>> something new when library mainly supported by one
>>> author
>>>>> so
>>>>>>>> there
>>>>>>>>> is
>>>>>>>>>>>>> always a risk that the library will no longer be
>>>>> supported /
>>>>>>>>>> abandoned
>>>>>>>>>>>>> And if takes in account that pendulum provides core
>>>>>>> functionality
>>>>>>>>> in
>>>>>>>>>>>>> Airflow it could have dramatical impact in the future.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Pendulum is a really nice library which helps a lot of
>>>>>>> developers
>>>>>>>>> to
>>>>>>>>>>> work
>>>>>>>>>>>>> with dates/datetimes. However there is one major
>>> problem,
>>>>> the
>>>>>>>> last
>>>>>>>>>>>> release
>>>>>>>>>>>>> of this library happened more than 3 years ago (
>>>>>>>>>>>>> https://pypi.org/project/pendulum/#history) in the
>> time
>>>>> when
>>>>>>>>> Airflow
>>>>>>>>>>>>> 1.10.11 was released
>>>>>>>>>>>>>
>>>>>>>>>>>>> Fortunately, the project is not abandoned and on a
>>> regular
>>>>>>> basis
>>>>>>>>>>> commits
>>>>>>>>>>>>> add into the master branch. However these commits are
>>> not
>>>>>>>> included
>>>>>>>>>> into
>>>>>>>>>>>> any
>>>>>>>>>>>>> final release and that's why some things related to
>>>>> datetime
>>>>>>>> don't
>>>>>>>>>> work
>>>>>>>>>>>> as
>>>>>>>>>>>>> expected in Airflow. There are list of known (for me)
>>>>> issues
>>>>>>>> which
>>>>>>>>>> are
>>>>>>>>>>>>> affect Airflow
>>>>>>>>>>>>>
>>>>>>>>>>>>> *Memory Leak on parse*:
>>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/720,
>>> this
>>>>> one
>>>>>>>> fixed
>>>>>>>>>> 2
>>>>>>>>>>>>> years ago but not available yet (
>>>>>>>>>>>>> https://github.com/sdispater/pendulum/pull/563).
>> Since
>>> we
>>>>>> use
>>>>>>>>> parse
>>>>>>>>>>>> dates
>>>>>>>>>>>>> in airflow codebase: datetime parameters and datetime
>> in
>>>>> logs
>>>>>>>> this
>>>>>>>>>> one
>>>>>>>>>>>>> could be a reason for memory leakage in Airflow:
>>>>>>>>>>>>> - https://github.com/apache/airflow/discussions/24694
>>>>>>>>>>>>> - https://github.com/apache/airflow/discussions/28597
>>>>>>>>>>>>>
>>>>>>>>>>>>> *Incorrect time zones*, known issues and should be
>>> already
>>>>>>> fixed
>>>>>>>> in
>>>>>>>>>>>>> master branch
>>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/700,
>>>>> Mexico
>>>>>> do
>>>>>>>> not
>>>>>>>>>> use
>>>>>>>>>>>> DST
>>>>>>>>>>>>> anymore
>>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/706,
>>> Egypt
>>>>>>>>> reinstate
>>>>>>>>>>> DST
>>>>>>>>>>>>>
>>>>>>>>>>>>> We add clarification in
>>>>>>>>> https://github.com/apache/airflow/pull/30467
>>>>>>>>>> ,
>>>>>>>>>>>>> however it seems like there is no other way rather
>> than
>>>>>>> patching
>>>>>>>>>>> Pendulum
>>>>>>>>>>>>> right now.
>>>>>>>>>>>>>
>>>>>>>>>>>>> All these issues should be solved as soon as pendulum
>> 3
>>> is
>>>>>>>>> released.
>>>>>>>>>>> The
>>>>>>>>>>>>> current announced estimation is end of september/
>>>>> beginning
>>>>>> of
>>>>>>>>>> October:
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>> https://github.com/sdispater/pendulum/issues/600#issuecomment-1711299677
>>>>>>>>>>>>>
>>>>>>>>>>>>> So in theory we would have a fixed version of pendulum
>>>>> soon,
>>>>>>> and
>>>>>>>> it
>>>>>>>>>>> might
>>>>>>>>>>>>> break something in Airflow but from my point of view
>> it
>>> is
>>>>>>> better
>>>>>>>>>> than
>>>>>>>>>>>>> current status.
>>>>>>>>>>>>>
>>>>>>>>>>>>> However there might be a situation where the release
>> of
>>>>> the
>>>>>>>>> pendulum
>>>>>>>>>>>> would
>>>>>>>>>>>>> be postponed, so maybe better to have a backup plan.
>>> What
>>>>>> could
>>>>>>>> we
>>>>>>>>> do
>>>>>>>>>>> in
>>>>>>>>>>>>> this case?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Maybe we should start to use zoneinfo.ZoneInfo instead
>>> of
>>>>>>>> pendulum
>>>>>>>>>>>>> datetime?
>>> https://github.com/apache/airflow/issues/19450
>>>>>>>>>>>>> Pros:
>>>>>>>>>>>>> - stdlib (python 3.9+)
>>>>>>>>>>>>> - In pendulum 3.0 Timezone based on zoneinfo.Zoneinfo
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cons:
>>>>>>>>>>>>> - Current serialization model can't deal with backport
>>>>>>> packages.
>>>>>>>>> E.g.
>>>>>>>>>>>>> timezone which are serialized in backport_zoneinfo
>> can't
>>>>> be
>>>>>>>>>>> deserialized
>>>>>>>>>>>> in
>>>>>>>>>>>>> zoneinfo
>>>>>>>>>>>>>
>>>>>>>>>>>>> Maybe we should replace parse datetime with another
>>>>> solution.
>>>>>>>> Does
>>>>>>>>>>> anyone
>>>>>>>>>>>>> know a good replacement?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Maybe someone from Airflow Community could propose
>> their
>>>>> help
>>>>>>>> with
>>>>>>>>>>>>> maintenance of library:
>>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/590
>>>>>>>>>>>>>
>>>>>>>>>>>>> Maybe we should get rid of the pendulum at all, as a
>>> last
>>>>>>> resort
>>>>>>>>>>>> solution.
>>>>>>>>>>>>> I can't imagine how we could do that, because a lot of
>>>>> stuff
>>>>>>>>> depends
>>>>>>>>>> on
>>>>>>>>>>>> the
>>>>>>>>>>>>> pendulum and removing it would be a breaking change.
>>>>>>>>>>>>>
>>>>>>>>>>>>> ----
>>>>>>>>>>>>> Best Wishes
>>>>>>>>>>>>> *Andrey Anshin*
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Bolke de Bruin
>>>>>>>>>>> bdbruin@gmail.com
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] Future of Pendulum in Airflow
Posted by Jarek Potiuk <ja...@potiuk.com>.
Agree. I think this is the best way forward.- we have to bite the bullet of
potential backwards compatibility issues and separating out a compat layer
where we could optionally use pendulum instead would be the most "easy"
approach for our users. Most of them would not even notice if we do a good
job, but those who somehow depend on pendulum objects would get a
possibility to switch back.
On Fri, Dec 1, 2023 at 1:09 PM Andrey Anshin <an...@taragol.is>
wrote:
> IMHO, if we would like to replace pendulum to alternatives, the better
> alternative would be combination of datetime.datetime + zoneinfo +
> datetime.timezone
>
> We still do hacks around pendulum functional, which makes hacks around
> datetime.datetime, so in my perspective better remove redundant layers and
> use native objects directly.
>
> What confuses me the most is how we get around type changes in Tasks
> Context. All dates return as pendulum.DateTime for a very long time
> (literally for ages) and it is probably could be classified as breaking
> changes, because it might break quite a bit users pipelines. It still could
> be resolved as optional pendulum dependency + config option for datetime
> types in context, which might help to get familiar types for a while.
>
>
> On Wed, 29 Nov 2023 at 05:58, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > "infamous fork of " should be "fork of infamous Akka" - just to be clear
> :)
> >
> > On Wed, Nov 29, 2023 at 2:57 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > It's a very rare occurrence that ASF accepts forks. Usually it has to
> be
> > > willingly donated by those who own the IP rights, - mainly because of
> > > trademark issues. Licence is one thing, but carrying things like the
> name
> > > of the project is not possible without clearing IP and donating the
> > > project.
> > >
> > > It happened last time last year - with Pekko project - infamous fork or
> > > Akka after Lightbend changed licensing for Akka -
> > >
> https://www.lightbend.com/blog/why-we-are-changing-the-license-for-akka
> > > making it impossible to be used in ASF projects with the new licence.
> But
> > > it was a really "strong" case and the "we do not accept forks usually"
> > was
> > > quite a contentious issue (I followed the discussion - it was
> fascinating
> > > actually). It tool a LOT of time and effort for those who led the
> effort
> > > eventually - and also mostly because things had to be crystal-clear and
> > not
> > > "litigable" because the other side had some business reasons to make
> the
> > > licensing changes and they did not understood why it is impossible to
> use
> > > Akka as-is even with all the exclusions they offered to ASF - and
> > > effectively it was a "hostile" move.
> > >
> > > I don't think it's "worth" it in this case to be honest - especially if
> > > the maintainers are effectively "silent" and seem to ignore problems of
> > one
> > > of the really serious users they have. If they were willing to
> cooperate,
> > > we could do a lot, but when we are left alone, I think it will be far
> > more
> > > convenient for us to look for alternatives - to be honest.
> > >
> > > J.
> > >
> > >
> > > On Wed, Nov 29, 2023 at 2:46 AM Austin Bennett <au...@apache.org>
> > wrote:
> > >
> > >> An option would be to fork Pendulum? It is MIT Licensed, I don't know
> > >> whether that poses problems to get in ASF?
> > >>
> > >> If forking (?) which is somewhat non-ideal, would we want that 'in'
> > >> airflow?
> > >>
> > >> If not 'in' airflow, I wonder if ASF incubator would accept a forked
> > >> project? [ anecdotally Linux Foundation has incubated Starrocks which
> > is
> > >> a
> > >> fork of Apache Doris, so there's precedence for this to be OK in the
> > wider
> > >> OSS ecosystem ].
> > >>
> > >>
> > >> On Tue, Nov 28, 2023 at 6:41 PM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > >>
> > >> > I have not heard back from maintainers, just a comment from someone
> > else
> > >> > who suggested donating a pendulum to the ASF (which is kinda
> > interesting
> > >> > idea). I followed up, let's see. I think if we do not hear back and
> > >> there
> > >> > will be another week or two where "Pendulum 3 supporting 3.12
> support
> > >> "is
> > >> > coming" but no-one knows when, effectively means that we should look
> > >> for an
> > >> > alternative like NOW.
> > >> >
> > >> > J.
> > >> >
> > >> >
> > >> >
> > >> > On Wed, Nov 22, 2023 at 3:27 AM Daniel Standish
> > >> > <da...@astronomer.io.invalid> wrote:
> > >> >
> > >> > > Thanks Jarek
> > >> > >
> > >> > > On Tue, Nov 21, 2023, 4:34 PM Jarek Potiuk <ja...@potiuk.com>
> > wrote:
> > >> > >
> > >> > > > I think we miss important insight - straight from the source.
> > >> > > >
> > >> > > > I believe it's time to be candid and simply ask questions for
> the
> > >> > future
> > >> > > of
> > >> > > > Pendulum directly where we should - ie. we should just ask
> > >> > maintainers.
> > >> > > >
> > >> > > > I've just started a very candid and open discussion there -
> > >> > > > https://github.com/sdispater/pendulum/discussions/771 .
> > >> > > >
> > >> > > > I linked back to this discussion and explained where we are
> coming
> > >> > from.
> > >> > > I
> > >> > > > even offered an option, that maybe - if they accept some help
> and
> > >> would
> > >> > > be
> > >> > > > open to make a few of us maintainers of Pendulum, that I will
> ask
> > if
> > >> > some
> > >> > > > of the maintainers here in Airflow would like to step up (I
> would
> > be
> > >> > > > willing to - for one) and help to at least move Pendulum through
> > >> 3.12
> > >> > and
> > >> > > > maybe keep it running for as long as it will be needed to
> maintain
> > >> it.
> > >> > > >
> > >> > > > I think if they would be open to some of us helping them - this
> > >> might
> > >> > be
> > >> > > > actually the "cheapest" option for us to be honest at least in a
> > >> > mid-term
> > >> > > > if we could gain influence and even take part and speed up
> > releases
> > >> and
> > >> > > > merging of issues that are blocking us (or would be blocking us
> in
> > >> the
> > >> > > > future).
> > >> > > >
> > >> > > > I tried to be very friendly but candid and direct on what kind
> of
> > >> > > problems
> > >> > > > it creates us and also expressed our thanks for supporting us
> for
> > so
> > >> > long
> > >> > > > (I do believe they deserve it). I think the sole (pretty much)
> > >> > > maintainer
> > >> > > > of Pendulum might really have a hard time maintaining and
> keeping
> > >> it up
> > >> > > for
> > >> > > > years - without being paid for and thanked. I hope - maybe all
> > that
> > >> > they
> > >> > > > need are a few words of encouragement and thanks and realising
> > that
> > >> > > others
> > >> > > > depend on their work - but also seeing that they are understood,
> > >> > > empathised
> > >> > > > with and that there are viable alternatives they might follow
> > might
> > >> be
> > >> > > > helpful for them to be also candid and respond. I hope my
> > intentions
> > >> > will
> > >> > > > be understood - that I am not complaining, but even trying to
> > help.
> > >> > > >
> > >> > > > Let\s see where it gets us.
> > >> > > >
> > >> > > > J.
> > >> > > >
> > >> > > > On Fri, Nov 17, 2023 at 5:32 PM Andrey Anshin <
> > >> > andrey.anshin@taragol.is>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Just to clarify I'd like us to consider the possibility that
> no
> > >> new
> > >> > > > > pendulum would be released or released at the end of 2024,
> like
> > a
> > >> > > > > pessimistic scenario:
> > >> > > > > - What should we do in this case?
> > >> > > > > - Work out a backup plan.
> > >> > > > >
> > >> > > > >
> > >> > > > > ----
> > >> > > > > Best Wishes
> > >> > > > > *Andrey Anshin*
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > On Fri, 17 Nov 2023 at 16:33, Jarek Potiuk <ja...@potiuk.com>
> > >> wrote:
> > >> > > > >
> > >> > > > > > Also I think TP - had a document in the past (years ago)
> > >> > > describing a
> > >> > > > > > draft of a more complete alternative we can take to approach
> > >> > datetime
> > >> > > > vs.
> > >> > > > > > pendulum dichotomy. I cannot easily find the document and
> > >> > discussion
> > >> > > -
> > >> > > > > but
> > >> > > > > > I do remember it was proposing some interesting changes in
> the
> > >> > > approach
> > >> > > > > of
> > >> > > > > > Airflow to have an abstraction layer on top of it (as far
> as I
> > >> > > > remember).
> > >> > > > > > Maybe we can resurrect that idea if TP might find the
> > proposal ?
> > >> > > > > >
> > >> > > > > > On Fri, Nov 17, 2023 at 1:06 PM Bolke de Bruin <
> > >> bdbruin@gmail.com>
> > >> > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Hi,
> > >> > > > > > >
> > >> > > > > > > I agree that the current speed of development of Pendulum
> > >> leaves
> > >> > > > > > something
> > >> > > > > > > to be desired. However, I think we should not
> underestimate
> > >> the
> > >> > > > effort
> > >> > > > > of
> > >> > > > > > > replacing it. It is not just a matter of
> > >> %s/pendulum/datetime/g
> > >> > so
> > >> > > to
> > >> > > > > > say.
> > >> > > > > > > If we are *truly* thinking about moving to native
> datetime /
> > >> > > zoneinfo
> > >> > > > > etc
> > >> > > > > > > we need *extensive* tests, basically copying what pendulum
> > >> does
> > >> > to
> > >> > > > > check
> > >> > > > > > > its behavior. The reason is that native implementations in
> > the
> > >> > past
> > >> > > > > made
> > >> > > > > > > serious mistakes and I do not put a lot of trust in them.
> > >> > > > > > >
> > >> > > > > > > An abstraction or vendoring in could be alternatives, but
> > they
> > >> > > bring
> > >> > > > > > their
> > >> > > > > > > own significant challenges.
> > >> > > > > > >
> > >> > > > > > > I have re-raised the issue here:
> > >> > > > > > >
> > >> > > > > > > https://github.com/sdispater/pendulum/issues/753
> > >> > > > > > >
> > >> > > > > > > The upside is that it seems the amount of issues with the
> > >> beta is
> > >> > > > > > limited,
> > >> > > > > > > so hopefully the maintainers can spend a couple of cycles
> to
> > >> > > address
> > >> > > > > > them.
> > >> > > > > > >
> > >> > > > > > > Bolke
> > >> > > > > > >
> > >> > > > > > > On Fri, 17 Nov 2023 at 10:28, Andrey Anshin <
> > >> > > > andrey.anshin@taragol.is>
> > >> > > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > There is no changes in stable pendulum so let's try to
> > >> continue
> > >> > > > this
> > >> > > > > > > > discussion and start think about "Plan B"
> > >> > > > > > > >
> > >> > > > > > > > Just a reminder:
> > >> > > > > > > > - pendulum 2.1.2 released 3 years ago (at the time
> Airflow
> > >> > > 1.10.x)
> > >> > > > > > > > - pendulum 2 doesn't work well in Python 3.12, this is a
> > >> > > > showstopper
> > >> > > > > > for
> > >> > > > > > > > the support Python 3.12
> > >> > > > > > > > - pendulum 2 have memory leaks (
> > >> > > > > > > > https://github.com/sdispater/pendulum/issues/720), and
> > >> Airflow
> > >> > > use
> > >> > > > > > > > approach
> > >> > > > > > > > to achieve this leaks, especially in K8S executor
> > >> > > > > > > > - pendulum 2 doesn't use system tzdata by default, but
> we
> > >> have
> > >> > a
> > >> > > > > > > > workaround, thanks Bolke for the documentation
> > >> > > > > > > > - pendulum it is a core dependency of Airflow, and I
> guess
> > >> we
> > >> > > can't
> > >> > > > > > > > remove/replace it without breaking changes.
> > >> > > > > > > >
> > >> > > > > > > > So my proposal if things won't change in the near future
> > >> then
> > >> > we
> > >> > > > need
> > >> > > > > > to
> > >> > > > > > > > start removing the pendulum from the core and replace it
> > >> with
> > >> > the
> > >> > > > > > native
> > >> > > > > > > > datetime / zoneinfo / tzinfo. But maybe we have another
> > way
> > >> to
> > >> > > > > resolve
> > >> > > > > > > this
> > >> > > > > > > > without breaking changes? Because for me it would be a
> > >> little
> > >> > > weird
> > >> > > > > if
> > >> > > > > > > > removal pendulum would be a main driver for release
> > Airflow
> > >> 3
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > ----
> > >> > > > > > > > Best Wishes
> > >> > > > > > > > *Andrey Anshin*
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On Thu, 28 Sept 2023 at 13:01, Andrey Anshin <
> > >> > > > > andrey.anshin@taragol.is
> > >> > > > > > >
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > This discussion is more about the known problem of
> > >> pendulum
> > >> > and
> > >> > > > how
> > >> > > > > > we
> > >> > > > > > > > > could deal with it and maybe how we (as Community)
> might
> > >> help
> > >> > > > > autor.
> > >> > > > > > > > >
> > >> > > > > > > > > The library is mostly supported by a single author
> > >> Sébastien
> > >> > > > > Eustace
> > >> > > > > > (
> > >> > > > > > > > > https://github.com/sdispater) and it seems like we
> bump
> > >> into
> > >> > > the
> > >> > > > > > > > > situation which is described in xkcd #2347 (
> > >> > > > > > > > > https://imgs.xkcd.com/comics/dependency.png). To be
> > >> honest
> > >> > it
> > >> > > is
> > >> > > > > not
> > >> > > > > > > > > something new when library mainly supported by one
> > author
> > >> so
> > >> > > > there
> > >> > > > > is
> > >> > > > > > > > > always a risk that the library will no longer be
> > >> supported /
> > >> > > > > > abandoned
> > >> > > > > > > > > And if takes in account that pendulum provides core
> > >> > > functionality
> > >> > > > > in
> > >> > > > > > > > > Airflow it could have dramatical impact in the future.
> > >> > > > > > > > >
> > >> > > > > > > > > Pendulum is a really nice library which helps a lot of
> > >> > > developers
> > >> > > > > to
> > >> > > > > > > work
> > >> > > > > > > > > with dates/datetimes. However there is one major
> > problem,
> > >> the
> > >> > > > last
> > >> > > > > > > > release
> > >> > > > > > > > > of this library happened more than 3 years ago (
> > >> > > > > > > > > https://pypi.org/project/pendulum/#history) in the
> time
> > >> when
> > >> > > > > Airflow
> > >> > > > > > > > > 1.10.11 was released
> > >> > > > > > > > >
> > >> > > > > > > > > Fortunately, the project is not abandoned and on a
> > regular
> > >> > > basis
> > >> > > > > > > commits
> > >> > > > > > > > > add into the master branch. However these commits are
> > not
> > >> > > > included
> > >> > > > > > into
> > >> > > > > > > > any
> > >> > > > > > > > > final release and that's why some things related to
> > >> datetime
> > >> > > > don't
> > >> > > > > > work
> > >> > > > > > > > as
> > >> > > > > > > > > expected in Airflow. There are list of known (for me)
> > >> issues
> > >> > > > which
> > >> > > > > > are
> > >> > > > > > > > > affect Airflow
> > >> > > > > > > > >
> > >> > > > > > > > > *Memory Leak on parse*:
> > >> > > > > > > > > - https://github.com/sdispater/pendulum/issues/720,
> > this
> > >> one
> > >> > > > fixed
> > >> > > > > > 2
> > >> > > > > > > > > years ago but not available yet (
> > >> > > > > > > > > https://github.com/sdispater/pendulum/pull/563).
> Since
> > we
> > >> > use
> > >> > > > > parse
> > >> > > > > > > > dates
> > >> > > > > > > > > in airflow codebase: datetime parameters and datetime
> in
> > >> logs
> > >> > > > this
> > >> > > > > > one
> > >> > > > > > > > > could be a reason for memory leakage in Airflow:
> > >> > > > > > > > > - https://github.com/apache/airflow/discussions/24694
> > >> > > > > > > > > - https://github.com/apache/airflow/discussions/28597
> > >> > > > > > > > >
> > >> > > > > > > > > *Incorrect time zones*, known issues and should be
> > already
> > >> > > fixed
> > >> > > > in
> > >> > > > > > > > > master branch
> > >> > > > > > > > > - https://github.com/sdispater/pendulum/issues/700,
> > >> Mexico
> > >> > do
> > >> > > > not
> > >> > > > > > use
> > >> > > > > > > > DST
> > >> > > > > > > > > anymore
> > >> > > > > > > > > - https://github.com/sdispater/pendulum/issues/706,
> > Egypt
> > >> > > > > reinstate
> > >> > > > > > > DST
> > >> > > > > > > > >
> > >> > > > > > > > > We add clarification in
> > >> > > > > https://github.com/apache/airflow/pull/30467
> > >> > > > > > ,
> > >> > > > > > > > > however it seems like there is no other way rather
> than
> > >> > > patching
> > >> > > > > > > Pendulum
> > >> > > > > > > > > right now.
> > >> > > > > > > > >
> > >> > > > > > > > > All these issues should be solved as soon as pendulum
> 3
> > is
> > >> > > > > released.
> > >> > > > > > > The
> > >> > > > > > > > > current announced estimation is end of september/
> > >> beginning
> > >> > of
> > >> > > > > > October:
> > >> > > > > > > > >
> > >> > > > > > >
> > >> > > > >
> > >> > >
> > >>
> > https://github.com/sdispater/pendulum/issues/600#issuecomment-1711299677
> > >> > > > > > > > >
> > >> > > > > > > > > So in theory we would have a fixed version of pendulum
> > >> soon,
> > >> > > and
> > >> > > > it
> > >> > > > > > > might
> > >> > > > > > > > > break something in Airflow but from my point of view
> it
> > is
> > >> > > better
> > >> > > > > > than
> > >> > > > > > > > > current status.
> > >> > > > > > > > >
> > >> > > > > > > > > However there might be a situation where the release
> of
> > >> the
> > >> > > > > pendulum
> > >> > > > > > > > would
> > >> > > > > > > > > be postponed, so maybe better to have a backup plan.
> > What
> > >> > could
> > >> > > > we
> > >> > > > > do
> > >> > > > > > > in
> > >> > > > > > > > > this case?
> > >> > > > > > > > >
> > >> > > > > > > > > Maybe we should start to use zoneinfo.ZoneInfo instead
> > of
> > >> > > > pendulum
> > >> > > > > > > > > datetime?
> > https://github.com/apache/airflow/issues/19450
> > >> > > > > > > > > Pros:
> > >> > > > > > > > > - stdlib (python 3.9+)
> > >> > > > > > > > > - In pendulum 3.0 Timezone based on zoneinfo.Zoneinfo
> > >> > > > > > > > >
> > >> > > > > > > > > Cons:
> > >> > > > > > > > > - Current serialization model can't deal with backport
> > >> > > packages.
> > >> > > > > E.g.
> > >> > > > > > > > > timezone which are serialized in backport_zoneinfo
> can't
> > >> be
> > >> > > > > > > deserialized
> > >> > > > > > > > in
> > >> > > > > > > > > zoneinfo
> > >> > > > > > > > >
> > >> > > > > > > > > Maybe we should replace parse datetime with another
> > >> solution.
> > >> > > > Does
> > >> > > > > > > anyone
> > >> > > > > > > > > know a good replacement?
> > >> > > > > > > > >
> > >> > > > > > > > > Maybe someone from Airflow Community could propose
> their
> > >> help
> > >> > > > with
> > >> > > > > > > > > maintenance of library:
> > >> > > > > > > > > - https://github.com/sdispater/pendulum/issues/590
> > >> > > > > > > > >
> > >> > > > > > > > > Maybe we should get rid of the pendulum at all, as a
> > last
> > >> > > resort
> > >> > > > > > > > solution.
> > >> > > > > > > > > I can't imagine how we could do that, because a lot of
> > >> stuff
> > >> > > > > depends
> > >> > > > > > on
> > >> > > > > > > > the
> > >> > > > > > > > > pendulum and removing it would be a breaking change.
> > >> > > > > > > > >
> > >> > > > > > > > > ----
> > >> > > > > > > > > Best Wishes
> > >> > > > > > > > > *Andrey Anshin*
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > --
> > >> > > > > > >
> > >> > > > > > > --
> > >> > > > > > > Bolke de Bruin
> > >> > > > > > > bdbruin@gmail.com
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>