[LAZY CONSENSUS] Bring back min-airflow-version-support for preinstalled providers (Bazel)

[Jarek Potiuk <ja...@potiuk.com>]

Hello everyone,

I had a short discussion with Elad and unless there is any objection, we
decided to bring back support for "min-airlfow-version" support for
pre-installed providers (common.sql, ftp, imap, http, sqlite). We will have
to release them ad-hoc now and `yank` the latest versions of those released
yesterday,

PR here: https://github.com/apache/airflow/pull/31469

Some context:

In the last wave of providers we bumped min-airlfow-version to 2.4 and
added mechanism to verify min-airflow version is ok while importing, but it
turned out that there are cases where installing just old version of
airflow (with no constraints) brings the latest version of those providers
and causes new installation of airflow to fail. This is far too common to
ignore or require to use constraints, unfortunately

We do not have min-airlfow-version in the preinstalled providers for one
reason only. For some tools that are NOT conforming to standards (such as
bazel), having min-airflow-version for those providers causes circular
dependency (even though technically dependencies in PyPI can - and often
are - circular, because dependencies in Python are not DAG and can contain
cycles.

This was in response to this issue in 2021
https://github.com/apache/airflow/issues/17795

However - this is really a bazel issue, and on top of it - it's recognized
as such and being fixed very recently in
https://github.com/bazelbuild/rules_python/issues/1188 because there are
other packages that have similar problems (pytorch and triton being
popular). Also Bazel is not that popular in the Python world.

Therefore - rather than trying to workaround the problem of bazel, we will
encourage them to merge and release the fix (I already did
https://github.com/bazelbuild/rules_python/pull/1166#issuecomment-1558690218)
and call it out in our installation instructions, that bazel installation
might lead to problems like that.

If bazel does not fix it, this will only be a problem for Future
installations of airflow in a few months most likely. It will not impact
current bazel users installing old versions of Airflow (actually they might
start having problems now if we do not fix it and yank the 5 providers
released yesterday)

Please let us know if you have any comments, and shout if you think this is
a bad idea.


J.

Re: [LAZY CONSENSUS] Bring back min-airflow-version-support for preinstalled providers (Bazel)

[Jarek Potiuk <ja...@potiuk.com>]

We will start preparing the release ASAP with Elad, and we will run a
parallel vote on those released providers, so if there will be no
objections the lazy consensus will conclude 72 hours from now (Friday 10am
CEST).

On Tue, May 23, 2023 at 10:01 AM Jarek Potiuk <ja...@potiuk.com> wrote:

> Hello everyone,
>
> I had a short discussion with Elad and unless there is any objection, we
> decided to bring back support for "min-airlfow-version" support for
> pre-installed providers (common.sql, ftp, imap, http, sqlite). We will have
> to release them ad-hoc now and `yank` the latest versions of those released
> yesterday,
>
> PR here: https://github.com/apache/airflow/pull/31469
>
> Some context:
>
> In the last wave of providers we bumped min-airlfow-version to 2.4 and
> added mechanism to verify min-airflow version is ok while importing, but it
> turned out that there are cases where installing just old version of
> airflow (with no constraints) brings the latest version of those providers
> and causes new installation of airflow to fail. This is far too common to
> ignore or require to use constraints, unfortunately
>
> We do not have min-airlfow-version in the preinstalled providers for one
> reason only. For some tools that are NOT conforming to standards (such as
> bazel), having min-airflow-version for those providers causes circular
> dependency (even though technically dependencies in PyPI can - and often
> are - circular, because dependencies in Python are not DAG and can contain
> cycles.
>
> This was in response to this issue in 2021
> https://github.com/apache/airflow/issues/17795
>
> However - this is really a bazel issue, and on top of it - it's recognized
> as such and being fixed very recently in
> https://github.com/bazelbuild/rules_python/issues/1188 because there are
> other packages that have similar problems (pytorch and triton being
> popular). Also Bazel is not that popular in the Python world.
>
> Therefore - rather than trying to workaround the problem of bazel, we will
> encourage them to merge and release the fix (I already did
> https://github.com/bazelbuild/rules_python/pull/1166#issuecomment-1558690218)
> and call it out in our installation instructions, that bazel installation
> might lead to problems like that.
>
> If bazel does not fix it, this will only be a problem for Future
> installations of airflow in a few months most likely. It will not impact
> current bazel users installing old versions of Airflow (actually they might
> start having problems now if we do not fix it and yank the 5 providers
> released yesterday)
>
> Please let us know if you have any comments, and shout if you think this
> is a bad idea.
>
>
> J.
>
>
>
>
>
>
>