You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <ja...@potiuk.com> on 2025/07/22 15:04:21 UTC
[DISCUSS] Removing dev dependencies from PyPI constraints ?
Hello here,
*Tl;DR; We had an interesting discussion on Slack today and I wanted to
bring it here to discuss what we should do - whether we should keep devel
dependencies in our PyPI constraints or not.*
*A bit of context:*
In our CI we generate several types of constraints. one of them are
"source" constraints that we use during our CI testing - including all our
testing tools, mypy, ruff etc., and another type is "PyPI" constraint -
which is more <What should be the "golden" versions of dependencies when
you install Airflow using `pip install` for production>.
The main difference between the two (roughly) is that "source constraints"
only uses dependencies from "branch tip" ("main", "v3-0-test"), where "PyPI
constraints" uses "Airflow core" branch-tip dependencies - but all the
providers are installed from PyPI. There are few nuances when some packages
are being released but basically - for example for Airflow 3.0.3, "PyPI"
constraints is "what is the current set of dependencies and providers when
we install airflow from PyPI at the moment of release".
So target for "source" constraints are "Airflow contributors" where target
for "PyPI constraints" are "Airflow users".
*The problem*
In the slack discussion
https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1753169914722049 -
slack user @dashmug (sorry - do not know name) noticed that the "PyPI"
constraints also contain Airflow development dependencies listed (like ruff
or mypy). That made me think a bit and I quickly came up with PR that could
"fix" it - i.e. only keep the "real" constraints in.
https://github.com/apache/airflow/pull/53631 - it's as easy as adding the
`--exact` (and `--strict` for verification) flag to the `uv pip install`
command we use.
I agree it's a bit confusing to have the devel dependencies there - it is
not the "intention", but there is very little problem with it - the fact
that mypy or ruff are listed with a version in constraints, does not mean
it must be installed - constraints mainly say "if you install this, it
should be this version" . And there is no problem whatsoever (and it is
even recommended by us) to have "airflow with its deps" installed with
constraints, and then anything else you need to install or update -
without. So other than potential confusion, it usually does not cause harm.
Now... It might seem that removing devel deps has no side effects and we
should **just do it** - but it's not that easy. Main problem is that those
devel dependencies might "hold" other dependencies (directly or
transitively) from being upgraded - and when we remove the devel
dependencies and do the resolution again - we might have a DIFFERENT set of
dependencies, simply because removing the devel dependencies, might "free"
the other deps and they might get upgraded. Yeah - I know it's not obvious
for someone who did not spend 3 years solving dependency issues :).
And the problem with it is that because we need those devel deps to test
things - the upgraded dependencies might have never been tested in our CI -
because we never upgraded to later versions. It's a bit of
Heisenberg-effect. By using test tools we are changing the state of the
tested thing. Currently our "PyPI" constraints reflect the state of Airflow
"when the testing tools are installed" - and we are generally not able to
test the state of Airflow "without the testing tools" - by the sheer fact
that we do not have those tools :)
Just to illustrate the case - you can see what happens when we removed
devel deps:
https://github.com/apache/airflow/actions/runs/16444553234 - in summary you
can see a colored version that is a bit clearer but I dumped the diff below
from an example - Python 3.10 case. Left side < is "what was with testing
tools" and > is without them. You can see a LOT of test tools removed
(pytest, ruff and others - including dependencies used only by those tools)
But also you can see (likely some of the devel/test deps limited those)
* Authlib - upgraded from 1.3.1 to 1.6.1
* Google - genai upgraded from 1.2.0 to 1.20.0
* Httpx - upgraded from 0.27.0 to 0.28.1
* Pinotdb - upgraded from 5.6.0 to 5.7.0
* Sentry-sdk - upgraded from 2.33.1 to 2.33.2 (this is false positive as
it has just been released and new version was picked)
* validators - upgraded form 0.34.0 to 0.35.0
* weaviate-client - upgraded from 4.9.0 to 4.16.3
* websockets - upgraded from 14.2 to 15.0.1
This means that all the "upgraded" packages above have not been - likely
ever tested in our CI.
And when we put them in "versioned" constraints, it might mean that they
will be obviously failing.
So ... there is a risk connected with just generating the "non-devel"
constraints without additional tests. But that also means that the user who
does not use constraints might get those versions anyway - default
behaviour of `pip install airflow` will be to install those upgraded
versions above if you do not use constraints. But also that is no different
than installing a released airflow version a few weeks or months after
release without constraints - there will be tens of packages that were
released since our release and installing airflow without constraints will
pick those.
However - when we manually install RC candidates - say by running airflow
from rc1 image, we will use those "upgraded" dependencies - so some
"obvious" problems might be caught during the rc manual testing - where you
do not use test tools but run airflow "as if" it was installed in
production.
*The solution ? *
There are few ways we can proceed with it:
a) leave it as is - no big harm, potential confusion, by people who see it
- potential differences
b) remove devel following my PR and take the risk that we never tested (in
CI) the dependencies we set in the constraints. That's a bit risky, but we
can always reactively downgrade some of those packages after the fact if
users will start reporting the issues
c) potentially most complex thing - trying to remove devel dependencies
somewhat combining the two - for example removing all the deps that would
be removed in b) from a) - but that has also some edge cases - when
different versions of dependencies can add or upgrade their dependencies,
we are very likely to occasionally remove or add too much.
I wonder (for those who managed to read that far - likely not many - what
do you think here? What would be the best option ? Maybe there are other
options I have not thought about ?
J
-----
The diff for Python 3.10
34c34
< Authlib==1.3.1
---
> Authlib==1.6.1
64d63
< Sphinx==8.2.3
78d76
< aioresponses==0.7.8
82d79
< alabaster==1.0.0
198d194
< arro3-core==0.5.1
202d197
< astroid==3.3.11
211,212d205
< aws-sam-translator==1.99.0
< aws-xray-sdk==2.14.0
258,259d250
< cfgv==3.4.0
< cfn-lint==1.38.0
262d252
< checksumdir==1.2.0
277d266
< coverage==7.9.2
291d279
< deltalake==1.1.0
293d280
< diagrams==0.24.4
301,302d287
< docutils==0.21.2
< duckdb==1.3.2
308d292
< eralchemy2==1.4.1
311d294
< execnet==2.1.1
321,322d303
< flit==3.12.0
< flit_core==3.12.0
386c367
< google-genai==1.2.0
---
> google-genai==1.20.0
389d369
< graphql-core==3.2.6
398d377
< grpcio-tools==1.62.3
403,404d381
< hatch==1.14.1
< hatchling==1.27.0
413c390
< httpx==0.27.0
---
> httpx==0.28.1
418d394
< hyperlink==21.0.0
421,423d396
< icdiff==2.0.7
< id==1.5.0
< identify==2.6.12
426d398
< imagesize==1.4.1
430d401
< incremental==24.7.2
433,435d403
< iniconfig==2.1.0
< inputimeout==1.0.4
< ipdb==0.13.13
449d416
< joserfc==1.2.2
451d417
< jsonpatch==1.33
455,456d420
< jsonpointer==3.0.0
< jsonschema-path==0.3.4
463d426
< kerberos==1.3.1
465,466d427
< keyrings.alt==5.0.2
< kgb==7.2
494,495d454
< mmh3==5.1.0
< mongomock==4.3.0
497,498d455
< moto==5.1.8
< mpmath==1.3.0
508,512d464
< mypy-boto3-appflow==1.39.0
< mypy-boto3-rds==1.39.1
< mypy-boto3-redshift-data==1.39.0
< mypy-boto3-s3==1.39.5
< mypy==1.17.0
514c466
< mysql-connector-python==9.3.0
---
> mysql-connector-python==9.4.0
521,523d472
< networkx==3.5
< nh3==0.3.0
< nodeenv==1.9.1
528,529d476
< openapi-schema-validator==0.6.3
< openapi-spec-validator==0.7.2
560d506
< pathable==0.4.4
563d508
< pdbr==0.9.2
569,570c514
< pinotdb==5.6.0
< pipdeptree==2.28.0
---
> pinotdb==5.7.0
574d517
< plyvel==1.5.1
577,579d519
< pprintpp==0.4.0
< pre-commit-uv==4.1.4
< pre_commit==4.2.0
592d531
< py-partiql-parser==0.6.1
608d546
< pyenchant==3.2.2
610,611d547
< pygraphviz==1.14
< pyiceberg==0.9.1
622,632d557
< pytest-asyncio==0.25.0
< pytest-cov==6.2.1
< pytest-custom-exit-code==0.3.0
< pytest-icdiff==0.9
< pytest-instafail==0.5.0
< pytest-mock==3.14.1
< pytest-rerunfailures==15.1
< pytest-timeouts==1.2.1
< pytest-unordered==0.7.0
< pytest-xdist==3.8.0
< pytest==8.4.1
642d566
< python-on-whales==0.78.0
652d575
< readme_renderer==44.0
659d581
< requests-mock==1.12.1
664,665d585
< responses==0.25.7
< restructuredtext_lint==1.4.0
667,668d586
< rfc3339-validator==0.1.4
< rfc3986==2.0.0
670d587
< rich-click==1.8.9
674d590
< roman-numerals-py==3.1.0
679d594
< ruff==0.12.3
688d602
< semver==3.0.4
690,691c604
< sentinels==1.0.0
< sentry-sdk==2.33.1
---
> sentry-sdk==2.33.2
703d615
< snowballstemmer==3.0.1
709,725d620
< sphinx-argparse==0.5.2
< sphinx-autoapi==3.6.0
< sphinx-autobuild==2024.10.3
< sphinx-copybutton==0.5.2
< sphinx-jinja==2.0.2
< sphinx-rtd-theme==3.0.2
< sphinx_design==0.6.1
< sphinxcontrib-applehelp==2.0.0
< sphinxcontrib-devhelp==2.0.0
< sphinxcontrib-htmlhelp==2.1.0
< sphinxcontrib-httpdomain==1.8.1
< sphinxcontrib-jquery==4.1
< sphinxcontrib-jsmath==1.0.1
< sphinxcontrib-qthelp==2.0.0
< sphinxcontrib-redoc==1.6.0
< sphinxcontrib-serializinghtml==2.0.0
< sphinxcontrib-spelling==8.0.1
737d631
< strictyaml==1.7.3
740d633
< sympy==1.14.0
752d644
< time-machine==2.16.0
755d646
< tomli_w==1.2.0
758d648
< towncrier==24.8.0
762,763d651
< trove-classifiers==2025.5.9.12
< twine==6.1.0
765,774d652
< types-Deprecated==1.2.15.20250304
< types-Markdown==3.8.0.20250708
< types-PyMySQL==1.1.0.20250711
< types-PyYAML==6.0.12.20250516
< types-aiofiles==24.1.0.20250708
< types-certifi==2021.10.8.3
< types-cffi==1.17.0.20250523
< types-croniter==6.0.0.20250626
< types-docutils==0.21.0.20250722
< types-paramiko==3.5.0.20250708
776,778d653
< types-pyOpenSSL==24.1.0.20240722
< types-python-dateutil==2.9.0.20250708
< types-python-slugify==8.0.2.20240310
780d654
< types-redis==4.6.0.20241004
782,784d655
< types-setuptools==80.9.0.20250529
< types-tabulate==0.9.0.20241207
< types-toml==0.10.8.20240310
794d664
< userpath==1.9.2
799c669
< validators==0.34.0
---
> validators==0.35.0
806c676
< weaviate-client==4.9.6
---
> weaviate-client==4.16.3
809c679
< websockets==14.2
---
> websockets==15.0.1
815d684
< yamllint==1.37.1
Re: [DISCUSS] Removing dev dependencies from PyPI constraints ?
Posted by Jarek Potiuk <ja...@potiuk.com>.
3.1.0 + only.
On Fri, Aug 1, 2025 at 8:03 PM Elad Kalif <el...@apache.org> wrote:
> Will the change take effect for all future releases or just from 3.11
> moving forward?
>
> On Fri, Aug 1, 2025 at 8:57 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > OK. I will run LAZY_CONSENSUS then
> >
> > On Fri, Jul 25, 2025 at 9:22 AM Pavankumar Gopidesu <
> > gopidesupavan@gmail.com>
> > wrote:
> >
> > > Thanks Jarek,
> > >
> > > I am fine with option b as far as if we find anything while testing we
> > can
> > > downgrade.
> > >
> > > Pavan
> > >
> > > On Thu, Jul 24, 2025 at 10:44 PM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > >
> > > > Would love to hear more if anyone has an opinion :). Yep. It's an
> > > advanced
> > > > topic and I can make arbitrary decision :) ...
> > > >
> > > > I am thinking now that it should be quite "OK" to accept those
> > > "potentially
> > > > untested" versions - especially that they will be tested during
> manual
> > > > testing. Unless someone objects i will likely update PR with some
> > > changelog
> > > > / newsfragment to announce the change in 3.1. I will run LAZY
> CONSENSUS
> > > > soon if I don't hear any opposition.
> > > >
> > > > J.
> > > >
> > > >
> > > > On Tue, Jul 22, 2025 at 5:52 PM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > > >
> > > > > > Exactly to remove the dev requirements airflow has from
> interfering
> > > > with
> > > > > our own dev requirements. Though this approach notably doesn't
> change
> > > any
> > > > > of the used non-dev requirements, it's not clear to me (and I'm
> > > probably
> > > > > just missing something) why your diff includes non-dev
> requirements.
> > > > >
> > > > > The thing is that I do not "remove" devel features - because due to
> > > > > transitive dependencies I would have to go deep down the rabbit
> hole
> > > > which
> > > > > dependencies are "devel" and which not. So what I do, I ask `uv` to
> > do
> > > > the
> > > > > job for me and repeat the resolution but with the `--exact` flag
> > > > (combined
> > > > > with the already used `--resolution higest` and `--no-sources`).
> > > > >
> > > > > Apparently in this case `uv` will sometimes decide that it's ok to
> > bump
> > > > > some dependencies that are left after initial resolution. The thing
> > is
> > > > that
> > > > > uv will not install some versions of some packages if they are
> > > > conflicting
> > > > > with already installed packages. What I believe `uv` does when I
> use
> > > the
> > > > *--exact
> > > > > *flag of `uv pip install` it "fixes" the initial resolution. It
> finds
> > > out
> > > > > what is needed, removes those packages that are not needed, and
> then
> > it
> > > > > resolves again and improves the resolution (that would be my guess
> at
> > > > least
> > > > > - it could be done slightly differently and in a single step I
> guess.
> > > > >
> > > > > Essentially the difference between the left and right side is:
> > > > >
> > > > > a) install absolutely everything from the workspace
> > > > >
> > > > > b) do this:
> > > > > left: uv pip install --no-sources 'apache-airflow[all]'
> > > > > 'apache-airflow-core[all]' apache-airflow-task-sdk ./airflow-ctl
> > > > > --reinstall --resolution highest --find-links file:///dist
> > > > > right: uv pip install --no-sources *--exact --strict*
> > > > > 'apache-airflow[all]' 'apache-airflow-core[all]'
> > > apache-airflow-task-sdk
> > > > > ./airflow-ctl --reinstall --resolution highest --find-links
> > > file:///dist
> > > > >
> > > > > In the current state content of the fille://dist should be the same
> > in
> > > > > both cases:
> > > > >
> > > > > apache-airflow-providers-edge3 (1.1.2): wheel -
> > > > > apache_airflow_providers_edge3-1.1.2-py3-none-any.whl
> > > > > apache-airflow-providers-keycloak (1.0.0): wheel -
> > > > > apache_airflow_providers_keycloak-1.0.0-py3-none-any.whl
> > > > > apache-airflow-providers-snowflake (6.5.1): wheel -
> > > > > apache_airflow_providers_snowflake-6.5.1-py3-none-any.whl
> > > > >
> > > > > This is a bit of in-process quirk of the "PyPI" resolution - those
> > are
> > > > the
> > > > > three packages that we have updated versions of in the sources, but
> > > they
> > > > > are not released yet. But they should be essentially the same in
> both
> > > > > cases. So the difference between left and right is `--exact
> --strict`
> > > > > (strict I added to make sure that we are not removing something
> that
> > is
> > > > > still needed).
> > > > >
> > > > > J.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Jul 22, 2025 at 5:22 PM Damian Shaw <
> > > > dshaw@striketechnologies.com>
> > > > > wrote:
> > > > >
> > > > >> As I user of Airflow I have ended up implementing this for my
> > > > >> environments:
> > > > >>
> > > > >> pip install apache-airflow[all or select-extras] -c {official
> > > > >> constraints} > smaller-run-time-constraints.txt
> > > > >> pip-compile {all my requirements including development}.txt -c
> > > > >> smaller-run-time-constraints.txt
> > > > >>
> > > > >> Exactly to remove the dev requirements airflow has from
> interfering
> > > with
> > > > >> our own dev requirements. Though this approach notably doesn't
> > change
> > > > any
> > > > >> of the used non-dev requirements, it's not clear to me (and I'm
> > > probably
> > > > >> just missing something) why your diff includes non-dev
> requirements.
> > > > >>
> > > > >> Damian
> > > > >>
> > > > >> -----Original Message-----
> > > > >> From: Jarek Potiuk <ja...@potiuk.com>
> > > > >> Sent: Tuesday, July 22, 2025 11:04 AM
> > > > >> To: dev@airflow.apache.org
> > > > >> Subject: [DISCUSS] Removing dev dependencies from PyPI
> constraints ?
> > > > >>
> > > > >> Hello here,
> > > > >>
> > > > >>
> > > > >> *Tl;DR; We had an interesting discussion on Slack today and I
> wanted
> > > to
> > > > >> bring it here to discuss what we should do - whether we should
> keep
> > > > devel
> > > > >> dependencies in our PyPI constraints or not.*
> > > > >>
> > > > >> *A bit of context:*
> > > > >> In our CI we generate several types of constraints. one of them
> are
> > > > >> "source" constraints that we use during our CI testing - including
> > all
> > > > our
> > > > >> testing tools, mypy, ruff etc., and another type is "PyPI"
> > constraint
> > > -
> > > > >> which is more <What should be the "golden" versions of
> dependencies
> > > when
> > > > >> you install Airflow using `pip install` for production>.
> > > > >>
> > > > >> The main difference between the two (roughly) is that "source
> > > > constraints"
> > > > >> only uses dependencies from "branch tip" ("main", "v3-0-test"),
> > where
> > > > >> "PyPI constraints" uses "Airflow core" branch-tip dependencies -
> but
> > > all
> > > > >> the providers are installed from PyPI. There are few nuances when
> > some
> > > > >> packages are being released but basically - for example for
> Airflow
> > > > 3.0.3,
> > > > >> "PyPI"
> > > > >> constraints is "what is the current set of dependencies and
> > providers
> > > > >> when we install airflow from PyPI at the moment of release".
> > > > >>
> > > > >> So target for "source" constraints are "Airflow contributors"
> where
> > > > >> target for "PyPI constraints" are "Airflow users".
> > > > >>
> > > > >>
> > > > >> *The problem*
> > > > >> In the slack discussion
> > > > >>
> > >
> https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1753169914722049
> > > > >> - slack user @dashmug (sorry - do not know name) noticed that the
> > > > "PyPI"
> > > > >> constraints also contain Airflow development dependencies listed
> > (like
> > > > >> ruff or mypy). That made me think a bit and I quickly came up with
> > PR
> > > > that
> > > > >> could "fix" it - i.e. only keep the "real" constraints in.
> > > > >> https://github.com/apache/airflow/pull/53631 - it's as easy as
> > adding
> > > > >> the `--exact` (and `--strict` for verification) flag to the `uv
> pip
> > > > >> install` command we use.
> > > > >>
> > > > >> I agree it's a bit confusing to have the devel dependencies there
> -
> > it
> > > > is
> > > > >> not the "intention", but there is very little problem with it -
> the
> > > fact
> > > > >> that mypy or ruff are listed with a version in constraints, does
> not
> > > > mean
> > > > >> it must be installed - constraints mainly say "if you install
> this,
> > it
> > > > >> should be this version" . And there is no problem whatsoever (and
> it
> > > is
> > > > >> even recommended by us) to have "airflow with its deps" installed
> > with
> > > > >> constraints, and then anything else you need to install or update
> -
> > > > >> without. So other than potential confusion, it usually does not
> > cause
> > > > harm.
> > > > >>
> > > > >> Now... It might seem that removing devel deps has no side effects
> > and
> > > we
> > > > >> should **just do it** - but it's not that easy. Main problem is
> that
> > > > those
> > > > >> devel dependencies might "hold" other dependencies (directly or
> > > > >> transitively) from being upgraded - and when we remove the devel
> > > > >> dependencies and do the resolution again - we might have a
> DIFFERENT
> > > > set of
> > > > >> dependencies, simply because removing the devel dependencies,
> might
> > > > "free"
> > > > >> the other deps and they might get upgraded. Yeah - I know it's not
> > > > >> obvious for someone who did not spend 3 years solving dependency
> > > issues
> > > > :).
> > > > >>
> > > > >> And the problem with it is that because we need those devel deps
> to
> > > test
> > > > >> things - the upgraded dependencies might have never been tested in
> > our
> > > > CI -
> > > > >> because we never upgraded to later versions. It's a bit of
> > > > >> Heisenberg-effect. By using test tools we are changing the state
> of
> > > the
> > > > >> tested thing. Currently our "PyPI" constraints reflect the state
> of
> > > > Airflow
> > > > >> "when the testing tools are installed" - and we are generally not
> > able
> > > > to
> > > > >> test the state of Airflow "without the testing tools" - by the
> sheer
> > > > fact
> > > > >> that we do not have those tools :)
> > > > >>
> > > > >> Just to illustrate the case - you can see what happens when we
> > removed
> > > > >> devel deps:
> > > > >> https://github.com/apache/airflow/actions/runs/16444553234 - in
> > > summary
> > > > >> you can see a colored version that is a bit clearer but I dumped
> the
> > > > diff
> > > > >> below from an example - Python 3.10 case. Left side < is "what was
> > > with
> > > > >> testing tools" and > is without them. You can see a LOT of test
> > tools
> > > > >> removed (pytest, ruff and others - including dependencies used
> only
> > by
> > > > >> those tools)
> > > > >>
> > > > >> But also you can see (likely some of the devel/test deps limited
> > > those)
> > > > >>
> > > > >> * Authlib - upgraded from 1.3.1 to 1.6.1
> > > > >> * Google - genai upgraded from 1.2.0 to 1.20.0
> > > > >> * Httpx - upgraded from 0.27.0 to 0.28.1
> > > > >> * Pinotdb - upgraded from 5.6.0 to 5.7.0
> > > > >> * Sentry-sdk - upgraded from 2.33.1 to 2.33.2 (this is false
> > positive
> > > > as
> > > > >> it has just been released and new version was picked)
> > > > >> * validators - upgraded form 0.34.0 to 0.35.0
> > > > >> * weaviate-client - upgraded from 4.9.0 to 4.16.3
> > > > >> * websockets - upgraded from 14.2 to 15.0.1
> > > > >>
> > > > >>
> > > > >> This means that all the "upgraded" packages above have not been -
> > > likely
> > > > >> ever tested in our CI.
> > > > >>
> > > > >> And when we put them in "versioned" constraints, it might mean
> that
> > > they
> > > > >> will be obviously failing.
> > > > >>
> > > > >> So ... there is a risk connected with just generating the
> > "non-devel"
> > > > >> constraints without additional tests. But that also means that the
> > > user
> > > > >> who does not use constraints might get those versions anyway -
> > default
> > > > >> behaviour of `pip install airflow` will be to install those
> upgraded
> > > > >> versions above if you do not use constraints. But also that is no
> > > > different
> > > > >> than installing a released airflow version a few weeks or months
> > after
> > > > >> release without constraints - there will be tens of packages that
> > were
> > > > >> released since our release and installing airflow without
> > constraints
> > > > will
> > > > >> pick those.
> > > > >>
> > > > >> However - when we manually install RC candidates - say by running
> > > > airflow
> > > > >> from rc1 image, we will use those "upgraded" dependencies - so
> some
> > > > >> "obvious" problems might be caught during the rc manual testing -
> > > where
> > > > you
> > > > >> do not use test tools but run airflow "as if" it was installed in
> > > > >> production.
> > > > >>
> > > > >> *The solution ? *
> > > > >>
> > > > >> There are few ways we can proceed with it:
> > > > >>
> > > > >> a) leave it as is - no big harm, potential confusion, by people
> who
> > > see
> > > > >> it
> > > > >> - potential differences
> > > > >>
> > > > >> b) remove devel following my PR and take the risk that we never
> > tested
> > > > (in
> > > > >> CI) the dependencies we set in the constraints. That's a bit
> risky,
> > > but
> > > > >> we can always reactively downgrade some of those packages after
> the
> > > > fact if
> > > > >> users will start reporting the issues
> > > > >>
> > > > >> c) potentially most complex thing - trying to remove devel
> > > dependencies
> > > > >> somewhat combining the two - for example removing all the deps
> that
> > > > would
> > > > >> be removed in b) from a) - but that has also some edge cases -
> when
> > > > >> different versions of dependencies can add or upgrade their
> > > > dependencies,
> > > > >> we are very likely to occasionally remove or add too much.
> > > > >>
> > > > >>
> > > > >> I wonder (for those who managed to read that far - likely not
> many -
> > > > what
> > > > >> do you think here? What would be the best option ? Maybe there are
> > > other
> > > > >> options I have not thought about ?
> > > > >>
> > > > >>
> > > > >> J
> > > > >>
> > > > >>
> > > > >> -----
> > > > >>
> > > > >> The diff for Python 3.10
> > > > >>
> > > > >>
> > > > >>
> > > > >> 34c34
> > > > >> < Authlib==1.3.1
> > > > >> ---
> > > > >> > Authlib==1.6.1
> > > > >> 64d63
> > > > >> < Sphinx==8.2.3
> > > > >> 78d76
> > > > >> < aioresponses==0.7.8
> > > > >> 82d79
> > > > >> < alabaster==1.0.0
> > > > >> 198d194
> > > > >> < arro3-core==0.5.1
> > > > >> 202d197
> > > > >> < astroid==3.3.11
> > > > >> 211,212d205
> > > > >> < aws-sam-translator==1.99.0
> > > > >> < aws-xray-sdk==2.14.0
> > > > >> 258,259d250
> > > > >> < cfgv==3.4.0
> > > > >> < cfn-lint==1.38.0
> > > > >> 262d252
> > > > >> < checksumdir==1.2.0
> > > > >> 277d266
> > > > >> < coverage==7.9.2
> > > > >> 291d279
> > > > >> < deltalake==1.1.0
> > > > >> 293d280
> > > > >> < diagrams==0.24.4
> > > > >> 301,302d287
> > > > >> < docutils==0.21.2
> > > > >> < duckdb==1.3.2
> > > > >> 308d292
> > > > >> < eralchemy2==1.4.1
> > > > >> 311d294
> > > > >> < execnet==2.1.1
> > > > >> 321,322d303
> > > > >> < flit==3.12.0
> > > > >> < flit_core==3.12.0
> > > > >> 386c367
> > > > >> < google-genai==1.2.0
> > > > >> ---
> > > > >> > google-genai==1.20.0
> > > > >> 389d369
> > > > >> < graphql-core==3.2.6
> > > > >> 398d377
> > > > >> < grpcio-tools==1.62.3
> > > > >> 403,404d381
> > > > >> < hatch==1.14.1
> > > > >> < hatchling==1.27.0
> > > > >> 413c390
> > > > >> < httpx==0.27.0
> > > > >> ---
> > > > >> > httpx==0.28.1
> > > > >> 418d394
> > > > >> < hyperlink==21.0.0
> > > > >> 421,423d396
> > > > >> < icdiff==2.0.7
> > > > >> < id==1.5.0
> > > > >> < identify==2.6.12
> > > > >> 426d398
> > > > >> < imagesize==1.4.1
> > > > >> 430d401
> > > > >> < incremental==24.7.2
> > > > >> 433,435d403
> > > > >> < iniconfig==2.1.0
> > > > >> < inputimeout==1.0.4
> > > > >> < ipdb==0.13.13
> > > > >> 449d416
> > > > >> < joserfc==1.2.2
> > > > >> 451d417
> > > > >> < jsonpatch==1.33
> > > > >> 455,456d420
> > > > >> < jsonpointer==3.0.0
> > > > >> < jsonschema-path==0.3.4
> > > > >> 463d426
> > > > >> < kerberos==1.3.1
> > > > >> 465,466d427
> > > > >> < keyrings.alt==5.0.2
> > > > >> < kgb==7.2
> > > > >> 494,495d454
> > > > >> < mmh3==5.1.0
> > > > >> < mongomock==4.3.0
> > > > >> 497,498d455
> > > > >> < moto==5.1.8
> > > > >> < mpmath==1.3.0
> > > > >> 508,512d464
> > > > >> < mypy-boto3-appflow==1.39.0
> > > > >> < mypy-boto3-rds==1.39.1
> > > > >> < mypy-boto3-redshift-data==1.39.0
> > > > >> < mypy-boto3-s3==1.39.5
> > > > >> < mypy==1.17.0
> > > > >> 514c466
> > > > >> < mysql-connector-python==9.3.0
> > > > >> ---
> > > > >> > mysql-connector-python==9.4.0
> > > > >> 521,523d472
> > > > >> < networkx==3.5
> > > > >> < nh3==0.3.0
> > > > >> < nodeenv==1.9.1
> > > > >> 528,529d476
> > > > >> < openapi-schema-validator==0.6.3
> > > > >> < openapi-spec-validator==0.7.2
> > > > >> 560d506
> > > > >> < pathable==0.4.4
> > > > >> 563d508
> > > > >> < pdbr==0.9.2
> > > > >> 569,570c514
> > > > >> < pinotdb==5.6.0
> > > > >> < pipdeptree==2.28.0
> > > > >> ---
> > > > >> > pinotdb==5.7.0
> > > > >> 574d517
> > > > >> < plyvel==1.5.1
> > > > >> 577,579d519
> > > > >> < pprintpp==0.4.0
> > > > >> < pre-commit-uv==4.1.4
> > > > >> < pre_commit==4.2.0
> > > > >> 592d531
> > > > >> < py-partiql-parser==0.6.1
> > > > >> 608d546
> > > > >> < pyenchant==3.2.2
> > > > >> 610,611d547
> > > > >> < pygraphviz==1.14
> > > > >> < pyiceberg==0.9.1
> > > > >> 622,632d557
> > > > >> < pytest-asyncio==0.25.0
> > > > >> < pytest-cov==6.2.1
> > > > >> < pytest-custom-exit-code==0.3.0
> > > > >> < pytest-icdiff==0.9
> > > > >> < pytest-instafail==0.5.0
> > > > >> < pytest-mock==3.14.1
> > > > >> < pytest-rerunfailures==15.1
> > > > >> < pytest-timeouts==1.2.1
> > > > >> < pytest-unordered==0.7.0
> > > > >> < pytest-xdist==3.8.0
> > > > >> < pytest==8.4.1
> > > > >> 642d566
> > > > >> < python-on-whales==0.78.0
> > > > >> 652d575
> > > > >> < readme_renderer==44.0
> > > > >> 659d581
> > > > >> < requests-mock==1.12.1
> > > > >> 664,665d585
> > > > >> < responses==0.25.7
> > > > >> < restructuredtext_lint==1.4.0
> > > > >> 667,668d586
> > > > >> < rfc3339-validator==0.1.4
> > > > >> < rfc3986==2.0.0
> > > > >> 670d587
> > > > >> < rich-click==1.8.9
> > > > >> 674d590
> > > > >> < roman-numerals-py==3.1.0
> > > > >> 679d594
> > > > >> < ruff==0.12.3
> > > > >> 688d602
> > > > >> < semver==3.0.4
> > > > >> 690,691c604
> > > > >> < sentinels==1.0.0
> > > > >> < sentry-sdk==2.33.1
> > > > >> ---
> > > > >> > sentry-sdk==2.33.2
> > > > >> 703d615
> > > > >> < snowballstemmer==3.0.1
> > > > >> 709,725d620
> > > > >> < sphinx-argparse==0.5.2
> > > > >> < sphinx-autoapi==3.6.0
> > > > >> < sphinx-autobuild==2024.10.3
> > > > >> < sphinx-copybutton==0.5.2
> > > > >> < sphinx-jinja==2.0.2
> > > > >> < sphinx-rtd-theme==3.0.2
> > > > >> < sphinx_design==0.6.1
> > > > >> < sphinxcontrib-applehelp==2.0.0
> > > > >> < sphinxcontrib-devhelp==2.0.0
> > > > >> < sphinxcontrib-htmlhelp==2.1.0
> > > > >> < sphinxcontrib-httpdomain==1.8.1
> > > > >> < sphinxcontrib-jquery==4.1
> > > > >> < sphinxcontrib-jsmath==1.0.1
> > > > >> < sphinxcontrib-qthelp==2.0.0
> > > > >> < sphinxcontrib-redoc==1.6.0
> > > > >> < sphinxcontrib-serializinghtml==2.0.0
> > > > >> < sphinxcontrib-spelling==8.0.1
> > > > >> 737d631
> > > > >> < strictyaml==1.7.3
> > > > >> 740d633
> > > > >> < sympy==1.14.0
> > > > >> 752d644
> > > > >> < time-machine==2.16.0
> > > > >> 755d646
> > > > >> < tomli_w==1.2.0
> > > > >> 758d648
> > > > >> < towncrier==24.8.0
> > > > >> 762,763d651
> > > > >> < trove-classifiers==2025.5.9.12
> > > > >> < twine==6.1.0
> > > > >> 765,774d652
> > > > >> < types-Deprecated==1.2.15.20250304
> > > > >> < types-Markdown==3.8.0.20250708
> > > > >> < types-PyMySQL==1.1.0.20250711
> > > > >> < types-PyYAML==6.0.12.20250516
> > > > >> < types-aiofiles==24.1.0.20250708
> > > > >> < types-certifi==2021.10.8.3
> > > > >> < types-cffi==1.17.0.20250523
> > > > >> < types-croniter==6.0.0.20250626
> > > > >> < types-docutils==0.21.0.20250722
> > > > >> < types-paramiko==3.5.0.20250708
> > > > >> 776,778d653
> > > > >> < types-pyOpenSSL==24.1.0.20240722
> > > > >> < types-python-dateutil==2.9.0.20250708
> > > > >> < types-python-slugify==8.0.2.20240310
> > > > >> 780d654
> > > > >> < types-redis==4.6.0.20241004
> > > > >> 782,784d655
> > > > >> < types-setuptools==80.9.0.20250529
> > > > >> < types-tabulate==0.9.0.20241207
> > > > >> < types-toml==0.10.8.20240310
> > > > >> 794d664
> > > > >> < userpath==1.9.2
> > > > >> 799c669
> > > > >> < validators==0.34.0
> > > > >> ---
> > > > >> > validators==0.35.0
> > > > >> 806c676
> > > > >> < weaviate-client==4.9.6
> > > > >> ---
> > > > >> > weaviate-client==4.16.3
> > > > >> 809c679
> > > > >> < websockets==14.2
> > > > >> ---
> > > > >> > websockets==15.0.1
> > > > >> 815d684
> > > > >> < yamllint==1.37.1
> > > > >> ________________________________
> > > > >> Strike Technologies, LLC (“Strike”) is part of the GTS family of
> > > > >> companies. Strike is a technology solutions provider, and is not a
> > > > broker
> > > > >> or dealer and does not transact any securities related business
> > > directly
> > > > >> whatsoever. This communication is the property of Strike and its
> > > > >> affiliates, and does not constitute an offer to sell or the
> > > > solicitation of
> > > > >> an offer to buy any security in any jurisdiction. It is intended
> > only
> > > > for
> > > > >> the person to whom it is addressed and may contain information
> that
> > is
> > > > >> privileged, confidential, or otherwise protected from disclosure.
> > > > >> Distribution or copying of this communication, or the information
> > > > contained
> > > > >> herein, by anyone other than the intended recipient is prohibited.
> > If
> > > > you
> > > > >> have received this communication in error, please immediately
> notify
> > > > Strike
> > > > >> at info@striketechnologies.com, and delete and destroy any copies
> > > > hereof.
> > > > >> ________________________________
> > > > >>
> > > > >> CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any
> > > > attachments
> > > > >> are intended solely for the addressee. This transmission is
> covered
> > by
> > > > the
> > > > >> Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The
> > > > >> information contained in this transmission is confidential in
> nature
> > > and
> > > > >> protected from further use or disclosure under U.S. Pub. L.
> 106-102,
> > > 113
> > > > >> U.S. Stat. 1338 (1999), and may be subject to attorney-client or
> > other
> > > > >> legal privilege. Your use or disclosure of this information for
> any
> > > > purpose
> > > > >> other than that intended by its transmittal is strictly
> prohibited,
> > > and
> > > > may
> > > > >> subject you to fines and/or penalties under federal and state law.
> > If
> > > > you
> > > > >> are not the intended recipient of this transmission, please
> DESTROY
> > > ALL
> > > > >> COPIES RECEIVED and confirm destruction to the sender via return
> > > > >> transmittal.
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Removing dev dependencies from PyPI constraints ?
Posted by Elad Kalif <el...@apache.org>.
Will the change take effect for all future releases or just from 3.11
moving forward?
On Fri, Aug 1, 2025 at 8:57 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> OK. I will run LAZY_CONSENSUS then
>
> On Fri, Jul 25, 2025 at 9:22 AM Pavankumar Gopidesu <
> gopidesupavan@gmail.com>
> wrote:
>
> > Thanks Jarek,
> >
> > I am fine with option b as far as if we find anything while testing we
> can
> > downgrade.
> >
> > Pavan
> >
> > On Thu, Jul 24, 2025 at 10:44 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > Would love to hear more if anyone has an opinion :). Yep. It's an
> > advanced
> > > topic and I can make arbitrary decision :) ...
> > >
> > > I am thinking now that it should be quite "OK" to accept those
> > "potentially
> > > untested" versions - especially that they will be tested during manual
> > > testing. Unless someone objects i will likely update PR with some
> > changelog
> > > / newsfragment to announce the change in 3.1. I will run LAZY CONSENSUS
> > > soon if I don't hear any opposition.
> > >
> > > J.
> > >
> > >
> > > On Tue, Jul 22, 2025 at 5:52 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > > > Exactly to remove the dev requirements airflow has from interfering
> > > with
> > > > our own dev requirements. Though this approach notably doesn't change
> > any
> > > > of the used non-dev requirements, it's not clear to me (and I'm
> > probably
> > > > just missing something) why your diff includes non-dev requirements.
> > > >
> > > > The thing is that I do not "remove" devel features - because due to
> > > > transitive dependencies I would have to go deep down the rabbit hole
> > > which
> > > > dependencies are "devel" and which not. So what I do, I ask `uv` to
> do
> > > the
> > > > job for me and repeat the resolution but with the `--exact` flag
> > > (combined
> > > > with the already used `--resolution higest` and `--no-sources`).
> > > >
> > > > Apparently in this case `uv` will sometimes decide that it's ok to
> bump
> > > > some dependencies that are left after initial resolution. The thing
> is
> > > that
> > > > uv will not install some versions of some packages if they are
> > > conflicting
> > > > with already installed packages. What I believe `uv` does when I use
> > the
> > > *--exact
> > > > *flag of `uv pip install` it "fixes" the initial resolution. It finds
> > out
> > > > what is needed, removes those packages that are not needed, and then
> it
> > > > resolves again and improves the resolution (that would be my guess at
> > > least
> > > > - it could be done slightly differently and in a single step I guess.
> > > >
> > > > Essentially the difference between the left and right side is:
> > > >
> > > > a) install absolutely everything from the workspace
> > > >
> > > > b) do this:
> > > > left: uv pip install --no-sources 'apache-airflow[all]'
> > > > 'apache-airflow-core[all]' apache-airflow-task-sdk ./airflow-ctl
> > > > --reinstall --resolution highest --find-links file:///dist
> > > > right: uv pip install --no-sources *--exact --strict*
> > > > 'apache-airflow[all]' 'apache-airflow-core[all]'
> > apache-airflow-task-sdk
> > > > ./airflow-ctl --reinstall --resolution highest --find-links
> > file:///dist
> > > >
> > > > In the current state content of the fille://dist should be the same
> in
> > > > both cases:
> > > >
> > > > apache-airflow-providers-edge3 (1.1.2): wheel -
> > > > apache_airflow_providers_edge3-1.1.2-py3-none-any.whl
> > > > apache-airflow-providers-keycloak (1.0.0): wheel -
> > > > apache_airflow_providers_keycloak-1.0.0-py3-none-any.whl
> > > > apache-airflow-providers-snowflake (6.5.1): wheel -
> > > > apache_airflow_providers_snowflake-6.5.1-py3-none-any.whl
> > > >
> > > > This is a bit of in-process quirk of the "PyPI" resolution - those
> are
> > > the
> > > > three packages that we have updated versions of in the sources, but
> > they
> > > > are not released yet. But they should be essentially the same in both
> > > > cases. So the difference between left and right is `--exact --strict`
> > > > (strict I added to make sure that we are not removing something that
> is
> > > > still needed).
> > > >
> > > > J.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, Jul 22, 2025 at 5:22 PM Damian Shaw <
> > > dshaw@striketechnologies.com>
> > > > wrote:
> > > >
> > > >> As I user of Airflow I have ended up implementing this for my
> > > >> environments:
> > > >>
> > > >> pip install apache-airflow[all or select-extras] -c {official
> > > >> constraints} > smaller-run-time-constraints.txt
> > > >> pip-compile {all my requirements including development}.txt -c
> > > >> smaller-run-time-constraints.txt
> > > >>
> > > >> Exactly to remove the dev requirements airflow has from interfering
> > with
> > > >> our own dev requirements. Though this approach notably doesn't
> change
> > > any
> > > >> of the used non-dev requirements, it's not clear to me (and I'm
> > probably
> > > >> just missing something) why your diff includes non-dev requirements.
> > > >>
> > > >> Damian
> > > >>
> > > >> -----Original Message-----
> > > >> From: Jarek Potiuk <ja...@potiuk.com>
> > > >> Sent: Tuesday, July 22, 2025 11:04 AM
> > > >> To: dev@airflow.apache.org
> > > >> Subject: [DISCUSS] Removing dev dependencies from PyPI constraints ?
> > > >>
> > > >> Hello here,
> > > >>
> > > >>
> > > >> *Tl;DR; We had an interesting discussion on Slack today and I wanted
> > to
> > > >> bring it here to discuss what we should do - whether we should keep
> > > devel
> > > >> dependencies in our PyPI constraints or not.*
> > > >>
> > > >> *A bit of context:*
> > > >> In our CI we generate several types of constraints. one of them are
> > > >> "source" constraints that we use during our CI testing - including
> all
> > > our
> > > >> testing tools, mypy, ruff etc., and another type is "PyPI"
> constraint
> > -
> > > >> which is more <What should be the "golden" versions of dependencies
> > when
> > > >> you install Airflow using `pip install` for production>.
> > > >>
> > > >> The main difference between the two (roughly) is that "source
> > > constraints"
> > > >> only uses dependencies from "branch tip" ("main", "v3-0-test"),
> where
> > > >> "PyPI constraints" uses "Airflow core" branch-tip dependencies - but
> > all
> > > >> the providers are installed from PyPI. There are few nuances when
> some
> > > >> packages are being released but basically - for example for Airflow
> > > 3.0.3,
> > > >> "PyPI"
> > > >> constraints is "what is the current set of dependencies and
> providers
> > > >> when we install airflow from PyPI at the moment of release".
> > > >>
> > > >> So target for "source" constraints are "Airflow contributors" where
> > > >> target for "PyPI constraints" are "Airflow users".
> > > >>
> > > >>
> > > >> *The problem*
> > > >> In the slack discussion
> > > >>
> > https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1753169914722049
> > > >> - slack user @dashmug (sorry - do not know name) noticed that the
> > > "PyPI"
> > > >> constraints also contain Airflow development dependencies listed
> (like
> > > >> ruff or mypy). That made me think a bit and I quickly came up with
> PR
> > > that
> > > >> could "fix" it - i.e. only keep the "real" constraints in.
> > > >> https://github.com/apache/airflow/pull/53631 - it's as easy as
> adding
> > > >> the `--exact` (and `--strict` for verification) flag to the `uv pip
> > > >> install` command we use.
> > > >>
> > > >> I agree it's a bit confusing to have the devel dependencies there -
> it
> > > is
> > > >> not the "intention", but there is very little problem with it - the
> > fact
> > > >> that mypy or ruff are listed with a version in constraints, does not
> > > mean
> > > >> it must be installed - constraints mainly say "if you install this,
> it
> > > >> should be this version" . And there is no problem whatsoever (and it
> > is
> > > >> even recommended by us) to have "airflow with its deps" installed
> with
> > > >> constraints, and then anything else you need to install or update -
> > > >> without. So other than potential confusion, it usually does not
> cause
> > > harm.
> > > >>
> > > >> Now... It might seem that removing devel deps has no side effects
> and
> > we
> > > >> should **just do it** - but it's not that easy. Main problem is that
> > > those
> > > >> devel dependencies might "hold" other dependencies (directly or
> > > >> transitively) from being upgraded - and when we remove the devel
> > > >> dependencies and do the resolution again - we might have a DIFFERENT
> > > set of
> > > >> dependencies, simply because removing the devel dependencies, might
> > > "free"
> > > >> the other deps and they might get upgraded. Yeah - I know it's not
> > > >> obvious for someone who did not spend 3 years solving dependency
> > issues
> > > :).
> > > >>
> > > >> And the problem with it is that because we need those devel deps to
> > test
> > > >> things - the upgraded dependencies might have never been tested in
> our
> > > CI -
> > > >> because we never upgraded to later versions. It's a bit of
> > > >> Heisenberg-effect. By using test tools we are changing the state of
> > the
> > > >> tested thing. Currently our "PyPI" constraints reflect the state of
> > > Airflow
> > > >> "when the testing tools are installed" - and we are generally not
> able
> > > to
> > > >> test the state of Airflow "without the testing tools" - by the sheer
> > > fact
> > > >> that we do not have those tools :)
> > > >>
> > > >> Just to illustrate the case - you can see what happens when we
> removed
> > > >> devel deps:
> > > >> https://github.com/apache/airflow/actions/runs/16444553234 - in
> > summary
> > > >> you can see a colored version that is a bit clearer but I dumped the
> > > diff
> > > >> below from an example - Python 3.10 case. Left side < is "what was
> > with
> > > >> testing tools" and > is without them. You can see a LOT of test
> tools
> > > >> removed (pytest, ruff and others - including dependencies used only
> by
> > > >> those tools)
> > > >>
> > > >> But also you can see (likely some of the devel/test deps limited
> > those)
> > > >>
> > > >> * Authlib - upgraded from 1.3.1 to 1.6.1
> > > >> * Google - genai upgraded from 1.2.0 to 1.20.0
> > > >> * Httpx - upgraded from 0.27.0 to 0.28.1
> > > >> * Pinotdb - upgraded from 5.6.0 to 5.7.0
> > > >> * Sentry-sdk - upgraded from 2.33.1 to 2.33.2 (this is false
> positive
> > > as
> > > >> it has just been released and new version was picked)
> > > >> * validators - upgraded form 0.34.0 to 0.35.0
> > > >> * weaviate-client - upgraded from 4.9.0 to 4.16.3
> > > >> * websockets - upgraded from 14.2 to 15.0.1
> > > >>
> > > >>
> > > >> This means that all the "upgraded" packages above have not been -
> > likely
> > > >> ever tested in our CI.
> > > >>
> > > >> And when we put them in "versioned" constraints, it might mean that
> > they
> > > >> will be obviously failing.
> > > >>
> > > >> So ... there is a risk connected with just generating the
> "non-devel"
> > > >> constraints without additional tests. But that also means that the
> > user
> > > >> who does not use constraints might get those versions anyway -
> default
> > > >> behaviour of `pip install airflow` will be to install those upgraded
> > > >> versions above if you do not use constraints. But also that is no
> > > different
> > > >> than installing a released airflow version a few weeks or months
> after
> > > >> release without constraints - there will be tens of packages that
> were
> > > >> released since our release and installing airflow without
> constraints
> > > will
> > > >> pick those.
> > > >>
> > > >> However - when we manually install RC candidates - say by running
> > > airflow
> > > >> from rc1 image, we will use those "upgraded" dependencies - so some
> > > >> "obvious" problems might be caught during the rc manual testing -
> > where
> > > you
> > > >> do not use test tools but run airflow "as if" it was installed in
> > > >> production.
> > > >>
> > > >> *The solution ? *
> > > >>
> > > >> There are few ways we can proceed with it:
> > > >>
> > > >> a) leave it as is - no big harm, potential confusion, by people who
> > see
> > > >> it
> > > >> - potential differences
> > > >>
> > > >> b) remove devel following my PR and take the risk that we never
> tested
> > > (in
> > > >> CI) the dependencies we set in the constraints. That's a bit risky,
> > but
> > > >> we can always reactively downgrade some of those packages after the
> > > fact if
> > > >> users will start reporting the issues
> > > >>
> > > >> c) potentially most complex thing - trying to remove devel
> > dependencies
> > > >> somewhat combining the two - for example removing all the deps that
> > > would
> > > >> be removed in b) from a) - but that has also some edge cases - when
> > > >> different versions of dependencies can add or upgrade their
> > > dependencies,
> > > >> we are very likely to occasionally remove or add too much.
> > > >>
> > > >>
> > > >> I wonder (for those who managed to read that far - likely not many -
> > > what
> > > >> do you think here? What would be the best option ? Maybe there are
> > other
> > > >> options I have not thought about ?
> > > >>
> > > >>
> > > >> J
> > > >>
> > > >>
> > > >> -----
> > > >>
> > > >> The diff for Python 3.10
> > > >>
> > > >>
> > > >>
> > > >> 34c34
> > > >> < Authlib==1.3.1
> > > >> ---
> > > >> > Authlib==1.6.1
> > > >> 64d63
> > > >> < Sphinx==8.2.3
> > > >> 78d76
> > > >> < aioresponses==0.7.8
> > > >> 82d79
> > > >> < alabaster==1.0.0
> > > >> 198d194
> > > >> < arro3-core==0.5.1
> > > >> 202d197
> > > >> < astroid==3.3.11
> > > >> 211,212d205
> > > >> < aws-sam-translator==1.99.0
> > > >> < aws-xray-sdk==2.14.0
> > > >> 258,259d250
> > > >> < cfgv==3.4.0
> > > >> < cfn-lint==1.38.0
> > > >> 262d252
> > > >> < checksumdir==1.2.0
> > > >> 277d266
> > > >> < coverage==7.9.2
> > > >> 291d279
> > > >> < deltalake==1.1.0
> > > >> 293d280
> > > >> < diagrams==0.24.4
> > > >> 301,302d287
> > > >> < docutils==0.21.2
> > > >> < duckdb==1.3.2
> > > >> 308d292
> > > >> < eralchemy2==1.4.1
> > > >> 311d294
> > > >> < execnet==2.1.1
> > > >> 321,322d303
> > > >> < flit==3.12.0
> > > >> < flit_core==3.12.0
> > > >> 386c367
> > > >> < google-genai==1.2.0
> > > >> ---
> > > >> > google-genai==1.20.0
> > > >> 389d369
> > > >> < graphql-core==3.2.6
> > > >> 398d377
> > > >> < grpcio-tools==1.62.3
> > > >> 403,404d381
> > > >> < hatch==1.14.1
> > > >> < hatchling==1.27.0
> > > >> 413c390
> > > >> < httpx==0.27.0
> > > >> ---
> > > >> > httpx==0.28.1
> > > >> 418d394
> > > >> < hyperlink==21.0.0
> > > >> 421,423d396
> > > >> < icdiff==2.0.7
> > > >> < id==1.5.0
> > > >> < identify==2.6.12
> > > >> 426d398
> > > >> < imagesize==1.4.1
> > > >> 430d401
> > > >> < incremental==24.7.2
> > > >> 433,435d403
> > > >> < iniconfig==2.1.0
> > > >> < inputimeout==1.0.4
> > > >> < ipdb==0.13.13
> > > >> 449d416
> > > >> < joserfc==1.2.2
> > > >> 451d417
> > > >> < jsonpatch==1.33
> > > >> 455,456d420
> > > >> < jsonpointer==3.0.0
> > > >> < jsonschema-path==0.3.4
> > > >> 463d426
> > > >> < kerberos==1.3.1
> > > >> 465,466d427
> > > >> < keyrings.alt==5.0.2
> > > >> < kgb==7.2
> > > >> 494,495d454
> > > >> < mmh3==5.1.0
> > > >> < mongomock==4.3.0
> > > >> 497,498d455
> > > >> < moto==5.1.8
> > > >> < mpmath==1.3.0
> > > >> 508,512d464
> > > >> < mypy-boto3-appflow==1.39.0
> > > >> < mypy-boto3-rds==1.39.1
> > > >> < mypy-boto3-redshift-data==1.39.0
> > > >> < mypy-boto3-s3==1.39.5
> > > >> < mypy==1.17.0
> > > >> 514c466
> > > >> < mysql-connector-python==9.3.0
> > > >> ---
> > > >> > mysql-connector-python==9.4.0
> > > >> 521,523d472
> > > >> < networkx==3.5
> > > >> < nh3==0.3.0
> > > >> < nodeenv==1.9.1
> > > >> 528,529d476
> > > >> < openapi-schema-validator==0.6.3
> > > >> < openapi-spec-validator==0.7.2
> > > >> 560d506
> > > >> < pathable==0.4.4
> > > >> 563d508
> > > >> < pdbr==0.9.2
> > > >> 569,570c514
> > > >> < pinotdb==5.6.0
> > > >> < pipdeptree==2.28.0
> > > >> ---
> > > >> > pinotdb==5.7.0
> > > >> 574d517
> > > >> < plyvel==1.5.1
> > > >> 577,579d519
> > > >> < pprintpp==0.4.0
> > > >> < pre-commit-uv==4.1.4
> > > >> < pre_commit==4.2.0
> > > >> 592d531
> > > >> < py-partiql-parser==0.6.1
> > > >> 608d546
> > > >> < pyenchant==3.2.2
> > > >> 610,611d547
> > > >> < pygraphviz==1.14
> > > >> < pyiceberg==0.9.1
> > > >> 622,632d557
> > > >> < pytest-asyncio==0.25.0
> > > >> < pytest-cov==6.2.1
> > > >> < pytest-custom-exit-code==0.3.0
> > > >> < pytest-icdiff==0.9
> > > >> < pytest-instafail==0.5.0
> > > >> < pytest-mock==3.14.1
> > > >> < pytest-rerunfailures==15.1
> > > >> < pytest-timeouts==1.2.1
> > > >> < pytest-unordered==0.7.0
> > > >> < pytest-xdist==3.8.0
> > > >> < pytest==8.4.1
> > > >> 642d566
> > > >> < python-on-whales==0.78.0
> > > >> 652d575
> > > >> < readme_renderer==44.0
> > > >> 659d581
> > > >> < requests-mock==1.12.1
> > > >> 664,665d585
> > > >> < responses==0.25.7
> > > >> < restructuredtext_lint==1.4.0
> > > >> 667,668d586
> > > >> < rfc3339-validator==0.1.4
> > > >> < rfc3986==2.0.0
> > > >> 670d587
> > > >> < rich-click==1.8.9
> > > >> 674d590
> > > >> < roman-numerals-py==3.1.0
> > > >> 679d594
> > > >> < ruff==0.12.3
> > > >> 688d602
> > > >> < semver==3.0.4
> > > >> 690,691c604
> > > >> < sentinels==1.0.0
> > > >> < sentry-sdk==2.33.1
> > > >> ---
> > > >> > sentry-sdk==2.33.2
> > > >> 703d615
> > > >> < snowballstemmer==3.0.1
> > > >> 709,725d620
> > > >> < sphinx-argparse==0.5.2
> > > >> < sphinx-autoapi==3.6.0
> > > >> < sphinx-autobuild==2024.10.3
> > > >> < sphinx-copybutton==0.5.2
> > > >> < sphinx-jinja==2.0.2
> > > >> < sphinx-rtd-theme==3.0.2
> > > >> < sphinx_design==0.6.1
> > > >> < sphinxcontrib-applehelp==2.0.0
> > > >> < sphinxcontrib-devhelp==2.0.0
> > > >> < sphinxcontrib-htmlhelp==2.1.0
> > > >> < sphinxcontrib-httpdomain==1.8.1
> > > >> < sphinxcontrib-jquery==4.1
> > > >> < sphinxcontrib-jsmath==1.0.1
> > > >> < sphinxcontrib-qthelp==2.0.0
> > > >> < sphinxcontrib-redoc==1.6.0
> > > >> < sphinxcontrib-serializinghtml==2.0.0
> > > >> < sphinxcontrib-spelling==8.0.1
> > > >> 737d631
> > > >> < strictyaml==1.7.3
> > > >> 740d633
> > > >> < sympy==1.14.0
> > > >> 752d644
> > > >> < time-machine==2.16.0
> > > >> 755d646
> > > >> < tomli_w==1.2.0
> > > >> 758d648
> > > >> < towncrier==24.8.0
> > > >> 762,763d651
> > > >> < trove-classifiers==2025.5.9.12
> > > >> < twine==6.1.0
> > > >> 765,774d652
> > > >> < types-Deprecated==1.2.15.20250304
> > > >> < types-Markdown==3.8.0.20250708
> > > >> < types-PyMySQL==1.1.0.20250711
> > > >> < types-PyYAML==6.0.12.20250516
> > > >> < types-aiofiles==24.1.0.20250708
> > > >> < types-certifi==2021.10.8.3
> > > >> < types-cffi==1.17.0.20250523
> > > >> < types-croniter==6.0.0.20250626
> > > >> < types-docutils==0.21.0.20250722
> > > >> < types-paramiko==3.5.0.20250708
> > > >> 776,778d653
> > > >> < types-pyOpenSSL==24.1.0.20240722
> > > >> < types-python-dateutil==2.9.0.20250708
> > > >> < types-python-slugify==8.0.2.20240310
> > > >> 780d654
> > > >> < types-redis==4.6.0.20241004
> > > >> 782,784d655
> > > >> < types-setuptools==80.9.0.20250529
> > > >> < types-tabulate==0.9.0.20241207
> > > >> < types-toml==0.10.8.20240310
> > > >> 794d664
> > > >> < userpath==1.9.2
> > > >> 799c669
> > > >> < validators==0.34.0
> > > >> ---
> > > >> > validators==0.35.0
> > > >> 806c676
> > > >> < weaviate-client==4.9.6
> > > >> ---
> > > >> > weaviate-client==4.16.3
> > > >> 809c679
> > > >> < websockets==14.2
> > > >> ---
> > > >> > websockets==15.0.1
> > > >> 815d684
> > > >> < yamllint==1.37.1
> > > >> ________________________________
> > > >> Strike Technologies, LLC (“Strike”) is part of the GTS family of
> > > >> companies. Strike is a technology solutions provider, and is not a
> > > broker
> > > >> or dealer and does not transact any securities related business
> > directly
> > > >> whatsoever. This communication is the property of Strike and its
> > > >> affiliates, and does not constitute an offer to sell or the
> > > solicitation of
> > > >> an offer to buy any security in any jurisdiction. It is intended
> only
> > > for
> > > >> the person to whom it is addressed and may contain information that
> is
> > > >> privileged, confidential, or otherwise protected from disclosure.
> > > >> Distribution or copying of this communication, or the information
> > > contained
> > > >> herein, by anyone other than the intended recipient is prohibited.
> If
> > > you
> > > >> have received this communication in error, please immediately notify
> > > Strike
> > > >> at info@striketechnologies.com, and delete and destroy any copies
> > > hereof.
> > > >> ________________________________
> > > >>
> > > >> CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any
> > > attachments
> > > >> are intended solely for the addressee. This transmission is covered
> by
> > > the
> > > >> Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The
> > > >> information contained in this transmission is confidential in nature
> > and
> > > >> protected from further use or disclosure under U.S. Pub. L. 106-102,
> > 113
> > > >> U.S. Stat. 1338 (1999), and may be subject to attorney-client or
> other
> > > >> legal privilege. Your use or disclosure of this information for any
> > > purpose
> > > >> other than that intended by its transmittal is strictly prohibited,
> > and
> > > may
> > > >> subject you to fines and/or penalties under federal and state law.
> If
> > > you
> > > >> are not the intended recipient of this transmission, please DESTROY
> > ALL
> > > >> COPIES RECEIVED and confirm destruction to the sender via return
> > > >> transmittal.
> > > >>
> > > >
> > >
> >
>
Re: [DISCUSS] Removing dev dependencies from PyPI constraints ?
Posted by Jarek Potiuk <ja...@potiuk.com>.
OK. I will run LAZY_CONSENSUS then
On Fri, Jul 25, 2025 at 9:22 AM Pavankumar Gopidesu <go...@gmail.com>
wrote:
> Thanks Jarek,
>
> I am fine with option b as far as if we find anything while testing we can
> downgrade.
>
> Pavan
>
> On Thu, Jul 24, 2025 at 10:44 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Would love to hear more if anyone has an opinion :). Yep. It's an
> advanced
> > topic and I can make arbitrary decision :) ...
> >
> > I am thinking now that it should be quite "OK" to accept those
> "potentially
> > untested" versions - especially that they will be tested during manual
> > testing. Unless someone objects i will likely update PR with some
> changelog
> > / newsfragment to announce the change in 3.1. I will run LAZY CONSENSUS
> > soon if I don't hear any opposition.
> >
> > J.
> >
> >
> > On Tue, Jul 22, 2025 at 5:52 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > > Exactly to remove the dev requirements airflow has from interfering
> > with
> > > our own dev requirements. Though this approach notably doesn't change
> any
> > > of the used non-dev requirements, it's not clear to me (and I'm
> probably
> > > just missing something) why your diff includes non-dev requirements.
> > >
> > > The thing is that I do not "remove" devel features - because due to
> > > transitive dependencies I would have to go deep down the rabbit hole
> > which
> > > dependencies are "devel" and which not. So what I do, I ask `uv` to do
> > the
> > > job for me and repeat the resolution but with the `--exact` flag
> > (combined
> > > with the already used `--resolution higest` and `--no-sources`).
> > >
> > > Apparently in this case `uv` will sometimes decide that it's ok to bump
> > > some dependencies that are left after initial resolution. The thing is
> > that
> > > uv will not install some versions of some packages if they are
> > conflicting
> > > with already installed packages. What I believe `uv` does when I use
> the
> > *--exact
> > > *flag of `uv pip install` it "fixes" the initial resolution. It finds
> out
> > > what is needed, removes those packages that are not needed, and then it
> > > resolves again and improves the resolution (that would be my guess at
> > least
> > > - it could be done slightly differently and in a single step I guess.
> > >
> > > Essentially the difference between the left and right side is:
> > >
> > > a) install absolutely everything from the workspace
> > >
> > > b) do this:
> > > left: uv pip install --no-sources 'apache-airflow[all]'
> > > 'apache-airflow-core[all]' apache-airflow-task-sdk ./airflow-ctl
> > > --reinstall --resolution highest --find-links file:///dist
> > > right: uv pip install --no-sources *--exact --strict*
> > > 'apache-airflow[all]' 'apache-airflow-core[all]'
> apache-airflow-task-sdk
> > > ./airflow-ctl --reinstall --resolution highest --find-links
> file:///dist
> > >
> > > In the current state content of the fille://dist should be the same in
> > > both cases:
> > >
> > > apache-airflow-providers-edge3 (1.1.2): wheel -
> > > apache_airflow_providers_edge3-1.1.2-py3-none-any.whl
> > > apache-airflow-providers-keycloak (1.0.0): wheel -
> > > apache_airflow_providers_keycloak-1.0.0-py3-none-any.whl
> > > apache-airflow-providers-snowflake (6.5.1): wheel -
> > > apache_airflow_providers_snowflake-6.5.1-py3-none-any.whl
> > >
> > > This is a bit of in-process quirk of the "PyPI" resolution - those are
> > the
> > > three packages that we have updated versions of in the sources, but
> they
> > > are not released yet. But they should be essentially the same in both
> > > cases. So the difference between left and right is `--exact --strict`
> > > (strict I added to make sure that we are not removing something that is
> > > still needed).
> > >
> > > J.
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Jul 22, 2025 at 5:22 PM Damian Shaw <
> > dshaw@striketechnologies.com>
> > > wrote:
> > >
> > >> As I user of Airflow I have ended up implementing this for my
> > >> environments:
> > >>
> > >> pip install apache-airflow[all or select-extras] -c {official
> > >> constraints} > smaller-run-time-constraints.txt
> > >> pip-compile {all my requirements including development}.txt -c
> > >> smaller-run-time-constraints.txt
> > >>
> > >> Exactly to remove the dev requirements airflow has from interfering
> with
> > >> our own dev requirements. Though this approach notably doesn't change
> > any
> > >> of the used non-dev requirements, it's not clear to me (and I'm
> probably
> > >> just missing something) why your diff includes non-dev requirements.
> > >>
> > >> Damian
> > >>
> > >> -----Original Message-----
> > >> From: Jarek Potiuk <ja...@potiuk.com>
> > >> Sent: Tuesday, July 22, 2025 11:04 AM
> > >> To: dev@airflow.apache.org
> > >> Subject: [DISCUSS] Removing dev dependencies from PyPI constraints ?
> > >>
> > >> Hello here,
> > >>
> > >>
> > >> *Tl;DR; We had an interesting discussion on Slack today and I wanted
> to
> > >> bring it here to discuss what we should do - whether we should keep
> > devel
> > >> dependencies in our PyPI constraints or not.*
> > >>
> > >> *A bit of context:*
> > >> In our CI we generate several types of constraints. one of them are
> > >> "source" constraints that we use during our CI testing - including all
> > our
> > >> testing tools, mypy, ruff etc., and another type is "PyPI" constraint
> -
> > >> which is more <What should be the "golden" versions of dependencies
> when
> > >> you install Airflow using `pip install` for production>.
> > >>
> > >> The main difference between the two (roughly) is that "source
> > constraints"
> > >> only uses dependencies from "branch tip" ("main", "v3-0-test"), where
> > >> "PyPI constraints" uses "Airflow core" branch-tip dependencies - but
> all
> > >> the providers are installed from PyPI. There are few nuances when some
> > >> packages are being released but basically - for example for Airflow
> > 3.0.3,
> > >> "PyPI"
> > >> constraints is "what is the current set of dependencies and providers
> > >> when we install airflow from PyPI at the moment of release".
> > >>
> > >> So target for "source" constraints are "Airflow contributors" where
> > >> target for "PyPI constraints" are "Airflow users".
> > >>
> > >>
> > >> *The problem*
> > >> In the slack discussion
> > >>
> https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1753169914722049
> > >> - slack user @dashmug (sorry - do not know name) noticed that the
> > "PyPI"
> > >> constraints also contain Airflow development dependencies listed (like
> > >> ruff or mypy). That made me think a bit and I quickly came up with PR
> > that
> > >> could "fix" it - i.e. only keep the "real" constraints in.
> > >> https://github.com/apache/airflow/pull/53631 - it's as easy as adding
> > >> the `--exact` (and `--strict` for verification) flag to the `uv pip
> > >> install` command we use.
> > >>
> > >> I agree it's a bit confusing to have the devel dependencies there - it
> > is
> > >> not the "intention", but there is very little problem with it - the
> fact
> > >> that mypy or ruff are listed with a version in constraints, does not
> > mean
> > >> it must be installed - constraints mainly say "if you install this, it
> > >> should be this version" . And there is no problem whatsoever (and it
> is
> > >> even recommended by us) to have "airflow with its deps" installed with
> > >> constraints, and then anything else you need to install or update -
> > >> without. So other than potential confusion, it usually does not cause
> > harm.
> > >>
> > >> Now... It might seem that removing devel deps has no side effects and
> we
> > >> should **just do it** - but it's not that easy. Main problem is that
> > those
> > >> devel dependencies might "hold" other dependencies (directly or
> > >> transitively) from being upgraded - and when we remove the devel
> > >> dependencies and do the resolution again - we might have a DIFFERENT
> > set of
> > >> dependencies, simply because removing the devel dependencies, might
> > "free"
> > >> the other deps and they might get upgraded. Yeah - I know it's not
> > >> obvious for someone who did not spend 3 years solving dependency
> issues
> > :).
> > >>
> > >> And the problem with it is that because we need those devel deps to
> test
> > >> things - the upgraded dependencies might have never been tested in our
> > CI -
> > >> because we never upgraded to later versions. It's a bit of
> > >> Heisenberg-effect. By using test tools we are changing the state of
> the
> > >> tested thing. Currently our "PyPI" constraints reflect the state of
> > Airflow
> > >> "when the testing tools are installed" - and we are generally not able
> > to
> > >> test the state of Airflow "without the testing tools" - by the sheer
> > fact
> > >> that we do not have those tools :)
> > >>
> > >> Just to illustrate the case - you can see what happens when we removed
> > >> devel deps:
> > >> https://github.com/apache/airflow/actions/runs/16444553234 - in
> summary
> > >> you can see a colored version that is a bit clearer but I dumped the
> > diff
> > >> below from an example - Python 3.10 case. Left side < is "what was
> with
> > >> testing tools" and > is without them. You can see a LOT of test tools
> > >> removed (pytest, ruff and others - including dependencies used only by
> > >> those tools)
> > >>
> > >> But also you can see (likely some of the devel/test deps limited
> those)
> > >>
> > >> * Authlib - upgraded from 1.3.1 to 1.6.1
> > >> * Google - genai upgraded from 1.2.0 to 1.20.0
> > >> * Httpx - upgraded from 0.27.0 to 0.28.1
> > >> * Pinotdb - upgraded from 5.6.0 to 5.7.0
> > >> * Sentry-sdk - upgraded from 2.33.1 to 2.33.2 (this is false positive
> > as
> > >> it has just been released and new version was picked)
> > >> * validators - upgraded form 0.34.0 to 0.35.0
> > >> * weaviate-client - upgraded from 4.9.0 to 4.16.3
> > >> * websockets - upgraded from 14.2 to 15.0.1
> > >>
> > >>
> > >> This means that all the "upgraded" packages above have not been -
> likely
> > >> ever tested in our CI.
> > >>
> > >> And when we put them in "versioned" constraints, it might mean that
> they
> > >> will be obviously failing.
> > >>
> > >> So ... there is a risk connected with just generating the "non-devel"
> > >> constraints without additional tests. But that also means that the
> user
> > >> who does not use constraints might get those versions anyway - default
> > >> behaviour of `pip install airflow` will be to install those upgraded
> > >> versions above if you do not use constraints. But also that is no
> > different
> > >> than installing a released airflow version a few weeks or months after
> > >> release without constraints - there will be tens of packages that were
> > >> released since our release and installing airflow without constraints
> > will
> > >> pick those.
> > >>
> > >> However - when we manually install RC candidates - say by running
> > airflow
> > >> from rc1 image, we will use those "upgraded" dependencies - so some
> > >> "obvious" problems might be caught during the rc manual testing -
> where
> > you
> > >> do not use test tools but run airflow "as if" it was installed in
> > >> production.
> > >>
> > >> *The solution ? *
> > >>
> > >> There are few ways we can proceed with it:
> > >>
> > >> a) leave it as is - no big harm, potential confusion, by people who
> see
> > >> it
> > >> - potential differences
> > >>
> > >> b) remove devel following my PR and take the risk that we never tested
> > (in
> > >> CI) the dependencies we set in the constraints. That's a bit risky,
> but
> > >> we can always reactively downgrade some of those packages after the
> > fact if
> > >> users will start reporting the issues
> > >>
> > >> c) potentially most complex thing - trying to remove devel
> dependencies
> > >> somewhat combining the two - for example removing all the deps that
> > would
> > >> be removed in b) from a) - but that has also some edge cases - when
> > >> different versions of dependencies can add or upgrade their
> > dependencies,
> > >> we are very likely to occasionally remove or add too much.
> > >>
> > >>
> > >> I wonder (for those who managed to read that far - likely not many -
> > what
> > >> do you think here? What would be the best option ? Maybe there are
> other
> > >> options I have not thought about ?
> > >>
> > >>
> > >> J
> > >>
> > >>
> > >> -----
> > >>
> > >> The diff for Python 3.10
> > >>
> > >>
> > >>
> > >> 34c34
> > >> < Authlib==1.3.1
> > >> ---
> > >> > Authlib==1.6.1
> > >> 64d63
> > >> < Sphinx==8.2.3
> > >> 78d76
> > >> < aioresponses==0.7.8
> > >> 82d79
> > >> < alabaster==1.0.0
> > >> 198d194
> > >> < arro3-core==0.5.1
> > >> 202d197
> > >> < astroid==3.3.11
> > >> 211,212d205
> > >> < aws-sam-translator==1.99.0
> > >> < aws-xray-sdk==2.14.0
> > >> 258,259d250
> > >> < cfgv==3.4.0
> > >> < cfn-lint==1.38.0
> > >> 262d252
> > >> < checksumdir==1.2.0
> > >> 277d266
> > >> < coverage==7.9.2
> > >> 291d279
> > >> < deltalake==1.1.0
> > >> 293d280
> > >> < diagrams==0.24.4
> > >> 301,302d287
> > >> < docutils==0.21.2
> > >> < duckdb==1.3.2
> > >> 308d292
> > >> < eralchemy2==1.4.1
> > >> 311d294
> > >> < execnet==2.1.1
> > >> 321,322d303
> > >> < flit==3.12.0
> > >> < flit_core==3.12.0
> > >> 386c367
> > >> < google-genai==1.2.0
> > >> ---
> > >> > google-genai==1.20.0
> > >> 389d369
> > >> < graphql-core==3.2.6
> > >> 398d377
> > >> < grpcio-tools==1.62.3
> > >> 403,404d381
> > >> < hatch==1.14.1
> > >> < hatchling==1.27.0
> > >> 413c390
> > >> < httpx==0.27.0
> > >> ---
> > >> > httpx==0.28.1
> > >> 418d394
> > >> < hyperlink==21.0.0
> > >> 421,423d396
> > >> < icdiff==2.0.7
> > >> < id==1.5.0
> > >> < identify==2.6.12
> > >> 426d398
> > >> < imagesize==1.4.1
> > >> 430d401
> > >> < incremental==24.7.2
> > >> 433,435d403
> > >> < iniconfig==2.1.0
> > >> < inputimeout==1.0.4
> > >> < ipdb==0.13.13
> > >> 449d416
> > >> < joserfc==1.2.2
> > >> 451d417
> > >> < jsonpatch==1.33
> > >> 455,456d420
> > >> < jsonpointer==3.0.0
> > >> < jsonschema-path==0.3.4
> > >> 463d426
> > >> < kerberos==1.3.1
> > >> 465,466d427
> > >> < keyrings.alt==5.0.2
> > >> < kgb==7.2
> > >> 494,495d454
> > >> < mmh3==5.1.0
> > >> < mongomock==4.3.0
> > >> 497,498d455
> > >> < moto==5.1.8
> > >> < mpmath==1.3.0
> > >> 508,512d464
> > >> < mypy-boto3-appflow==1.39.0
> > >> < mypy-boto3-rds==1.39.1
> > >> < mypy-boto3-redshift-data==1.39.0
> > >> < mypy-boto3-s3==1.39.5
> > >> < mypy==1.17.0
> > >> 514c466
> > >> < mysql-connector-python==9.3.0
> > >> ---
> > >> > mysql-connector-python==9.4.0
> > >> 521,523d472
> > >> < networkx==3.5
> > >> < nh3==0.3.0
> > >> < nodeenv==1.9.1
> > >> 528,529d476
> > >> < openapi-schema-validator==0.6.3
> > >> < openapi-spec-validator==0.7.2
> > >> 560d506
> > >> < pathable==0.4.4
> > >> 563d508
> > >> < pdbr==0.9.2
> > >> 569,570c514
> > >> < pinotdb==5.6.0
> > >> < pipdeptree==2.28.0
> > >> ---
> > >> > pinotdb==5.7.0
> > >> 574d517
> > >> < plyvel==1.5.1
> > >> 577,579d519
> > >> < pprintpp==0.4.0
> > >> < pre-commit-uv==4.1.4
> > >> < pre_commit==4.2.0
> > >> 592d531
> > >> < py-partiql-parser==0.6.1
> > >> 608d546
> > >> < pyenchant==3.2.2
> > >> 610,611d547
> > >> < pygraphviz==1.14
> > >> < pyiceberg==0.9.1
> > >> 622,632d557
> > >> < pytest-asyncio==0.25.0
> > >> < pytest-cov==6.2.1
> > >> < pytest-custom-exit-code==0.3.0
> > >> < pytest-icdiff==0.9
> > >> < pytest-instafail==0.5.0
> > >> < pytest-mock==3.14.1
> > >> < pytest-rerunfailures==15.1
> > >> < pytest-timeouts==1.2.1
> > >> < pytest-unordered==0.7.0
> > >> < pytest-xdist==3.8.0
> > >> < pytest==8.4.1
> > >> 642d566
> > >> < python-on-whales==0.78.0
> > >> 652d575
> > >> < readme_renderer==44.0
> > >> 659d581
> > >> < requests-mock==1.12.1
> > >> 664,665d585
> > >> < responses==0.25.7
> > >> < restructuredtext_lint==1.4.0
> > >> 667,668d586
> > >> < rfc3339-validator==0.1.4
> > >> < rfc3986==2.0.0
> > >> 670d587
> > >> < rich-click==1.8.9
> > >> 674d590
> > >> < roman-numerals-py==3.1.0
> > >> 679d594
> > >> < ruff==0.12.3
> > >> 688d602
> > >> < semver==3.0.4
> > >> 690,691c604
> > >> < sentinels==1.0.0
> > >> < sentry-sdk==2.33.1
> > >> ---
> > >> > sentry-sdk==2.33.2
> > >> 703d615
> > >> < snowballstemmer==3.0.1
> > >> 709,725d620
> > >> < sphinx-argparse==0.5.2
> > >> < sphinx-autoapi==3.6.0
> > >> < sphinx-autobuild==2024.10.3
> > >> < sphinx-copybutton==0.5.2
> > >> < sphinx-jinja==2.0.2
> > >> < sphinx-rtd-theme==3.0.2
> > >> < sphinx_design==0.6.1
> > >> < sphinxcontrib-applehelp==2.0.0
> > >> < sphinxcontrib-devhelp==2.0.0
> > >> < sphinxcontrib-htmlhelp==2.1.0
> > >> < sphinxcontrib-httpdomain==1.8.1
> > >> < sphinxcontrib-jquery==4.1
> > >> < sphinxcontrib-jsmath==1.0.1
> > >> < sphinxcontrib-qthelp==2.0.0
> > >> < sphinxcontrib-redoc==1.6.0
> > >> < sphinxcontrib-serializinghtml==2.0.0
> > >> < sphinxcontrib-spelling==8.0.1
> > >> 737d631
> > >> < strictyaml==1.7.3
> > >> 740d633
> > >> < sympy==1.14.0
> > >> 752d644
> > >> < time-machine==2.16.0
> > >> 755d646
> > >> < tomli_w==1.2.0
> > >> 758d648
> > >> < towncrier==24.8.0
> > >> 762,763d651
> > >> < trove-classifiers==2025.5.9.12
> > >> < twine==6.1.0
> > >> 765,774d652
> > >> < types-Deprecated==1.2.15.20250304
> > >> < types-Markdown==3.8.0.20250708
> > >> < types-PyMySQL==1.1.0.20250711
> > >> < types-PyYAML==6.0.12.20250516
> > >> < types-aiofiles==24.1.0.20250708
> > >> < types-certifi==2021.10.8.3
> > >> < types-cffi==1.17.0.20250523
> > >> < types-croniter==6.0.0.20250626
> > >> < types-docutils==0.21.0.20250722
> > >> < types-paramiko==3.5.0.20250708
> > >> 776,778d653
> > >> < types-pyOpenSSL==24.1.0.20240722
> > >> < types-python-dateutil==2.9.0.20250708
> > >> < types-python-slugify==8.0.2.20240310
> > >> 780d654
> > >> < types-redis==4.6.0.20241004
> > >> 782,784d655
> > >> < types-setuptools==80.9.0.20250529
> > >> < types-tabulate==0.9.0.20241207
> > >> < types-toml==0.10.8.20240310
> > >> 794d664
> > >> < userpath==1.9.2
> > >> 799c669
> > >> < validators==0.34.0
> > >> ---
> > >> > validators==0.35.0
> > >> 806c676
> > >> < weaviate-client==4.9.6
> > >> ---
> > >> > weaviate-client==4.16.3
> > >> 809c679
> > >> < websockets==14.2
> > >> ---
> > >> > websockets==15.0.1
> > >> 815d684
> > >> < yamllint==1.37.1
> > >> ________________________________
> > >> Strike Technologies, LLC (“Strike”) is part of the GTS family of
> > >> companies. Strike is a technology solutions provider, and is not a
> > broker
> > >> or dealer and does not transact any securities related business
> directly
> > >> whatsoever. This communication is the property of Strike and its
> > >> affiliates, and does not constitute an offer to sell or the
> > solicitation of
> > >> an offer to buy any security in any jurisdiction. It is intended only
> > for
> > >> the person to whom it is addressed and may contain information that is
> > >> privileged, confidential, or otherwise protected from disclosure.
> > >> Distribution or copying of this communication, or the information
> > contained
> > >> herein, by anyone other than the intended recipient is prohibited. If
> > you
> > >> have received this communication in error, please immediately notify
> > Strike
> > >> at info@striketechnologies.com, and delete and destroy any copies
> > hereof.
> > >> ________________________________
> > >>
> > >> CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any
> > attachments
> > >> are intended solely for the addressee. This transmission is covered by
> > the
> > >> Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The
> > >> information contained in this transmission is confidential in nature
> and
> > >> protected from further use or disclosure under U.S. Pub. L. 106-102,
> 113
> > >> U.S. Stat. 1338 (1999), and may be subject to attorney-client or other
> > >> legal privilege. Your use or disclosure of this information for any
> > purpose
> > >> other than that intended by its transmittal is strictly prohibited,
> and
> > may
> > >> subject you to fines and/or penalties under federal and state law. If
> > you
> > >> are not the intended recipient of this transmission, please DESTROY
> ALL
> > >> COPIES RECEIVED and confirm destruction to the sender via return
> > >> transmittal.
> > >>
> > >
> >
>
Re: [DISCUSS] Removing dev dependencies from PyPI constraints ?
Posted by Pavankumar Gopidesu <go...@gmail.com>.
Thanks Jarek,
I am fine with option b as far as if we find anything while testing we can
downgrade.
Pavan
On Thu, Jul 24, 2025 at 10:44 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Would love to hear more if anyone has an opinion :). Yep. It's an advanced
> topic and I can make arbitrary decision :) ...
>
> I am thinking now that it should be quite "OK" to accept those "potentially
> untested" versions - especially that they will be tested during manual
> testing. Unless someone objects i will likely update PR with some changelog
> / newsfragment to announce the change in 3.1. I will run LAZY CONSENSUS
> soon if I don't hear any opposition.
>
> J.
>
>
> On Tue, Jul 22, 2025 at 5:52 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > > Exactly to remove the dev requirements airflow has from interfering
> with
> > our own dev requirements. Though this approach notably doesn't change any
> > of the used non-dev requirements, it's not clear to me (and I'm probably
> > just missing something) why your diff includes non-dev requirements.
> >
> > The thing is that I do not "remove" devel features - because due to
> > transitive dependencies I would have to go deep down the rabbit hole
> which
> > dependencies are "devel" and which not. So what I do, I ask `uv` to do
> the
> > job for me and repeat the resolution but with the `--exact` flag
> (combined
> > with the already used `--resolution higest` and `--no-sources`).
> >
> > Apparently in this case `uv` will sometimes decide that it's ok to bump
> > some dependencies that are left after initial resolution. The thing is
> that
> > uv will not install some versions of some packages if they are
> conflicting
> > with already installed packages. What I believe `uv` does when I use the
> *--exact
> > *flag of `uv pip install` it "fixes" the initial resolution. It finds out
> > what is needed, removes those packages that are not needed, and then it
> > resolves again and improves the resolution (that would be my guess at
> least
> > - it could be done slightly differently and in a single step I guess.
> >
> > Essentially the difference between the left and right side is:
> >
> > a) install absolutely everything from the workspace
> >
> > b) do this:
> > left: uv pip install --no-sources 'apache-airflow[all]'
> > 'apache-airflow-core[all]' apache-airflow-task-sdk ./airflow-ctl
> > --reinstall --resolution highest --find-links file:///dist
> > right: uv pip install --no-sources *--exact --strict*
> > 'apache-airflow[all]' 'apache-airflow-core[all]' apache-airflow-task-sdk
> > ./airflow-ctl --reinstall --resolution highest --find-links file:///dist
> >
> > In the current state content of the fille://dist should be the same in
> > both cases:
> >
> > apache-airflow-providers-edge3 (1.1.2): wheel -
> > apache_airflow_providers_edge3-1.1.2-py3-none-any.whl
> > apache-airflow-providers-keycloak (1.0.0): wheel -
> > apache_airflow_providers_keycloak-1.0.0-py3-none-any.whl
> > apache-airflow-providers-snowflake (6.5.1): wheel -
> > apache_airflow_providers_snowflake-6.5.1-py3-none-any.whl
> >
> > This is a bit of in-process quirk of the "PyPI" resolution - those are
> the
> > three packages that we have updated versions of in the sources, but they
> > are not released yet. But they should be essentially the same in both
> > cases. So the difference between left and right is `--exact --strict`
> > (strict I added to make sure that we are not removing something that is
> > still needed).
> >
> > J.
> >
> >
> >
> >
> >
> > On Tue, Jul 22, 2025 at 5:22 PM Damian Shaw <
> dshaw@striketechnologies.com>
> > wrote:
> >
> >> As I user of Airflow I have ended up implementing this for my
> >> environments:
> >>
> >> pip install apache-airflow[all or select-extras] -c {official
> >> constraints} > smaller-run-time-constraints.txt
> >> pip-compile {all my requirements including development}.txt -c
> >> smaller-run-time-constraints.txt
> >>
> >> Exactly to remove the dev requirements airflow has from interfering with
> >> our own dev requirements. Though this approach notably doesn't change
> any
> >> of the used non-dev requirements, it's not clear to me (and I'm probably
> >> just missing something) why your diff includes non-dev requirements.
> >>
> >> Damian
> >>
> >> -----Original Message-----
> >> From: Jarek Potiuk <ja...@potiuk.com>
> >> Sent: Tuesday, July 22, 2025 11:04 AM
> >> To: dev@airflow.apache.org
> >> Subject: [DISCUSS] Removing dev dependencies from PyPI constraints ?
> >>
> >> Hello here,
> >>
> >>
> >> *Tl;DR; We had an interesting discussion on Slack today and I wanted to
> >> bring it here to discuss what we should do - whether we should keep
> devel
> >> dependencies in our PyPI constraints or not.*
> >>
> >> *A bit of context:*
> >> In our CI we generate several types of constraints. one of them are
> >> "source" constraints that we use during our CI testing - including all
> our
> >> testing tools, mypy, ruff etc., and another type is "PyPI" constraint -
> >> which is more <What should be the "golden" versions of dependencies when
> >> you install Airflow using `pip install` for production>.
> >>
> >> The main difference between the two (roughly) is that "source
> constraints"
> >> only uses dependencies from "branch tip" ("main", "v3-0-test"), where
> >> "PyPI constraints" uses "Airflow core" branch-tip dependencies - but all
> >> the providers are installed from PyPI. There are few nuances when some
> >> packages are being released but basically - for example for Airflow
> 3.0.3,
> >> "PyPI"
> >> constraints is "what is the current set of dependencies and providers
> >> when we install airflow from PyPI at the moment of release".
> >>
> >> So target for "source" constraints are "Airflow contributors" where
> >> target for "PyPI constraints" are "Airflow users".
> >>
> >>
> >> *The problem*
> >> In the slack discussion
> >> https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1753169914722049
> >> - slack user @dashmug (sorry - do not know name) noticed that the
> "PyPI"
> >> constraints also contain Airflow development dependencies listed (like
> >> ruff or mypy). That made me think a bit and I quickly came up with PR
> that
> >> could "fix" it - i.e. only keep the "real" constraints in.
> >> https://github.com/apache/airflow/pull/53631 - it's as easy as adding
> >> the `--exact` (and `--strict` for verification) flag to the `uv pip
> >> install` command we use.
> >>
> >> I agree it's a bit confusing to have the devel dependencies there - it
> is
> >> not the "intention", but there is very little problem with it - the fact
> >> that mypy or ruff are listed with a version in constraints, does not
> mean
> >> it must be installed - constraints mainly say "if you install this, it
> >> should be this version" . And there is no problem whatsoever (and it is
> >> even recommended by us) to have "airflow with its deps" installed with
> >> constraints, and then anything else you need to install or update -
> >> without. So other than potential confusion, it usually does not cause
> harm.
> >>
> >> Now... It might seem that removing devel deps has no side effects and we
> >> should **just do it** - but it's not that easy. Main problem is that
> those
> >> devel dependencies might "hold" other dependencies (directly or
> >> transitively) from being upgraded - and when we remove the devel
> >> dependencies and do the resolution again - we might have a DIFFERENT
> set of
> >> dependencies, simply because removing the devel dependencies, might
> "free"
> >> the other deps and they might get upgraded. Yeah - I know it's not
> >> obvious for someone who did not spend 3 years solving dependency issues
> :).
> >>
> >> And the problem with it is that because we need those devel deps to test
> >> things - the upgraded dependencies might have never been tested in our
> CI -
> >> because we never upgraded to later versions. It's a bit of
> >> Heisenberg-effect. By using test tools we are changing the state of the
> >> tested thing. Currently our "PyPI" constraints reflect the state of
> Airflow
> >> "when the testing tools are installed" - and we are generally not able
> to
> >> test the state of Airflow "without the testing tools" - by the sheer
> fact
> >> that we do not have those tools :)
> >>
> >> Just to illustrate the case - you can see what happens when we removed
> >> devel deps:
> >> https://github.com/apache/airflow/actions/runs/16444553234 - in summary
> >> you can see a colored version that is a bit clearer but I dumped the
> diff
> >> below from an example - Python 3.10 case. Left side < is "what was with
> >> testing tools" and > is without them. You can see a LOT of test tools
> >> removed (pytest, ruff and others - including dependencies used only by
> >> those tools)
> >>
> >> But also you can see (likely some of the devel/test deps limited those)
> >>
> >> * Authlib - upgraded from 1.3.1 to 1.6.1
> >> * Google - genai upgraded from 1.2.0 to 1.20.0
> >> * Httpx - upgraded from 0.27.0 to 0.28.1
> >> * Pinotdb - upgraded from 5.6.0 to 5.7.0
> >> * Sentry-sdk - upgraded from 2.33.1 to 2.33.2 (this is false positive
> as
> >> it has just been released and new version was picked)
> >> * validators - upgraded form 0.34.0 to 0.35.0
> >> * weaviate-client - upgraded from 4.9.0 to 4.16.3
> >> * websockets - upgraded from 14.2 to 15.0.1
> >>
> >>
> >> This means that all the "upgraded" packages above have not been - likely
> >> ever tested in our CI.
> >>
> >> And when we put them in "versioned" constraints, it might mean that they
> >> will be obviously failing.
> >>
> >> So ... there is a risk connected with just generating the "non-devel"
> >> constraints without additional tests. But that also means that the user
> >> who does not use constraints might get those versions anyway - default
> >> behaviour of `pip install airflow` will be to install those upgraded
> >> versions above if you do not use constraints. But also that is no
> different
> >> than installing a released airflow version a few weeks or months after
> >> release without constraints - there will be tens of packages that were
> >> released since our release and installing airflow without constraints
> will
> >> pick those.
> >>
> >> However - when we manually install RC candidates - say by running
> airflow
> >> from rc1 image, we will use those "upgraded" dependencies - so some
> >> "obvious" problems might be caught during the rc manual testing - where
> you
> >> do not use test tools but run airflow "as if" it was installed in
> >> production.
> >>
> >> *The solution ? *
> >>
> >> There are few ways we can proceed with it:
> >>
> >> a) leave it as is - no big harm, potential confusion, by people who see
> >> it
> >> - potential differences
> >>
> >> b) remove devel following my PR and take the risk that we never tested
> (in
> >> CI) the dependencies we set in the constraints. That's a bit risky, but
> >> we can always reactively downgrade some of those packages after the
> fact if
> >> users will start reporting the issues
> >>
> >> c) potentially most complex thing - trying to remove devel dependencies
> >> somewhat combining the two - for example removing all the deps that
> would
> >> be removed in b) from a) - but that has also some edge cases - when
> >> different versions of dependencies can add or upgrade their
> dependencies,
> >> we are very likely to occasionally remove or add too much.
> >>
> >>
> >> I wonder (for those who managed to read that far - likely not many -
> what
> >> do you think here? What would be the best option ? Maybe there are other
> >> options I have not thought about ?
> >>
> >>
> >> J
> >>
> >>
> >> -----
> >>
> >> The diff for Python 3.10
> >>
> >>
> >>
> >> 34c34
> >> < Authlib==1.3.1
> >> ---
> >> > Authlib==1.6.1
> >> 64d63
> >> < Sphinx==8.2.3
> >> 78d76
> >> < aioresponses==0.7.8
> >> 82d79
> >> < alabaster==1.0.0
> >> 198d194
> >> < arro3-core==0.5.1
> >> 202d197
> >> < astroid==3.3.11
> >> 211,212d205
> >> < aws-sam-translator==1.99.0
> >> < aws-xray-sdk==2.14.0
> >> 258,259d250
> >> < cfgv==3.4.0
> >> < cfn-lint==1.38.0
> >> 262d252
> >> < checksumdir==1.2.0
> >> 277d266
> >> < coverage==7.9.2
> >> 291d279
> >> < deltalake==1.1.0
> >> 293d280
> >> < diagrams==0.24.4
> >> 301,302d287
> >> < docutils==0.21.2
> >> < duckdb==1.3.2
> >> 308d292
> >> < eralchemy2==1.4.1
> >> 311d294
> >> < execnet==2.1.1
> >> 321,322d303
> >> < flit==3.12.0
> >> < flit_core==3.12.0
> >> 386c367
> >> < google-genai==1.2.0
> >> ---
> >> > google-genai==1.20.0
> >> 389d369
> >> < graphql-core==3.2.6
> >> 398d377
> >> < grpcio-tools==1.62.3
> >> 403,404d381
> >> < hatch==1.14.1
> >> < hatchling==1.27.0
> >> 413c390
> >> < httpx==0.27.0
> >> ---
> >> > httpx==0.28.1
> >> 418d394
> >> < hyperlink==21.0.0
> >> 421,423d396
> >> < icdiff==2.0.7
> >> < id==1.5.0
> >> < identify==2.6.12
> >> 426d398
> >> < imagesize==1.4.1
> >> 430d401
> >> < incremental==24.7.2
> >> 433,435d403
> >> < iniconfig==2.1.0
> >> < inputimeout==1.0.4
> >> < ipdb==0.13.13
> >> 449d416
> >> < joserfc==1.2.2
> >> 451d417
> >> < jsonpatch==1.33
> >> 455,456d420
> >> < jsonpointer==3.0.0
> >> < jsonschema-path==0.3.4
> >> 463d426
> >> < kerberos==1.3.1
> >> 465,466d427
> >> < keyrings.alt==5.0.2
> >> < kgb==7.2
> >> 494,495d454
> >> < mmh3==5.1.0
> >> < mongomock==4.3.0
> >> 497,498d455
> >> < moto==5.1.8
> >> < mpmath==1.3.0
> >> 508,512d464
> >> < mypy-boto3-appflow==1.39.0
> >> < mypy-boto3-rds==1.39.1
> >> < mypy-boto3-redshift-data==1.39.0
> >> < mypy-boto3-s3==1.39.5
> >> < mypy==1.17.0
> >> 514c466
> >> < mysql-connector-python==9.3.0
> >> ---
> >> > mysql-connector-python==9.4.0
> >> 521,523d472
> >> < networkx==3.5
> >> < nh3==0.3.0
> >> < nodeenv==1.9.1
> >> 528,529d476
> >> < openapi-schema-validator==0.6.3
> >> < openapi-spec-validator==0.7.2
> >> 560d506
> >> < pathable==0.4.4
> >> 563d508
> >> < pdbr==0.9.2
> >> 569,570c514
> >> < pinotdb==5.6.0
> >> < pipdeptree==2.28.0
> >> ---
> >> > pinotdb==5.7.0
> >> 574d517
> >> < plyvel==1.5.1
> >> 577,579d519
> >> < pprintpp==0.4.0
> >> < pre-commit-uv==4.1.4
> >> < pre_commit==4.2.0
> >> 592d531
> >> < py-partiql-parser==0.6.1
> >> 608d546
> >> < pyenchant==3.2.2
> >> 610,611d547
> >> < pygraphviz==1.14
> >> < pyiceberg==0.9.1
> >> 622,632d557
> >> < pytest-asyncio==0.25.0
> >> < pytest-cov==6.2.1
> >> < pytest-custom-exit-code==0.3.0
> >> < pytest-icdiff==0.9
> >> < pytest-instafail==0.5.0
> >> < pytest-mock==3.14.1
> >> < pytest-rerunfailures==15.1
> >> < pytest-timeouts==1.2.1
> >> < pytest-unordered==0.7.0
> >> < pytest-xdist==3.8.0
> >> < pytest==8.4.1
> >> 642d566
> >> < python-on-whales==0.78.0
> >> 652d575
> >> < readme_renderer==44.0
> >> 659d581
> >> < requests-mock==1.12.1
> >> 664,665d585
> >> < responses==0.25.7
> >> < restructuredtext_lint==1.4.0
> >> 667,668d586
> >> < rfc3339-validator==0.1.4
> >> < rfc3986==2.0.0
> >> 670d587
> >> < rich-click==1.8.9
> >> 674d590
> >> < roman-numerals-py==3.1.0
> >> 679d594
> >> < ruff==0.12.3
> >> 688d602
> >> < semver==3.0.4
> >> 690,691c604
> >> < sentinels==1.0.0
> >> < sentry-sdk==2.33.1
> >> ---
> >> > sentry-sdk==2.33.2
> >> 703d615
> >> < snowballstemmer==3.0.1
> >> 709,725d620
> >> < sphinx-argparse==0.5.2
> >> < sphinx-autoapi==3.6.0
> >> < sphinx-autobuild==2024.10.3
> >> < sphinx-copybutton==0.5.2
> >> < sphinx-jinja==2.0.2
> >> < sphinx-rtd-theme==3.0.2
> >> < sphinx_design==0.6.1
> >> < sphinxcontrib-applehelp==2.0.0
> >> < sphinxcontrib-devhelp==2.0.0
> >> < sphinxcontrib-htmlhelp==2.1.0
> >> < sphinxcontrib-httpdomain==1.8.1
> >> < sphinxcontrib-jquery==4.1
> >> < sphinxcontrib-jsmath==1.0.1
> >> < sphinxcontrib-qthelp==2.0.0
> >> < sphinxcontrib-redoc==1.6.0
> >> < sphinxcontrib-serializinghtml==2.0.0
> >> < sphinxcontrib-spelling==8.0.1
> >> 737d631
> >> < strictyaml==1.7.3
> >> 740d633
> >> < sympy==1.14.0
> >> 752d644
> >> < time-machine==2.16.0
> >> 755d646
> >> < tomli_w==1.2.0
> >> 758d648
> >> < towncrier==24.8.0
> >> 762,763d651
> >> < trove-classifiers==2025.5.9.12
> >> < twine==6.1.0
> >> 765,774d652
> >> < types-Deprecated==1.2.15.20250304
> >> < types-Markdown==3.8.0.20250708
> >> < types-PyMySQL==1.1.0.20250711
> >> < types-PyYAML==6.0.12.20250516
> >> < types-aiofiles==24.1.0.20250708
> >> < types-certifi==2021.10.8.3
> >> < types-cffi==1.17.0.20250523
> >> < types-croniter==6.0.0.20250626
> >> < types-docutils==0.21.0.20250722
> >> < types-paramiko==3.5.0.20250708
> >> 776,778d653
> >> < types-pyOpenSSL==24.1.0.20240722
> >> < types-python-dateutil==2.9.0.20250708
> >> < types-python-slugify==8.0.2.20240310
> >> 780d654
> >> < types-redis==4.6.0.20241004
> >> 782,784d655
> >> < types-setuptools==80.9.0.20250529
> >> < types-tabulate==0.9.0.20241207
> >> < types-toml==0.10.8.20240310
> >> 794d664
> >> < userpath==1.9.2
> >> 799c669
> >> < validators==0.34.0
> >> ---
> >> > validators==0.35.0
> >> 806c676
> >> < weaviate-client==4.9.6
> >> ---
> >> > weaviate-client==4.16.3
> >> 809c679
> >> < websockets==14.2
> >> ---
> >> > websockets==15.0.1
> >> 815d684
> >> < yamllint==1.37.1
> >> ________________________________
> >> Strike Technologies, LLC (“Strike”) is part of the GTS family of
> >> companies. Strike is a technology solutions provider, and is not a
> broker
> >> or dealer and does not transact any securities related business directly
> >> whatsoever. This communication is the property of Strike and its
> >> affiliates, and does not constitute an offer to sell or the
> solicitation of
> >> an offer to buy any security in any jurisdiction. It is intended only
> for
> >> the person to whom it is addressed and may contain information that is
> >> privileged, confidential, or otherwise protected from disclosure.
> >> Distribution or copying of this communication, or the information
> contained
> >> herein, by anyone other than the intended recipient is prohibited. If
> you
> >> have received this communication in error, please immediately notify
> Strike
> >> at info@striketechnologies.com, and delete and destroy any copies
> hereof.
> >> ________________________________
> >>
> >> CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any
> attachments
> >> are intended solely for the addressee. This transmission is covered by
> the
> >> Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The
> >> information contained in this transmission is confidential in nature and
> >> protected from further use or disclosure under U.S. Pub. L. 106-102, 113
> >> U.S. Stat. 1338 (1999), and may be subject to attorney-client or other
> >> legal privilege. Your use or disclosure of this information for any
> purpose
> >> other than that intended by its transmittal is strictly prohibited, and
> may
> >> subject you to fines and/or penalties under federal and state law. If
> you
> >> are not the intended recipient of this transmission, please DESTROY ALL
> >> COPIES RECEIVED and confirm destruction to the sender via return
> >> transmittal.
> >>
> >
>
Re: [DISCUSS] Removing dev dependencies from PyPI constraints ?
Posted by Jarek Potiuk <ja...@potiuk.com>.
Would love to hear more if anyone has an opinion :). Yep. It's an advanced
topic and I can make arbitrary decision :) ...
I am thinking now that it should be quite "OK" to accept those "potentially
untested" versions - especially that they will be tested during manual
testing. Unless someone objects i will likely update PR with some changelog
/ newsfragment to announce the change in 3.1. I will run LAZY CONSENSUS
soon if I don't hear any opposition.
J.
On Tue, Jul 22, 2025 at 5:52 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> > Exactly to remove the dev requirements airflow has from interfering with
> our own dev requirements. Though this approach notably doesn't change any
> of the used non-dev requirements, it's not clear to me (and I'm probably
> just missing something) why your diff includes non-dev requirements.
>
> The thing is that I do not "remove" devel features - because due to
> transitive dependencies I would have to go deep down the rabbit hole which
> dependencies are "devel" and which not. So what I do, I ask `uv` to do the
> job for me and repeat the resolution but with the `--exact` flag (combined
> with the already used `--resolution higest` and `--no-sources`).
>
> Apparently in this case `uv` will sometimes decide that it's ok to bump
> some dependencies that are left after initial resolution. The thing is that
> uv will not install some versions of some packages if they are conflicting
> with already installed packages. What I believe `uv` does when I use the *--exact
> *flag of `uv pip install` it "fixes" the initial resolution. It finds out
> what is needed, removes those packages that are not needed, and then it
> resolves again and improves the resolution (that would be my guess at least
> - it could be done slightly differently and in a single step I guess.
>
> Essentially the difference between the left and right side is:
>
> a) install absolutely everything from the workspace
>
> b) do this:
> left: uv pip install --no-sources 'apache-airflow[all]'
> 'apache-airflow-core[all]' apache-airflow-task-sdk ./airflow-ctl
> --reinstall --resolution highest --find-links file:///dist
> right: uv pip install --no-sources *--exact --strict*
> 'apache-airflow[all]' 'apache-airflow-core[all]' apache-airflow-task-sdk
> ./airflow-ctl --reinstall --resolution highest --find-links file:///dist
>
> In the current state content of the fille://dist should be the same in
> both cases:
>
> apache-airflow-providers-edge3 (1.1.2): wheel -
> apache_airflow_providers_edge3-1.1.2-py3-none-any.whl
> apache-airflow-providers-keycloak (1.0.0): wheel -
> apache_airflow_providers_keycloak-1.0.0-py3-none-any.whl
> apache-airflow-providers-snowflake (6.5.1): wheel -
> apache_airflow_providers_snowflake-6.5.1-py3-none-any.whl
>
> This is a bit of in-process quirk of the "PyPI" resolution - those are the
> three packages that we have updated versions of in the sources, but they
> are not released yet. But they should be essentially the same in both
> cases. So the difference between left and right is `--exact --strict`
> (strict I added to make sure that we are not removing something that is
> still needed).
>
> J.
>
>
>
>
>
> On Tue, Jul 22, 2025 at 5:22 PM Damian Shaw <ds...@striketechnologies.com>
> wrote:
>
>> As I user of Airflow I have ended up implementing this for my
>> environments:
>>
>> pip install apache-airflow[all or select-extras] -c {official
>> constraints} > smaller-run-time-constraints.txt
>> pip-compile {all my requirements including development}.txt -c
>> smaller-run-time-constraints.txt
>>
>> Exactly to remove the dev requirements airflow has from interfering with
>> our own dev requirements. Though this approach notably doesn't change any
>> of the used non-dev requirements, it's not clear to me (and I'm probably
>> just missing something) why your diff includes non-dev requirements.
>>
>> Damian
>>
>> -----Original Message-----
>> From: Jarek Potiuk <ja...@potiuk.com>
>> Sent: Tuesday, July 22, 2025 11:04 AM
>> To: dev@airflow.apache.org
>> Subject: [DISCUSS] Removing dev dependencies from PyPI constraints ?
>>
>> Hello here,
>>
>>
>> *Tl;DR; We had an interesting discussion on Slack today and I wanted to
>> bring it here to discuss what we should do - whether we should keep devel
>> dependencies in our PyPI constraints or not.*
>>
>> *A bit of context:*
>> In our CI we generate several types of constraints. one of them are
>> "source" constraints that we use during our CI testing - including all our
>> testing tools, mypy, ruff etc., and another type is "PyPI" constraint -
>> which is more <What should be the "golden" versions of dependencies when
>> you install Airflow using `pip install` for production>.
>>
>> The main difference between the two (roughly) is that "source constraints"
>> only uses dependencies from "branch tip" ("main", "v3-0-test"), where
>> "PyPI constraints" uses "Airflow core" branch-tip dependencies - but all
>> the providers are installed from PyPI. There are few nuances when some
>> packages are being released but basically - for example for Airflow 3.0.3,
>> "PyPI"
>> constraints is "what is the current set of dependencies and providers
>> when we install airflow from PyPI at the moment of release".
>>
>> So target for "source" constraints are "Airflow contributors" where
>> target for "PyPI constraints" are "Airflow users".
>>
>>
>> *The problem*
>> In the slack discussion
>> https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1753169914722049
>> - slack user @dashmug (sorry - do not know name) noticed that the "PyPI"
>> constraints also contain Airflow development dependencies listed (like
>> ruff or mypy). That made me think a bit and I quickly came up with PR that
>> could "fix" it - i.e. only keep the "real" constraints in.
>> https://github.com/apache/airflow/pull/53631 - it's as easy as adding
>> the `--exact` (and `--strict` for verification) flag to the `uv pip
>> install` command we use.
>>
>> I agree it's a bit confusing to have the devel dependencies there - it is
>> not the "intention", but there is very little problem with it - the fact
>> that mypy or ruff are listed with a version in constraints, does not mean
>> it must be installed - constraints mainly say "if you install this, it
>> should be this version" . And there is no problem whatsoever (and it is
>> even recommended by us) to have "airflow with its deps" installed with
>> constraints, and then anything else you need to install or update -
>> without. So other than potential confusion, it usually does not cause harm.
>>
>> Now... It might seem that removing devel deps has no side effects and we
>> should **just do it** - but it's not that easy. Main problem is that those
>> devel dependencies might "hold" other dependencies (directly or
>> transitively) from being upgraded - and when we remove the devel
>> dependencies and do the resolution again - we might have a DIFFERENT set of
>> dependencies, simply because removing the devel dependencies, might "free"
>> the other deps and they might get upgraded. Yeah - I know it's not
>> obvious for someone who did not spend 3 years solving dependency issues :).
>>
>> And the problem with it is that because we need those devel deps to test
>> things - the upgraded dependencies might have never been tested in our CI -
>> because we never upgraded to later versions. It's a bit of
>> Heisenberg-effect. By using test tools we are changing the state of the
>> tested thing. Currently our "PyPI" constraints reflect the state of Airflow
>> "when the testing tools are installed" - and we are generally not able to
>> test the state of Airflow "without the testing tools" - by the sheer fact
>> that we do not have those tools :)
>>
>> Just to illustrate the case - you can see what happens when we removed
>> devel deps:
>> https://github.com/apache/airflow/actions/runs/16444553234 - in summary
>> you can see a colored version that is a bit clearer but I dumped the diff
>> below from an example - Python 3.10 case. Left side < is "what was with
>> testing tools" and > is without them. You can see a LOT of test tools
>> removed (pytest, ruff and others - including dependencies used only by
>> those tools)
>>
>> But also you can see (likely some of the devel/test deps limited those)
>>
>> * Authlib - upgraded from 1.3.1 to 1.6.1
>> * Google - genai upgraded from 1.2.0 to 1.20.0
>> * Httpx - upgraded from 0.27.0 to 0.28.1
>> * Pinotdb - upgraded from 5.6.0 to 5.7.0
>> * Sentry-sdk - upgraded from 2.33.1 to 2.33.2 (this is false positive as
>> it has just been released and new version was picked)
>> * validators - upgraded form 0.34.0 to 0.35.0
>> * weaviate-client - upgraded from 4.9.0 to 4.16.3
>> * websockets - upgraded from 14.2 to 15.0.1
>>
>>
>> This means that all the "upgraded" packages above have not been - likely
>> ever tested in our CI.
>>
>> And when we put them in "versioned" constraints, it might mean that they
>> will be obviously failing.
>>
>> So ... there is a risk connected with just generating the "non-devel"
>> constraints without additional tests. But that also means that the user
>> who does not use constraints might get those versions anyway - default
>> behaviour of `pip install airflow` will be to install those upgraded
>> versions above if you do not use constraints. But also that is no different
>> than installing a released airflow version a few weeks or months after
>> release without constraints - there will be tens of packages that were
>> released since our release and installing airflow without constraints will
>> pick those.
>>
>> However - when we manually install RC candidates - say by running airflow
>> from rc1 image, we will use those "upgraded" dependencies - so some
>> "obvious" problems might be caught during the rc manual testing - where you
>> do not use test tools but run airflow "as if" it was installed in
>> production.
>>
>> *The solution ? *
>>
>> There are few ways we can proceed with it:
>>
>> a) leave it as is - no big harm, potential confusion, by people who see
>> it
>> - potential differences
>>
>> b) remove devel following my PR and take the risk that we never tested (in
>> CI) the dependencies we set in the constraints. That's a bit risky, but
>> we can always reactively downgrade some of those packages after the fact if
>> users will start reporting the issues
>>
>> c) potentially most complex thing - trying to remove devel dependencies
>> somewhat combining the two - for example removing all the deps that would
>> be removed in b) from a) - but that has also some edge cases - when
>> different versions of dependencies can add or upgrade their dependencies,
>> we are very likely to occasionally remove or add too much.
>>
>>
>> I wonder (for those who managed to read that far - likely not many - what
>> do you think here? What would be the best option ? Maybe there are other
>> options I have not thought about ?
>>
>>
>> J
>>
>>
>> -----
>>
>> The diff for Python 3.10
>>
>>
>>
>> 34c34
>> < Authlib==1.3.1
>> ---
>> > Authlib==1.6.1
>> 64d63
>> < Sphinx==8.2.3
>> 78d76
>> < aioresponses==0.7.8
>> 82d79
>> < alabaster==1.0.0
>> 198d194
>> < arro3-core==0.5.1
>> 202d197
>> < astroid==3.3.11
>> 211,212d205
>> < aws-sam-translator==1.99.0
>> < aws-xray-sdk==2.14.0
>> 258,259d250
>> < cfgv==3.4.0
>> < cfn-lint==1.38.0
>> 262d252
>> < checksumdir==1.2.0
>> 277d266
>> < coverage==7.9.2
>> 291d279
>> < deltalake==1.1.0
>> 293d280
>> < diagrams==0.24.4
>> 301,302d287
>> < docutils==0.21.2
>> < duckdb==1.3.2
>> 308d292
>> < eralchemy2==1.4.1
>> 311d294
>> < execnet==2.1.1
>> 321,322d303
>> < flit==3.12.0
>> < flit_core==3.12.0
>> 386c367
>> < google-genai==1.2.0
>> ---
>> > google-genai==1.20.0
>> 389d369
>> < graphql-core==3.2.6
>> 398d377
>> < grpcio-tools==1.62.3
>> 403,404d381
>> < hatch==1.14.1
>> < hatchling==1.27.0
>> 413c390
>> < httpx==0.27.0
>> ---
>> > httpx==0.28.1
>> 418d394
>> < hyperlink==21.0.0
>> 421,423d396
>> < icdiff==2.0.7
>> < id==1.5.0
>> < identify==2.6.12
>> 426d398
>> < imagesize==1.4.1
>> 430d401
>> < incremental==24.7.2
>> 433,435d403
>> < iniconfig==2.1.0
>> < inputimeout==1.0.4
>> < ipdb==0.13.13
>> 449d416
>> < joserfc==1.2.2
>> 451d417
>> < jsonpatch==1.33
>> 455,456d420
>> < jsonpointer==3.0.0
>> < jsonschema-path==0.3.4
>> 463d426
>> < kerberos==1.3.1
>> 465,466d427
>> < keyrings.alt==5.0.2
>> < kgb==7.2
>> 494,495d454
>> < mmh3==5.1.0
>> < mongomock==4.3.0
>> 497,498d455
>> < moto==5.1.8
>> < mpmath==1.3.0
>> 508,512d464
>> < mypy-boto3-appflow==1.39.0
>> < mypy-boto3-rds==1.39.1
>> < mypy-boto3-redshift-data==1.39.0
>> < mypy-boto3-s3==1.39.5
>> < mypy==1.17.0
>> 514c466
>> < mysql-connector-python==9.3.0
>> ---
>> > mysql-connector-python==9.4.0
>> 521,523d472
>> < networkx==3.5
>> < nh3==0.3.0
>> < nodeenv==1.9.1
>> 528,529d476
>> < openapi-schema-validator==0.6.3
>> < openapi-spec-validator==0.7.2
>> 560d506
>> < pathable==0.4.4
>> 563d508
>> < pdbr==0.9.2
>> 569,570c514
>> < pinotdb==5.6.0
>> < pipdeptree==2.28.0
>> ---
>> > pinotdb==5.7.0
>> 574d517
>> < plyvel==1.5.1
>> 577,579d519
>> < pprintpp==0.4.0
>> < pre-commit-uv==4.1.4
>> < pre_commit==4.2.0
>> 592d531
>> < py-partiql-parser==0.6.1
>> 608d546
>> < pyenchant==3.2.2
>> 610,611d547
>> < pygraphviz==1.14
>> < pyiceberg==0.9.1
>> 622,632d557
>> < pytest-asyncio==0.25.0
>> < pytest-cov==6.2.1
>> < pytest-custom-exit-code==0.3.0
>> < pytest-icdiff==0.9
>> < pytest-instafail==0.5.0
>> < pytest-mock==3.14.1
>> < pytest-rerunfailures==15.1
>> < pytest-timeouts==1.2.1
>> < pytest-unordered==0.7.0
>> < pytest-xdist==3.8.0
>> < pytest==8.4.1
>> 642d566
>> < python-on-whales==0.78.0
>> 652d575
>> < readme_renderer==44.0
>> 659d581
>> < requests-mock==1.12.1
>> 664,665d585
>> < responses==0.25.7
>> < restructuredtext_lint==1.4.0
>> 667,668d586
>> < rfc3339-validator==0.1.4
>> < rfc3986==2.0.0
>> 670d587
>> < rich-click==1.8.9
>> 674d590
>> < roman-numerals-py==3.1.0
>> 679d594
>> < ruff==0.12.3
>> 688d602
>> < semver==3.0.4
>> 690,691c604
>> < sentinels==1.0.0
>> < sentry-sdk==2.33.1
>> ---
>> > sentry-sdk==2.33.2
>> 703d615
>> < snowballstemmer==3.0.1
>> 709,725d620
>> < sphinx-argparse==0.5.2
>> < sphinx-autoapi==3.6.0
>> < sphinx-autobuild==2024.10.3
>> < sphinx-copybutton==0.5.2
>> < sphinx-jinja==2.0.2
>> < sphinx-rtd-theme==3.0.2
>> < sphinx_design==0.6.1
>> < sphinxcontrib-applehelp==2.0.0
>> < sphinxcontrib-devhelp==2.0.0
>> < sphinxcontrib-htmlhelp==2.1.0
>> < sphinxcontrib-httpdomain==1.8.1
>> < sphinxcontrib-jquery==4.1
>> < sphinxcontrib-jsmath==1.0.1
>> < sphinxcontrib-qthelp==2.0.0
>> < sphinxcontrib-redoc==1.6.0
>> < sphinxcontrib-serializinghtml==2.0.0
>> < sphinxcontrib-spelling==8.0.1
>> 737d631
>> < strictyaml==1.7.3
>> 740d633
>> < sympy==1.14.0
>> 752d644
>> < time-machine==2.16.0
>> 755d646
>> < tomli_w==1.2.0
>> 758d648
>> < towncrier==24.8.0
>> 762,763d651
>> < trove-classifiers==2025.5.9.12
>> < twine==6.1.0
>> 765,774d652
>> < types-Deprecated==1.2.15.20250304
>> < types-Markdown==3.8.0.20250708
>> < types-PyMySQL==1.1.0.20250711
>> < types-PyYAML==6.0.12.20250516
>> < types-aiofiles==24.1.0.20250708
>> < types-certifi==2021.10.8.3
>> < types-cffi==1.17.0.20250523
>> < types-croniter==6.0.0.20250626
>> < types-docutils==0.21.0.20250722
>> < types-paramiko==3.5.0.20250708
>> 776,778d653
>> < types-pyOpenSSL==24.1.0.20240722
>> < types-python-dateutil==2.9.0.20250708
>> < types-python-slugify==8.0.2.20240310
>> 780d654
>> < types-redis==4.6.0.20241004
>> 782,784d655
>> < types-setuptools==80.9.0.20250529
>> < types-tabulate==0.9.0.20241207
>> < types-toml==0.10.8.20240310
>> 794d664
>> < userpath==1.9.2
>> 799c669
>> < validators==0.34.0
>> ---
>> > validators==0.35.0
>> 806c676
>> < weaviate-client==4.9.6
>> ---
>> > weaviate-client==4.16.3
>> 809c679
>> < websockets==14.2
>> ---
>> > websockets==15.0.1
>> 815d684
>> < yamllint==1.37.1
>> ________________________________
>> Strike Technologies, LLC (“Strike”) is part of the GTS family of
>> companies. Strike is a technology solutions provider, and is not a broker
>> or dealer and does not transact any securities related business directly
>> whatsoever. This communication is the property of Strike and its
>> affiliates, and does not constitute an offer to sell or the solicitation of
>> an offer to buy any security in any jurisdiction. It is intended only for
>> the person to whom it is addressed and may contain information that is
>> privileged, confidential, or otherwise protected from disclosure.
>> Distribution or copying of this communication, or the information contained
>> herein, by anyone other than the intended recipient is prohibited. If you
>> have received this communication in error, please immediately notify Strike
>> at info@striketechnologies.com, and delete and destroy any copies hereof.
>> ________________________________
>>
>> CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any attachments
>> are intended solely for the addressee. This transmission is covered by the
>> Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The
>> information contained in this transmission is confidential in nature and
>> protected from further use or disclosure under U.S. Pub. L. 106-102, 113
>> U.S. Stat. 1338 (1999), and may be subject to attorney-client or other
>> legal privilege. Your use or disclosure of this information for any purpose
>> other than that intended by its transmittal is strictly prohibited, and may
>> subject you to fines and/or penalties under federal and state law. If you
>> are not the intended recipient of this transmission, please DESTROY ALL
>> COPIES RECEIVED and confirm destruction to the sender via return
>> transmittal.
>>
>
Re: [DISCUSS] Removing dev dependencies from PyPI constraints ?
Posted by Jarek Potiuk <ja...@potiuk.com>.
> Exactly to remove the dev requirements airflow has from interfering with
our own dev requirements. Though this approach notably doesn't change any
of the used non-dev requirements, it's not clear to me (and I'm probably
just missing something) why your diff includes non-dev requirements.
The thing is that I do not "remove" devel features - because due to
transitive dependencies I would have to go deep down the rabbit hole which
dependencies are "devel" and which not. So what I do, I ask `uv` to do the
job for me and repeat the resolution but with the `--exact` flag (combined
with the already used `--resolution higest` and `--no-sources`).
Apparently in this case `uv` will sometimes decide that it's ok to bump
some dependencies that are left after initial resolution. The thing is that
uv will not install some versions of some packages if they are conflicting
with already installed packages. What I believe `uv` does when I use
the *--exact
*flag of `uv pip install` it "fixes" the initial resolution. It finds out
what is needed, removes those packages that are not needed, and then it
resolves again and improves the resolution (that would be my guess at least
- it could be done slightly differently and in a single step I guess.
Essentially the difference between the left and right side is:
a) install absolutely everything from the workspace
b) do this:
left: uv pip install --no-sources 'apache-airflow[all]'
'apache-airflow-core[all]' apache-airflow-task-sdk ./airflow-ctl
--reinstall --resolution highest --find-links file:///dist
right: uv pip install --no-sources *--exact --strict* 'apache-airflow[all]'
'apache-airflow-core[all]' apache-airflow-task-sdk ./airflow-ctl
--reinstall --resolution highest --find-links file:///dist
In the current state content of the fille://dist should be the same in both
cases:
apache-airflow-providers-edge3 (1.1.2): wheel -
apache_airflow_providers_edge3-1.1.2-py3-none-any.whl
apache-airflow-providers-keycloak (1.0.0): wheel -
apache_airflow_providers_keycloak-1.0.0-py3-none-any.whl
apache-airflow-providers-snowflake (6.5.1): wheel -
apache_airflow_providers_snowflake-6.5.1-py3-none-any.whl
This is a bit of in-process quirk of the "PyPI" resolution - those are the
three packages that we have updated versions of in the sources, but they
are not released yet. But they should be essentially the same in both
cases. So the difference between left and right is `--exact --strict`
(strict I added to make sure that we are not removing something that is
still needed).
J.
On Tue, Jul 22, 2025 at 5:22 PM Damian Shaw <ds...@striketechnologies.com>
wrote:
> As I user of Airflow I have ended up implementing this for my environments:
>
> pip install apache-airflow[all or select-extras] -c {official constraints}
> > smaller-run-time-constraints.txt
> pip-compile {all my requirements including development}.txt -c
> smaller-run-time-constraints.txt
>
> Exactly to remove the dev requirements airflow has from interfering with
> our own dev requirements. Though this approach notably doesn't change any
> of the used non-dev requirements, it's not clear to me (and I'm probably
> just missing something) why your diff includes non-dev requirements.
>
> Damian
>
> -----Original Message-----
> From: Jarek Potiuk <ja...@potiuk.com>
> Sent: Tuesday, July 22, 2025 11:04 AM
> To: dev@airflow.apache.org
> Subject: [DISCUSS] Removing dev dependencies from PyPI constraints ?
>
> Hello here,
>
>
> *Tl;DR; We had an interesting discussion on Slack today and I wanted to
> bring it here to discuss what we should do - whether we should keep devel
> dependencies in our PyPI constraints or not.*
>
> *A bit of context:*
> In our CI we generate several types of constraints. one of them are
> "source" constraints that we use during our CI testing - including all our
> testing tools, mypy, ruff etc., and another type is "PyPI" constraint -
> which is more <What should be the "golden" versions of dependencies when
> you install Airflow using `pip install` for production>.
>
> The main difference between the two (roughly) is that "source constraints"
> only uses dependencies from "branch tip" ("main", "v3-0-test"), where
> "PyPI constraints" uses "Airflow core" branch-tip dependencies - but all
> the providers are installed from PyPI. There are few nuances when some
> packages are being released but basically - for example for Airflow 3.0.3,
> "PyPI"
> constraints is "what is the current set of dependencies and providers when
> we install airflow from PyPI at the moment of release".
>
> So target for "source" constraints are "Airflow contributors" where target
> for "PyPI constraints" are "Airflow users".
>
>
> *The problem*
> In the slack discussion
> https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1753169914722049 -
> slack user @dashmug (sorry - do not know name) noticed that the "PyPI"
> constraints also contain Airflow development dependencies listed (like
> ruff or mypy). That made me think a bit and I quickly came up with PR that
> could "fix" it - i.e. only keep the "real" constraints in.
> https://github.com/apache/airflow/pull/53631 - it's as easy as adding the
> `--exact` (and `--strict` for verification) flag to the `uv pip install`
> command we use.
>
> I agree it's a bit confusing to have the devel dependencies there - it is
> not the "intention", but there is very little problem with it - the fact
> that mypy or ruff are listed with a version in constraints, does not mean
> it must be installed - constraints mainly say "if you install this, it
> should be this version" . And there is no problem whatsoever (and it is
> even recommended by us) to have "airflow with its deps" installed with
> constraints, and then anything else you need to install or update -
> without. So other than potential confusion, it usually does not cause harm.
>
> Now... It might seem that removing devel deps has no side effects and we
> should **just do it** - but it's not that easy. Main problem is that those
> devel dependencies might "hold" other dependencies (directly or
> transitively) from being upgraded - and when we remove the devel
> dependencies and do the resolution again - we might have a DIFFERENT set of
> dependencies, simply because removing the devel dependencies, might "free"
> the other deps and they might get upgraded. Yeah - I know it's not obvious
> for someone who did not spend 3 years solving dependency issues :).
>
> And the problem with it is that because we need those devel deps to test
> things - the upgraded dependencies might have never been tested in our CI -
> because we never upgraded to later versions. It's a bit of
> Heisenberg-effect. By using test tools we are changing the state of the
> tested thing. Currently our "PyPI" constraints reflect the state of Airflow
> "when the testing tools are installed" - and we are generally not able to
> test the state of Airflow "without the testing tools" - by the sheer fact
> that we do not have those tools :)
>
> Just to illustrate the case - you can see what happens when we removed
> devel deps:
> https://github.com/apache/airflow/actions/runs/16444553234 - in summary
> you can see a colored version that is a bit clearer but I dumped the diff
> below from an example - Python 3.10 case. Left side < is "what was with
> testing tools" and > is without them. You can see a LOT of test tools
> removed (pytest, ruff and others - including dependencies used only by
> those tools)
>
> But also you can see (likely some of the devel/test deps limited those)
>
> * Authlib - upgraded from 1.3.1 to 1.6.1
> * Google - genai upgraded from 1.2.0 to 1.20.0
> * Httpx - upgraded from 0.27.0 to 0.28.1
> * Pinotdb - upgraded from 5.6.0 to 5.7.0
> * Sentry-sdk - upgraded from 2.33.1 to 2.33.2 (this is false positive as
> it has just been released and new version was picked)
> * validators - upgraded form 0.34.0 to 0.35.0
> * weaviate-client - upgraded from 4.9.0 to 4.16.3
> * websockets - upgraded from 14.2 to 15.0.1
>
>
> This means that all the "upgraded" packages above have not been - likely
> ever tested in our CI.
>
> And when we put them in "versioned" constraints, it might mean that they
> will be obviously failing.
>
> So ... there is a risk connected with just generating the "non-devel"
> constraints without additional tests. But that also means that the user
> who does not use constraints might get those versions anyway - default
> behaviour of `pip install airflow` will be to install those upgraded
> versions above if you do not use constraints. But also that is no different
> than installing a released airflow version a few weeks or months after
> release without constraints - there will be tens of packages that were
> released since our release and installing airflow without constraints will
> pick those.
>
> However - when we manually install RC candidates - say by running airflow
> from rc1 image, we will use those "upgraded" dependencies - so some
> "obvious" problems might be caught during the rc manual testing - where you
> do not use test tools but run airflow "as if" it was installed in
> production.
>
> *The solution ? *
>
> There are few ways we can proceed with it:
>
> a) leave it as is - no big harm, potential confusion, by people who see it
> - potential differences
>
> b) remove devel following my PR and take the risk that we never tested (in
> CI) the dependencies we set in the constraints. That's a bit risky, but we
> can always reactively downgrade some of those packages after the fact if
> users will start reporting the issues
>
> c) potentially most complex thing - trying to remove devel dependencies
> somewhat combining the two - for example removing all the deps that would
> be removed in b) from a) - but that has also some edge cases - when
> different versions of dependencies can add or upgrade their dependencies,
> we are very likely to occasionally remove or add too much.
>
>
> I wonder (for those who managed to read that far - likely not many - what
> do you think here? What would be the best option ? Maybe there are other
> options I have not thought about ?
>
>
> J
>
>
> -----
>
> The diff for Python 3.10
>
>
>
> 34c34
> < Authlib==1.3.1
> ---
> > Authlib==1.6.1
> 64d63
> < Sphinx==8.2.3
> 78d76
> < aioresponses==0.7.8
> 82d79
> < alabaster==1.0.0
> 198d194
> < arro3-core==0.5.1
> 202d197
> < astroid==3.3.11
> 211,212d205
> < aws-sam-translator==1.99.0
> < aws-xray-sdk==2.14.0
> 258,259d250
> < cfgv==3.4.0
> < cfn-lint==1.38.0
> 262d252
> < checksumdir==1.2.0
> 277d266
> < coverage==7.9.2
> 291d279
> < deltalake==1.1.0
> 293d280
> < diagrams==0.24.4
> 301,302d287
> < docutils==0.21.2
> < duckdb==1.3.2
> 308d292
> < eralchemy2==1.4.1
> 311d294
> < execnet==2.1.1
> 321,322d303
> < flit==3.12.0
> < flit_core==3.12.0
> 386c367
> < google-genai==1.2.0
> ---
> > google-genai==1.20.0
> 389d369
> < graphql-core==3.2.6
> 398d377
> < grpcio-tools==1.62.3
> 403,404d381
> < hatch==1.14.1
> < hatchling==1.27.0
> 413c390
> < httpx==0.27.0
> ---
> > httpx==0.28.1
> 418d394
> < hyperlink==21.0.0
> 421,423d396
> < icdiff==2.0.7
> < id==1.5.0
> < identify==2.6.12
> 426d398
> < imagesize==1.4.1
> 430d401
> < incremental==24.7.2
> 433,435d403
> < iniconfig==2.1.0
> < inputimeout==1.0.4
> < ipdb==0.13.13
> 449d416
> < joserfc==1.2.2
> 451d417
> < jsonpatch==1.33
> 455,456d420
> < jsonpointer==3.0.0
> < jsonschema-path==0.3.4
> 463d426
> < kerberos==1.3.1
> 465,466d427
> < keyrings.alt==5.0.2
> < kgb==7.2
> 494,495d454
> < mmh3==5.1.0
> < mongomock==4.3.0
> 497,498d455
> < moto==5.1.8
> < mpmath==1.3.0
> 508,512d464
> < mypy-boto3-appflow==1.39.0
> < mypy-boto3-rds==1.39.1
> < mypy-boto3-redshift-data==1.39.0
> < mypy-boto3-s3==1.39.5
> < mypy==1.17.0
> 514c466
> < mysql-connector-python==9.3.0
> ---
> > mysql-connector-python==9.4.0
> 521,523d472
> < networkx==3.5
> < nh3==0.3.0
> < nodeenv==1.9.1
> 528,529d476
> < openapi-schema-validator==0.6.3
> < openapi-spec-validator==0.7.2
> 560d506
> < pathable==0.4.4
> 563d508
> < pdbr==0.9.2
> 569,570c514
> < pinotdb==5.6.0
> < pipdeptree==2.28.0
> ---
> > pinotdb==5.7.0
> 574d517
> < plyvel==1.5.1
> 577,579d519
> < pprintpp==0.4.0
> < pre-commit-uv==4.1.4
> < pre_commit==4.2.0
> 592d531
> < py-partiql-parser==0.6.1
> 608d546
> < pyenchant==3.2.2
> 610,611d547
> < pygraphviz==1.14
> < pyiceberg==0.9.1
> 622,632d557
> < pytest-asyncio==0.25.0
> < pytest-cov==6.2.1
> < pytest-custom-exit-code==0.3.0
> < pytest-icdiff==0.9
> < pytest-instafail==0.5.0
> < pytest-mock==3.14.1
> < pytest-rerunfailures==15.1
> < pytest-timeouts==1.2.1
> < pytest-unordered==0.7.0
> < pytest-xdist==3.8.0
> < pytest==8.4.1
> 642d566
> < python-on-whales==0.78.0
> 652d575
> < readme_renderer==44.0
> 659d581
> < requests-mock==1.12.1
> 664,665d585
> < responses==0.25.7
> < restructuredtext_lint==1.4.0
> 667,668d586
> < rfc3339-validator==0.1.4
> < rfc3986==2.0.0
> 670d587
> < rich-click==1.8.9
> 674d590
> < roman-numerals-py==3.1.0
> 679d594
> < ruff==0.12.3
> 688d602
> < semver==3.0.4
> 690,691c604
> < sentinels==1.0.0
> < sentry-sdk==2.33.1
> ---
> > sentry-sdk==2.33.2
> 703d615
> < snowballstemmer==3.0.1
> 709,725d620
> < sphinx-argparse==0.5.2
> < sphinx-autoapi==3.6.0
> < sphinx-autobuild==2024.10.3
> < sphinx-copybutton==0.5.2
> < sphinx-jinja==2.0.2
> < sphinx-rtd-theme==3.0.2
> < sphinx_design==0.6.1
> < sphinxcontrib-applehelp==2.0.0
> < sphinxcontrib-devhelp==2.0.0
> < sphinxcontrib-htmlhelp==2.1.0
> < sphinxcontrib-httpdomain==1.8.1
> < sphinxcontrib-jquery==4.1
> < sphinxcontrib-jsmath==1.0.1
> < sphinxcontrib-qthelp==2.0.0
> < sphinxcontrib-redoc==1.6.0
> < sphinxcontrib-serializinghtml==2.0.0
> < sphinxcontrib-spelling==8.0.1
> 737d631
> < strictyaml==1.7.3
> 740d633
> < sympy==1.14.0
> 752d644
> < time-machine==2.16.0
> 755d646
> < tomli_w==1.2.0
> 758d648
> < towncrier==24.8.0
> 762,763d651
> < trove-classifiers==2025.5.9.12
> < twine==6.1.0
> 765,774d652
> < types-Deprecated==1.2.15.20250304
> < types-Markdown==3.8.0.20250708
> < types-PyMySQL==1.1.0.20250711
> < types-PyYAML==6.0.12.20250516
> < types-aiofiles==24.1.0.20250708
> < types-certifi==2021.10.8.3
> < types-cffi==1.17.0.20250523
> < types-croniter==6.0.0.20250626
> < types-docutils==0.21.0.20250722
> < types-paramiko==3.5.0.20250708
> 776,778d653
> < types-pyOpenSSL==24.1.0.20240722
> < types-python-dateutil==2.9.0.20250708
> < types-python-slugify==8.0.2.20240310
> 780d654
> < types-redis==4.6.0.20241004
> 782,784d655
> < types-setuptools==80.9.0.20250529
> < types-tabulate==0.9.0.20241207
> < types-toml==0.10.8.20240310
> 794d664
> < userpath==1.9.2
> 799c669
> < validators==0.34.0
> ---
> > validators==0.35.0
> 806c676
> < weaviate-client==4.9.6
> ---
> > weaviate-client==4.16.3
> 809c679
> < websockets==14.2
> ---
> > websockets==15.0.1
> 815d684
> < yamllint==1.37.1
> ________________________________
> Strike Technologies, LLC (“Strike”) is part of the GTS family of
> companies. Strike is a technology solutions provider, and is not a broker
> or dealer and does not transact any securities related business directly
> whatsoever. This communication is the property of Strike and its
> affiliates, and does not constitute an offer to sell or the solicitation of
> an offer to buy any security in any jurisdiction. It is intended only for
> the person to whom it is addressed and may contain information that is
> privileged, confidential, or otherwise protected from disclosure.
> Distribution or copying of this communication, or the information contained
> herein, by anyone other than the intended recipient is prohibited. If you
> have received this communication in error, please immediately notify Strike
> at info@striketechnologies.com, and delete and destroy any copies hereof.
> ________________________________
>
> CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any attachments
> are intended solely for the addressee. This transmission is covered by the
> Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The
> information contained in this transmission is confidential in nature and
> protected from further use or disclosure under U.S. Pub. L. 106-102, 113
> U.S. Stat. 1338 (1999), and may be subject to attorney-client or other
> legal privilege. Your use or disclosure of this information for any purpose
> other than that intended by its transmittal is strictly prohibited, and may
> subject you to fines and/or penalties under federal and state law. If you
> are not the intended recipient of this transmission, please DESTROY ALL
> COPIES RECEIVED and confirm destruction to the sender via return
> transmittal.
>
RE: [DISCUSS] Removing dev dependencies from PyPI constraints ?
Posted by Damian Shaw <ds...@striketechnologies.com>.
As I user of Airflow I have ended up implementing this for my environments:
pip install apache-airflow[all or select-extras] -c {official constraints} > smaller-run-time-constraints.txt
pip-compile {all my requirements including development}.txt -c smaller-run-time-constraints.txt
Exactly to remove the dev requirements airflow has from interfering with our own dev requirements. Though this approach notably doesn't change any of the used non-dev requirements, it's not clear to me (and I'm probably just missing something) why your diff includes non-dev requirements.
Damian
-----Original Message-----
From: Jarek Potiuk <ja...@potiuk.com>
Sent: Tuesday, July 22, 2025 11:04 AM
To: dev@airflow.apache.org
Subject: [DISCUSS] Removing dev dependencies from PyPI constraints ?
Hello here,
*Tl;DR; We had an interesting discussion on Slack today and I wanted to bring it here to discuss what we should do - whether we should keep devel dependencies in our PyPI constraints or not.*
*A bit of context:*
In our CI we generate several types of constraints. one of them are "source" constraints that we use during our CI testing - including all our testing tools, mypy, ruff etc., and another type is "PyPI" constraint - which is more <What should be the "golden" versions of dependencies when you install Airflow using `pip install` for production>.
The main difference between the two (roughly) is that "source constraints"
only uses dependencies from "branch tip" ("main", "v3-0-test"), where "PyPI constraints" uses "Airflow core" branch-tip dependencies - but all the providers are installed from PyPI. There are few nuances when some packages are being released but basically - for example for Airflow 3.0.3, "PyPI"
constraints is "what is the current set of dependencies and providers when we install airflow from PyPI at the moment of release".
So target for "source" constraints are "Airflow contributors" where target for "PyPI constraints" are "Airflow users".
*The problem*
In the slack discussion
https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1753169914722049 - slack user @dashmug (sorry - do not know name) noticed that the "PyPI"
constraints also contain Airflow development dependencies listed (like ruff or mypy). That made me think a bit and I quickly came up with PR that could "fix" it - i.e. only keep the "real" constraints in.
https://github.com/apache/airflow/pull/53631 - it's as easy as adding the `--exact` (and `--strict` for verification) flag to the `uv pip install` command we use.
I agree it's a bit confusing to have the devel dependencies there - it is not the "intention", but there is very little problem with it - the fact that mypy or ruff are listed with a version in constraints, does not mean it must be installed - constraints mainly say "if you install this, it should be this version" . And there is no problem whatsoever (and it is even recommended by us) to have "airflow with its deps" installed with constraints, and then anything else you need to install or update - without. So other than potential confusion, it usually does not cause harm.
Now... It might seem that removing devel deps has no side effects and we should **just do it** - but it's not that easy. Main problem is that those devel dependencies might "hold" other dependencies (directly or
transitively) from being upgraded - and when we remove the devel dependencies and do the resolution again - we might have a DIFFERENT set of dependencies, simply because removing the devel dependencies, might "free"
the other deps and they might get upgraded. Yeah - I know it's not obvious for someone who did not spend 3 years solving dependency issues :).
And the problem with it is that because we need those devel deps to test things - the upgraded dependencies might have never been tested in our CI - because we never upgraded to later versions. It's a bit of Heisenberg-effect. By using test tools we are changing the state of the tested thing. Currently our "PyPI" constraints reflect the state of Airflow "when the testing tools are installed" - and we are generally not able to test the state of Airflow "without the testing tools" - by the sheer fact that we do not have those tools :)
Just to illustrate the case - you can see what happens when we removed devel deps:
https://github.com/apache/airflow/actions/runs/16444553234 - in summary you can see a colored version that is a bit clearer but I dumped the diff below from an example - Python 3.10 case. Left side < is "what was with testing tools" and > is without them. You can see a LOT of test tools removed (pytest, ruff and others - including dependencies used only by those tools)
But also you can see (likely some of the devel/test deps limited those)
* Authlib - upgraded from 1.3.1 to 1.6.1
* Google - genai upgraded from 1.2.0 to 1.20.0
* Httpx - upgraded from 0.27.0 to 0.28.1
* Pinotdb - upgraded from 5.6.0 to 5.7.0
* Sentry-sdk - upgraded from 2.33.1 to 2.33.2 (this is false positive as it has just been released and new version was picked)
* validators - upgraded form 0.34.0 to 0.35.0
* weaviate-client - upgraded from 4.9.0 to 4.16.3
* websockets - upgraded from 14.2 to 15.0.1
This means that all the "upgraded" packages above have not been - likely ever tested in our CI.
And when we put them in "versioned" constraints, it might mean that they will be obviously failing.
So ... there is a risk connected with just generating the "non-devel"
constraints without additional tests. But that also means that the user who does not use constraints might get those versions anyway - default behaviour of `pip install airflow` will be to install those upgraded versions above if you do not use constraints. But also that is no different than installing a released airflow version a few weeks or months after release without constraints - there will be tens of packages that were released since our release and installing airflow without constraints will pick those.
However - when we manually install RC candidates - say by running airflow from rc1 image, we will use those "upgraded" dependencies - so some "obvious" problems might be caught during the rc manual testing - where you do not use test tools but run airflow "as if" it was installed in production.
*The solution ? *
There are few ways we can proceed with it:
a) leave it as is - no big harm, potential confusion, by people who see it
- potential differences
b) remove devel following my PR and take the risk that we never tested (in
CI) the dependencies we set in the constraints. That's a bit risky, but we can always reactively downgrade some of those packages after the fact if users will start reporting the issues
c) potentially most complex thing - trying to remove devel dependencies somewhat combining the two - for example removing all the deps that would be removed in b) from a) - but that has also some edge cases - when different versions of dependencies can add or upgrade their dependencies, we are very likely to occasionally remove or add too much.
I wonder (for those who managed to read that far - likely not many - what do you think here? What would be the best option ? Maybe there are other options I have not thought about ?
J
-----
The diff for Python 3.10
34c34
< Authlib==1.3.1
---
> Authlib==1.6.1
64d63
< Sphinx==8.2.3
78d76
< aioresponses==0.7.8
82d79
< alabaster==1.0.0
198d194
< arro3-core==0.5.1
202d197
< astroid==3.3.11
211,212d205
< aws-sam-translator==1.99.0
< aws-xray-sdk==2.14.0
258,259d250
< cfgv==3.4.0
< cfn-lint==1.38.0
262d252
< checksumdir==1.2.0
277d266
< coverage==7.9.2
291d279
< deltalake==1.1.0
293d280
< diagrams==0.24.4
301,302d287
< docutils==0.21.2
< duckdb==1.3.2
308d292
< eralchemy2==1.4.1
311d294
< execnet==2.1.1
321,322d303
< flit==3.12.0
< flit_core==3.12.0
386c367
< google-genai==1.2.0
---
> google-genai==1.20.0
389d369
< graphql-core==3.2.6
398d377
< grpcio-tools==1.62.3
403,404d381
< hatch==1.14.1
< hatchling==1.27.0
413c390
< httpx==0.27.0
---
> httpx==0.28.1
418d394
< hyperlink==21.0.0
421,423d396
< icdiff==2.0.7
< id==1.5.0
< identify==2.6.12
426d398
< imagesize==1.4.1
430d401
< incremental==24.7.2
433,435d403
< iniconfig==2.1.0
< inputimeout==1.0.4
< ipdb==0.13.13
449d416
< joserfc==1.2.2
451d417
< jsonpatch==1.33
455,456d420
< jsonpointer==3.0.0
< jsonschema-path==0.3.4
463d426
< kerberos==1.3.1
465,466d427
< keyrings.alt==5.0.2
< kgb==7.2
494,495d454
< mmh3==5.1.0
< mongomock==4.3.0
497,498d455
< moto==5.1.8
< mpmath==1.3.0
508,512d464
< mypy-boto3-appflow==1.39.0
< mypy-boto3-rds==1.39.1
< mypy-boto3-redshift-data==1.39.0
< mypy-boto3-s3==1.39.5
< mypy==1.17.0
514c466
< mysql-connector-python==9.3.0
---
> mysql-connector-python==9.4.0
521,523d472
< networkx==3.5
< nh3==0.3.0
< nodeenv==1.9.1
528,529d476
< openapi-schema-validator==0.6.3
< openapi-spec-validator==0.7.2
560d506
< pathable==0.4.4
563d508
< pdbr==0.9.2
569,570c514
< pinotdb==5.6.0
< pipdeptree==2.28.0
---
> pinotdb==5.7.0
574d517
< plyvel==1.5.1
577,579d519
< pprintpp==0.4.0
< pre-commit-uv==4.1.4
< pre_commit==4.2.0
592d531
< py-partiql-parser==0.6.1
608d546
< pyenchant==3.2.2
610,611d547
< pygraphviz==1.14
< pyiceberg==0.9.1
622,632d557
< pytest-asyncio==0.25.0
< pytest-cov==6.2.1
< pytest-custom-exit-code==0.3.0
< pytest-icdiff==0.9
< pytest-instafail==0.5.0
< pytest-mock==3.14.1
< pytest-rerunfailures==15.1
< pytest-timeouts==1.2.1
< pytest-unordered==0.7.0
< pytest-xdist==3.8.0
< pytest==8.4.1
642d566
< python-on-whales==0.78.0
652d575
< readme_renderer==44.0
659d581
< requests-mock==1.12.1
664,665d585
< responses==0.25.7
< restructuredtext_lint==1.4.0
667,668d586
< rfc3339-validator==0.1.4
< rfc3986==2.0.0
670d587
< rich-click==1.8.9
674d590
< roman-numerals-py==3.1.0
679d594
< ruff==0.12.3
688d602
< semver==3.0.4
690,691c604
< sentinels==1.0.0
< sentry-sdk==2.33.1
---
> sentry-sdk==2.33.2
703d615
< snowballstemmer==3.0.1
709,725d620
< sphinx-argparse==0.5.2
< sphinx-autoapi==3.6.0
< sphinx-autobuild==2024.10.3
< sphinx-copybutton==0.5.2
< sphinx-jinja==2.0.2
< sphinx-rtd-theme==3.0.2
< sphinx_design==0.6.1
< sphinxcontrib-applehelp==2.0.0
< sphinxcontrib-devhelp==2.0.0
< sphinxcontrib-htmlhelp==2.1.0
< sphinxcontrib-httpdomain==1.8.1
< sphinxcontrib-jquery==4.1
< sphinxcontrib-jsmath==1.0.1
< sphinxcontrib-qthelp==2.0.0
< sphinxcontrib-redoc==1.6.0
< sphinxcontrib-serializinghtml==2.0.0
< sphinxcontrib-spelling==8.0.1
737d631
< strictyaml==1.7.3
740d633
< sympy==1.14.0
752d644
< time-machine==2.16.0
755d646
< tomli_w==1.2.0
758d648
< towncrier==24.8.0
762,763d651
< trove-classifiers==2025.5.9.12
< twine==6.1.0
765,774d652
< types-Deprecated==1.2.15.20250304
< types-Markdown==3.8.0.20250708
< types-PyMySQL==1.1.0.20250711
< types-PyYAML==6.0.12.20250516
< types-aiofiles==24.1.0.20250708
< types-certifi==2021.10.8.3
< types-cffi==1.17.0.20250523
< types-croniter==6.0.0.20250626
< types-docutils==0.21.0.20250722
< types-paramiko==3.5.0.20250708
776,778d653
< types-pyOpenSSL==24.1.0.20240722
< types-python-dateutil==2.9.0.20250708
< types-python-slugify==8.0.2.20240310
780d654
< types-redis==4.6.0.20241004
782,784d655
< types-setuptools==80.9.0.20250529
< types-tabulate==0.9.0.20241207
< types-toml==0.10.8.20240310
794d664
< userpath==1.9.2
799c669
< validators==0.34.0
---
> validators==0.35.0
806c676
< weaviate-client==4.9.6
---
> weaviate-client==4.16.3
809c679
< websockets==14.2
---
> websockets==15.0.1
815d684
< yamllint==1.37.1
________________________________
Strike Technologies, LLC (“Strike”) is part of the GTS family of companies. Strike is a technology solutions provider, and is not a broker or dealer and does not transact any securities related business directly whatsoever. This communication is the property of Strike and its affiliates, and does not constitute an offer to sell or the solicitation of an offer to buy any security in any jurisdiction. It is intended only for the person to whom it is addressed and may contain information that is privileged, confidential, or otherwise protected from disclosure. Distribution or copying of this communication, or the information contained herein, by anyone other than the intended recipient is prohibited. If you have received this communication in error, please immediately notify Strike at info@striketechnologies.com, and delete and destroy any copies hereof.
________________________________
CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any attachments are intended solely for the addressee. This transmission is covered by the Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The information contained in this transmission is confidential in nature and protected from further use or disclosure under U.S. Pub. L. 106-102, 113 U.S. Stat. 1338 (1999), and may be subject to attorney-client or other legal privilege. Your use or disclosure of this information for any purpose other than that intended by its transmittal is strictly prohibited, and may subject you to fines and/or penalties under federal and state law. If you are not the intended recipient of this transmission, please DESTROY ALL COPIES RECEIVED and confirm destruction to the sender via return transmittal.