You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <Ja...@polidea.com> on 2019/11/16 11:21:57 UTC
Re: [DISCUSS] Back to (some) dependency pinning
After some discussions and some tests by myself and Astronomer docker, I
think I have finally a complete proposal for good, consistent solution for
our dependency management.
It uses well-known standards, does not introduce any new tools (like
poetry) and I think it serves all the use cases we have very well.
I wanted to summarise it here before I create it as an official AIP. I will
need to finalise the approach to complete the work on the official
production image.
Here are some basic assumptions:
1) We keep the current process of setup.py which is used to install Airflow
for development *'pip install -e .[extras]'*. It's nice and modular. It has
open-dependencies (some of them constrained due to compatibilities). This
will break from time to time for fresh install when some transitive
dependencies are changed (or like with yesterday's pymssql drama). No
changes whatsoever for people who continue installing it this way. The same
setup.py is used to install airflow in production (but see point 7 below
for constrained builds).
2) On top of setup.py, we will have a standard *requirements.txt* file as
well. This will be a snapshot of current "working" dependencies (including
all transitive dependencies!). This file might be generated automatically
by pip freeze (with some scripting) and possibly put as part of pre-commit
checks to make sure it is self-maintainable when new dependencies are
added. With the current Breeze + Pre-commit setup we can do it fully
automatically, so that it will happen behind the scenes (and it will be
verified in CI). This requirements.txt file will be a set of "KNOWN TO BE
GOOD" requirements. This requirements.txt file will have ONLY
'dependency==VERSION' - specific versions of each requirement . Nice
side-effect of the requirements.txt file is that if you use IDEs like
Pycharm, VSCode, it will automatically suggest installing missing/outdated
requirements in your virtualenv so you do not have to remember about it.
3) By having *requirements.txt *we can add a dependabot
<https://dependabot.com/> that will upgrade dependencies in this file
automatically (it works in a way that it will create a PR every time one of
the dependencies is released). It works nicely with some of our other
projects - see for example Oozie-2-Airflow here
<https://github.com/GoogleCloudPlatform/oozie-to-airflow/commits/master> .
It has the benefit that it prepares super-nice, isolated commits/PRs. Those
PRs (example here
<https://github.com/GoogleCloudPlatform/oozie-to-airflow/pull/438>) have
all the information about changes and commits. Those are nice and
actionable in terms of checking if we have breaking changes before merging
it (example below). The nice thing about dependabot is that it will create
a PR and our CI will test it automatically. So committers will have to just
double-check, take a look at the release notes and will be able to merge it
straight away. I will have to check if dependabot can be used for Apache
repos though (dependabot is owned by Github). It's an optional step and we
can do it manually though if we cannot use dependabot (but it requires some
more complex scripting).
4) We will use the requirements.txt file in CI in PRs so that we know that
CI always uses a "known to be good" set of requirements. Every PR will have
to have good set of requirements to be merged. No more transitive
dependency changes.
5) We have a special CRON job run in CI daily and this job will use '*pip
install -e . [all]*' (not using the requirements.txt) - this way we will
have a mechanism to detect early problems with transitive dependencies -
without breaking PR builds.
6) We can use the requirements.txt file as constraints file as well (!).
For example you should be able to run '*pip install -e . [gcp]
--constraints requirements.txt*' and it will install the "known to be good"
versions but only for those extras we want.
7) I believe I can customise setup.py to add extra "*constrained*" extra
which will also use the information from the requirements.txt. This way the
users will be able to do `*pip install
apache-airflow==1.10.6[gcp,aws,constrained]*` and have "known to be good"
installation of Airflow - even if some transitive dependencies break the
"clean" install. I am not 100% sure if it is possible exactly this way, but
I think in the worst case it will be `*pip install
apache-airflow=1.10.6[constrained,gcp-constrained,aws-constrained]*` - we
can generate a "-constrained" version of each of the extras automatically
based on the requirements.txt. I already had a POC for that. And then it
should be possible to upgrade each of the dependencies on its own.
8) Last but not least - we will use the requirements.txt file to build the
production image. One of the requirements for a good official image
<https://github.com/docker-library/official-images/blob/master/README.md#repeatability>
is repeatability - which boils down to dependency-pinning. Currently we
have no mechanism to enforce that. With requirements.txt this will be
possible.
Let me know what you think!
J.
Appendix: example dependabot-generated commit:
Bump pylint from 2.4.3 to 2.4.4
Bumps [pylint](https://github.com/PyCQA/pylint) from 2.4.3 to 2.4.4.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/master/ChangeLog)
- [Commits](PyCQA/pylint@pylint-2.4.3...pylint-2.4.4
<https://github.com/PyCQA/pylint/compare/pylint-2.4.3...pylint-2.4.4>)
On Fri, Aug 2, 2019 at 8:34 AM Felix Uellendall <fe...@pm.me.invalid>
wrote:
> I understand what Ash and Jarek are saying. I actually use airflow with
> custom plugins to have a end-product that fully satisfy our needs and when
> writing new hooks and operators I don't want to see "airflow uses
> requirement foo=1 but you have foo=2" - but actually that sometimes also
> just works. So why not let the developer decide if he want to risk breaking
> it?
> I like the idea of firstly defining groups for core and non-core
> dependencies. I think only need to manage the core ones to always work will
> be a lot easier.
> So I would suggest for core ones we have ranges including minor releases
> and non-core ones getting pinned by default installation (and ci).
>
> I think the more dependencies we add (which comes automatically as airflow
> is growing) the more we are willing to set to more specific version ranges
> and maybe even pin them at the end.
>
> Felix
>
> Sent from ProtonMail mobile
>
> -------- Original Message --------
> On Aug 2, 2019, 06:10, Jarek Potiuk wrote:
>
> > Ash is totally right - that's exactly the difficulty we face. Airflow is
> > both a library and end product and this makes the usual advice (pin if
> you
> > are end-product, don't pin if you are library) not really useful. From
> the
> > very beginning of my adventures with Airflow I was for pinning of
> > everything (and using dependabot or similar - I use it for other
> projects),
> > but over time I realised that this is very short-sighted approach. it
> does
> > not take into account the "library" point of view.. Depending which user
> of
> > airflow you are, you have contradicting requirements. If you are user of
> > airflow who just want to use it as "end product" you want pinning. If you
> > want to develop your own operators or extend existing ones - you use
> > airflow as "library" and you do not want pinning.
> >
> > I also proposed at the beginning of that thread that we split core
> > requirements (and pin it) and non-core ones (and don't pin it). But it
> > ain't easy to separate those two sets in a clear way unfortunately.
> >
> > That's why the idea of choosing at the installation time (and not at
> build
> > time) whether you want to install "loose" or "frozen" dependencies is so
> > appealing.
> > Possibly the best solution could be that you 'pip install airflow' and
> you
> > get the pinned versions and then some other way to get the loose one.
> But I
> > think we are a bit on the mercy of pip - this does not seem to be
> possible.
> >
> > Then - it looks like using extras to add "pinning" mechanism is the next
> > best idea.
> >
> > I am not afraid about complexity. We can fully automate generating those
> > pinned requirements. I already have some ideas how we can make sure that
> we
> > keep those requirements in sync while developing and how they can end up
> > frozen in release. I would like to run a POC on that but in short it is
> > another "by-product" of the CI image we have now. Our CI image is the
> > perfect source of frozen requirements - we know those requirements in CI
> > image are ok and we can use them to generate the standard
> > "requirements.txt" file and keep it updated via some local update script
> > (and pre-commit hooks) + we can verify that they are updated in the CI.
> We
> > can then write custom setup.py that will use that requirements.txt and
> the
> > existing "extras" and generate "pinned" extras automatically. That sounds
> > like fully doable and with very limited maintenance effort.
> >
> > J.
> >
> > On Thu, Aug 1, [2019](tel:2019) at 10:45 PM Qingping Hou <qp...@scribd.com>
> wrote:
> >
> >> On Thu, Aug 1, [2019](tel:2019) at 1:33 PM Chen Tong <ci...@gmail.com>
> wrote:
> >> > It is sometimes hard to distinguish if it is a library or an
> application.
> >> > Take operator as an example, a non-tech people may think it as a
> >> well-built
> >> > application while an engineer may consider it as a library and extends
> >> > functionalities on it.
> >> >
> >>
> >> Yeah, I agree. Personally, I would consider operator to be a library
> >> due to the expectation that other people will import them in their own
> >> projects/source tree.
> >>
> >> For things like REST endpoint handlers and perhaps scheduler, it seems
> >> safe to assume all changes and improvements will happen within Airflow
> >> source tree. In that case, it's safe to classify that part of code as
> >> application and freeze all its dependencies. The current plugin system
> >> might make this slightly complicated because people can still extend
> >> the core with custom code. But again, in an ideal world, plugins
> >> should be self-contained and communicating with core through a well
> >> defined interface ;)
> >>
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: [+48 660 796 129](tel:+48660796129) <[+48660796129](tel:+48660796129)>
> > [image: Polidea] <https://www.polidea.com/>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Back to (some) dependency pinning
Posted by Jarek Potiuk <Ja...@polidea.com>.
TL;DR; I think we can close the pinning story (for now at least ):
I just merged yesterday the change that introduced requirements
per-version. Rather than having single requirements.txt in the main
directory, we have now requirements folder and requirements-python3.7.txt
and requirements-python3.6.txt files (they are slightly different) I am
backporting it to 1.10 (there the differences are much bigger 2.7/3.5/3.6)
but it will work the same way.
Just to summarize how it works (it is all described in CONTRIBUTING.rst
<http://CONTRIBUTING.rsthttps://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#airflow-dependencies>
and Breeze commands).
- We have 'breeze generate-requirements' command (and a separate bash
script doing the same) where the requirements can be generated from the
latest images. So whenever someone updates setup.py they will have to
regenerate those (CI build will fail if that's not done and will provide a
helpful hint on how to fix it). You must specify python version when
generating the requirements.
- At the time of generation (locally), the requirements will be bumped
to the latest version matching the setup.py constraints -. after generating
the requirements, our CI tests will test if the updated requirements work
with all the tests.
- We have a CRON job that always bumps requirements to the latest
versions matching setyp.py constraints and will test if they can install.
That will be an early warning in case of breaking changes.
- We have an easy mechanism to install airflow in a consistent way -
described in the documentation including:
pip install apache-airflow[gcp]==1.10.10 \
--constraint
https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.6.txt
For me - that's all we need for now from the pinning point of view. We
might also want to remove current conflicting installs (I will do it while
finishing prod image).
J.
On Sat, Mar 28, 2020 at 8:30 AM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Yep. Second half 2020. We need to cleanup some inconsistencies before we
> move to it :).
>
> J.
>
>
> On Sat, Mar 28, 2020 at 8:28 AM Driesprong, Fokko <fo...@driesprong.frl>
> wrote:
>
>> The resolving will become more strict in the future:
>>
>> http://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html
>>
>> Cheers, Fokko
>>
>> Op wo 25 mrt. 2020 om 15:40 schreef Jarek Potiuk <
>> Jarek.Potiuk@polidea.com>:
>>
>> > OK. I think that one does not need voting then. I will proceed with my
>> PR
>> > for that :).
>> >
>> > J.
>> >
>> > On Wed, Mar 25, 2020 at 3:38 PM Daniel Imberman <
>> daniel.imberman@gmail.com
>> > >
>> > wrote:
>> >
>> > > Agreed. Kind of a “best we can do” considering the current nature of
>> > > python.
>> > > On Mar 24, 2020, 2:45 PM -0700, Driesprong, Fokko
>> <fokko@driesprong.frl
>> > >,
>> > > wrote:
>> > > > Yes, I'd be in favor of not having two packages, and just pinning
>> the
>> > > > versions then. In this case, all the versions will be pinned, so if
>> a
>> > > user
>> > > > wants to install a newer version of elastic, they have to do it
>> > > explicitly.
>> > > > For Java, you have nice packages that will check if you break any
>> > public
>> > > > API, but for Python this is impossible :'(
>> > > >
>> > > > Cheers, Fokko
>> > > >
>> > > >
>> > > > Op di 24 mrt. 2020 om 11:11 schreef Jarek Potiuk <
>> > > Jarek.Potiuk@polidea.com>:
>> > > >
>> > > > > And yet another update: - after seeing how it works I will remove
>> > > > > requirement generation from pre-commit - now that it needs to be
>> > > > > generated separately for different versions of python it's a bit
>> too
>> > > > > much overhead (you'd need to have more images downloaded for
>> > different
>> > > > > python versions for pre-commit). Instaad I will add breeze
>> commands
>> > to
>> > > > > re-ggenerate the requirements (and bash scripts if you do not use
>> > > > > breeze), and anyone changing setup.py will have to do it
>> (otherwise
>> > CI
>> > > > > builds will fail). I think this workflow will be great to keep our
>> > > > > requirements up-to-date and have a stable installation method.
>> > > > >
>> > > > > J.
>> > > > >
>> > > > >
>> > > > > On Mon, Mar 23, 2020 at 5:54 PM Jarek Potiuk <
>> > Jarek.Potiuk@polidea.com
>> > > >
>> > > > > wrote:
>> > > > > >
>> > > > > > Update - It seems that we won't need the -pinned version
>> > eventually.
>> > > I
>> > > > > realized that we need to have slightly different requirements for
>> > > different
>> > > > > python versions.
>> > > > > >
>> > > > > > I just added PR for that:
>> > > https://github.com/apache/airflow/pull/7841
>> > > > > >
>> > > > > > I also found out (during production image exercise) that we can
>> > > install
>> > > > > airflow predictably in a very simple way (once we release the
>> > > requirements
>> > > > > in 1.10.10):
>> > > > > >
>> > > > > > pip install apache-airflow[gcp]==1.10.10 --constraint
>> > > > >
>> > >
>> >
>> https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt
>> > > > > >
>> > > > > > I think this is simple enough to be used as installation
>> method. I
>> > > added
>> > > > > it to the documentation and I think I am ok with dropping -pinned
>> > > package
>> > > > > altogether.
>> > > > > >
>> > > > > > J.
>> > > > > >
>> > > > > >
>> > > > > > On Sun, Mar 22, 2020 at 10:15 AM Jarek Potiuk <
>> > > Jarek.Potiuk@polidea.com>
>> > > > > wrote:
>> > > > > > >
>> > > > > > > Yesterday we had another master breakage - this time from
>> > > elasticsearch
>> > > > > releasing MINOR version 7.6 breaking our builds (not it was MINOR
>> > > version
>> > > > > so should be compatible .... it was not for us). I fixed it
>> quickly
>> > > > > yesterday by limiting it to < 7.6 but for me - this is quite clear
>> > that
>> > > > > trying to rely on SemVer being followed by others is a futile
>> effort
>> > > (at
>> > > > > least in python's world).
>> > > > > > >
>> > > > > > > The theory is nice, but it breaks in practice. And it's not
>> > really
>> > > a
>> > > > > fault of the library maintainers. It's simply sometimes not so
>> easy
>> > to
>> > > see
>> > > > > how your APIs are used - and in Python, you cannot prevent using
>> > stuff
>> > > that
>> > > > > you think is an internal detail. This is what happened in
>> > elasticsearch
>> > > > > case yesterday - apparently, our plugin was using an "internal"
>> API
>> > > > > unknowingly and some parameters from that API were dropped during
>> > > > > refactoring of elasticsearch library.
>> > > > > > >
>> > > > > > > My observation (it's anecdotal though) is that the COVID-19
>> > > situation
>> > > > > made people have more time, fewer distractions, fewer things to
>> do,
>> > > and we
>> > > > > have higher frequency of OSS packages being released recently so
>> we
>> > > should
>> > > > > protect a bit from more often breakages.
>> > > > > > >
>> > > > > > > I think learning from yesterday is:
>> > > > > > >
>> > > > > > > * we should merge the requirements.txt solution quickly to
>> > prevent
>> > > > > further breakages (I am reading and testing it now) - I think
>> > everyone
>> > > > > agrees it's good to have it
>> > > > > > > * I think we can continue discussing whether
>> > apache-airflow-pinned
>> > > > > package should be released or not. I can leave the code building
>> the
>> > > > > package but we can decide about it after some more discussion
>> > > > > > >
>> > > > > > > Does it sound good?
>> > > > > > >
>> > > > > > > J
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > On Fri, Mar 20, 2020 at 2:47 PM Jarek Potiuk <
>> > > Jarek.Potiuk@polidea.com>
>> > > > > wrote:
>> > > > > > > >
>> > > > > > > > And rebased it right now and fixed automated requirements
>> > update.
>> > > > > > > >
>> > > > > > > > On Fri, Mar 20, 2020 at 2:28 PM Jarek Potiuk <
>> > > Jarek.Potiuk@polidea.com>
>> > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > Ah BTW. I just noticed that for some reason I pasted an
>> old
>> > PR
>> > > > > earlier in the thread :(.
>> > > > > > > > > This is the one with requirements.txt I am talking about:
>> > > > > https://github.com/apache/airflow/pull/7730
>> > > > > > > > >
>> > > > > > > > > On Fri, Mar 20, 2020 at 2:26 PM Jarek Potiuk <
>> > > > > Jarek.Potiuk@polidea.com> wrote:
>> > > > > > > > > >
>> > > > > > > > > > Nope. Not blocking. I can work with my branch just
>> > > requirements.txt
>> > > > > is enough for that :)
>> > > > > > > > > >
>> > > > > > > > > > I think the problem with semver is that it is loosely
>> > > followed - we
>> > > > > had a number of breakages in the past with minor version upgrades
>> :(.
>> > > > > > > > > >
>> > > > > > > > > > J.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On Fri, Mar 20, 2020 at 1:27 PM Kaxil Naik <
>> > > kaxilnaik@gmail.com>
>> > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > Thanks for the detailed explanation Jarek.
>> > > > > > > > > > >
>> > > > > > > > > > > How about we have an upper limit for all our
>> > dependencies,
>> > > example
>> > > > > instead
>> > > > > > > > > > > of "google-cloud-storage>=1.16", we have
>> > > > > "google-cloud-storage>=1.16,<2.0" ?
>> > > > > > > > > > >
>> > > > > > > > > > > If a dependency breaks compatibility in minor
>> versions,
>> > we
>> > > can't do
>> > > > > > > > > > > anything about it but if they follow SemVer, we
>> should be
>> > > safe and
>> > > > > the
>> > > > > > > > > > > first-time installers would have a non-breaking
>> package.
>> > > WDYT?
>> > > > > > > > > > >
>> > > > > > > > > > > Btw I hope this is not blocking you in building a
>> > > production image
>> > > > > as I
>> > > > > > > > > > > think requirements.txt is solving that? Please let me
>> > know
>> > > if it is
>> > > > > > > > > > > blocking.
>> > > > > > > > > > >
>> > > > > > > > > > > PS: I am also just dumping my ideas to solve this
>> issue.
>> > > Love to
>> > > > > hear what
>> > > > > > > > > > > others think too.
>> > > > > > > > > > >
>> > > > > > > > > > > Regards,
>> > > > > > > > > > > Kaxil
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > On Thu, Mar 19, 2020 at 2:43 PM Jarek Potiuk <
>> > > > > Jarek.Potiuk@polidea.com>
>> > > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > I think we have similar understanding. But let me
>> just
>> > > clarify
>> > > > > because I
>> > > > > > > > > > > > think we think about we think about solving two
>> > > different problems
>> > > > > > > > > > > > My proposal is not solving all problems with
>> > > dependencies - quite
>> > > > > the
>> > > > > > > > > > > > contrary, I want to solve just one specific
>> > > "repeatability"
>> > > > > problem - read
>> > > > > > > > > > > > on :)..
>> > > > > > > > > > > >
>> > > > > > > > > > > > 1. A potential source of confusion: using "-pinned"
>> for
>> > > > > installation but
>> > > > > > > > > > > > > using "non-pinned" for DAG development.
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > This could be confusing indeed - but they are the
>> same
>> > > in fact -
>> > > > > > > > > > > > just deps might be different over time.
>> > > > > > > > > > > >
>> > > > > > > > > > > > 2. Most of the users would still try to install
>> > > > > "apache-airflow" package
>> > > > > > > > > > > > > that might have been broken for example because
>> of a
>> > > > > dependency
>> > > > > > > > > > > > release,
>> > > > > > > > > > > > > either way, we would still have to suggest them to
>> > use
>> > > > > "pinned"
>> > > > > > > > > > > > version
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > True. I thought we might describe it in the README
>> and
>> > > make it
>> > > > > prominently
>> > > > > > > > > > > > explained. Usually people look at the readme in PyPI
>> > > when they are
>> > > > > > > > > > > > installing
>> > > > > > > > > > > > stuff and it does not work:
>> > > > > https://pypi.org/project/apache-airflow/.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Also - we could of course explain how to use
>> > > requirements.txt
>> > > > > from the
>> > > > > > > > > > > > released
>> > > > > > > > > > > > version when they are installing it. That would be
>> an
>> > > extra
>> > > > > friction point
>> > > > > > > > > > > > though
>> > > > > > > > > > > > and maybe having "always installable" version of
>> > airflow
>> > > is a
>> > > > > better
>> > > > > > > > > > > > choice.
>> > > > > > > > > > > >
>> > > > > > > > > > > > 3. If they install "pinned" version, it is no
>> longer a
>> > > library
>> > > > > again,
>> > > > > > > > > > > > > that is users won't be able to use new NumPy
>> release
>> > or
>> > > > > matplotlib for
>> > > > > > > > > > > > > example. In which case we are just circling back
>> to
>> > > the same
>> > > > > problem,
>> > > > > > > > > > > > > "either we risk broken package" while releasing
>> or we
>> > > risk
>> > > > > potentially
>> > > > > > > > > > > > > incompatible versions.
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > Yep. But maybe it's just a question of naming. Maybe
>> > > even we
>> > > > > could name
>> > > > > > > > > > > > this package differently to indicate that this
>> version
>> > > is a way
>> > > > > to quickly
>> > > > > > > > > > > > install
>> > > > > > > > > > > > airflow but not to do any serious development with
>> it.
>> > > > > > > > > > > >
>> > > > > > > > > > > > So speaking about THE problem I want to solve with
>> the
>> > > > > > > > > > > > requirements.txt and apache-airflow-pinned package:
>> > > > > > > > > > > >
>> > > > > > > > > > > > I really only want to solve "first-time-user"
>> > experience
>> > > here -
>> > > > > nothing
>> > > > > > > > > > > > more. I
>> > > > > > > > > > > > definitely do not want to replace the current
>> > > installation method
>> > > > > for
>> > > > > > > > > > > > experienced
>> > > > > > > > > > > > users - for them using --constraint
>> requirements.txt is
>> > > exactly
>> > > > > what they
>> > > > > > > > > > > > need.
>> > > > > > > > > > > > The only problem I am trying to solve with that is
>> > > > > "repeatability" of
>> > > > > > > > > > > > installation.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Maybe "apache-airflow-quickinstall" or something
>> like
>> > > that would
>> > > > > be better
>> > > > > > > > > > > > than "apache-airflow-pinned" or
>> > > > > "apache-airflow-repeatable-install" or
>> > > > > > > > > > > > something like that. I think about it as a
>> "flavour" of
>> > > ariflow
>> > > > > rather than
>> > > > > > > > > > > > anything else. I even originally implemented it as
>> > > [pinned] extra
>> > > > > where I
>> > > > > > > > > > > > pinned all requirements. Unfortunately I found that
>> if
>> > > you have
>> > > > > > > > > > > > main requirement without limits, adding the same
>> > > requirement as
>> > > > > extra with
>> > > > > > > > > > > > == does not make it pinned :(. That was my original
>> > plan.
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > > Btw I have been on "we should have pinned
>> dependency"
>> > > camp as
>> > > > > Airflow
>> > > > > > > > > > > > > should definitely install without breaking since
>> > day-1
>> > > but I
>> > > > > think a
>> > > > > > > > > > > > > separate "-pinned" package won't solve that issue.
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > Ah yeah we went the same route. I do not think we
>> can
>> > > solve the
>> > > > > > > > > > > > "library vs. app" problem easily. This is a bit of
>> > > > > "eat-and-have-cake"
>> > > > > > > > > > > > at the same time. I know people have problems
>> > > > > > > > > > > > with conflicting dependencies when they are trying
>> to
>> > > install
>> > > > > libraries
>> > > > > > > > > > > > with different requirements. And I am not even
>> trying
>> > to
>> > > solve
>> > > > > that
>> > > > > > > > > > > > problem now. Not even close. This requires some
>> other
>> > > solution
>> > > > > > > > > > > > (for example separate virtualenvs with different
>> > > dependencies
>> > > > > > > > > > > > build from wheels on per-task basis). But that's
>> > > something much
>> > > > > further
>> > > > > > > > > > > > in the future (if at all).
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > WDYT? Also please do let me know if I have
>> > > misunderstood
>> > > > > something
>> > > > > > > > > > > > > (definitely possible :D).
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Regards,
>> > > > > > > > > > > > > Kaxil
>> > > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > --
>> > > > > > > > > >
>> > > > > > > > > > Jarek Potiuk
>> > > > > > > > > > Polidea | Principal Software Engineer
>> > > > > > > > > >
>> > > > > > > > > > M: +48 660 796 129
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > --
>> > > > > > > > >
>> > > > > > > > > Jarek Potiuk
>> > > > > > > > > Polidea | Principal Software Engineer
>> > > > > > > > >
>> > > > > > > > > M: +48 660 796 129
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > --
>> > > > > > > >
>> > > > > > > > Jarek Potiuk
>> > > > > > > > Polidea | Principal Software Engineer
>> > > > > > > >
>> > > > > > > > M: +48 660 796 129
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > >
>> > > > > > > Jarek Potiuk
>> > > > > > > Polidea | Principal Software Engineer
>> > > > > > >
>> > > > > > > M: +48 660 796 129
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > >
>> > > > > > Jarek Potiuk
>> > > > > > Polidea | Principal Software Engineer
>> > > > > >
>> > > > > > M: +48 660 796 129
>> > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > >
>> > > > > Jarek Potiuk
>> > > > > Polidea | Principal Software Engineer
>> > > > >
>> > > > > M: +48 660 796 129
>> > > > >
>> > >
>> >
>> >
>> > --
>> >
>> > Jarek Potiuk
>> > Polidea <https://www.polidea.com/> | Principal Software Engineer
>> >
>> > M: +48 660 796 129 <+48660796129>
>> > [image: Polidea] <https://www.polidea.com/>
>> >
>>
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Back to (some) dependency pinning
Posted by Jarek Potiuk <Ja...@polidea.com>.
Yep. Second half 2020. We need to cleanup some inconsistencies before we
move to it :).
J.
On Sat, Mar 28, 2020 at 8:28 AM Driesprong, Fokko <fo...@driesprong.frl>
wrote:
> The resolving will become more strict in the future:
>
> http://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html
>
> Cheers, Fokko
>
> Op wo 25 mrt. 2020 om 15:40 schreef Jarek Potiuk <Jarek.Potiuk@polidea.com
> >:
>
> > OK. I think that one does not need voting then. I will proceed with my PR
> > for that :).
> >
> > J.
> >
> > On Wed, Mar 25, 2020 at 3:38 PM Daniel Imberman <
> daniel.imberman@gmail.com
> > >
> > wrote:
> >
> > > Agreed. Kind of a “best we can do” considering the current nature of
> > > python.
> > > On Mar 24, 2020, 2:45 PM -0700, Driesprong, Fokko <fokko@driesprong.frl
> > >,
> > > wrote:
> > > > Yes, I'd be in favor of not having two packages, and just pinning the
> > > > versions then. In this case, all the versions will be pinned, so if a
> > > user
> > > > wants to install a newer version of elastic, they have to do it
> > > explicitly.
> > > > For Java, you have nice packages that will check if you break any
> > public
> > > > API, but for Python this is impossible :'(
> > > >
> > > > Cheers, Fokko
> > > >
> > > >
> > > > Op di 24 mrt. 2020 om 11:11 schreef Jarek Potiuk <
> > > Jarek.Potiuk@polidea.com>:
> > > >
> > > > > And yet another update: - after seeing how it works I will remove
> > > > > requirement generation from pre-commit - now that it needs to be
> > > > > generated separately for different versions of python it's a bit
> too
> > > > > much overhead (you'd need to have more images downloaded for
> > different
> > > > > python versions for pre-commit). Instaad I will add breeze commands
> > to
> > > > > re-ggenerate the requirements (and bash scripts if you do not use
> > > > > breeze), and anyone changing setup.py will have to do it (otherwise
> > CI
> > > > > builds will fail). I think this workflow will be great to keep our
> > > > > requirements up-to-date and have a stable installation method.
> > > > >
> > > > > J.
> > > > >
> > > > >
> > > > > On Mon, Mar 23, 2020 at 5:54 PM Jarek Potiuk <
> > Jarek.Potiuk@polidea.com
> > > >
> > > > > wrote:
> > > > > >
> > > > > > Update - It seems that we won't need the -pinned version
> > eventually.
> > > I
> > > > > realized that we need to have slightly different requirements for
> > > different
> > > > > python versions.
> > > > > >
> > > > > > I just added PR for that:
> > > https://github.com/apache/airflow/pull/7841
> > > > > >
> > > > > > I also found out (during production image exercise) that we can
> > > install
> > > > > airflow predictably in a very simple way (once we release the
> > > requirements
> > > > > in 1.10.10):
> > > > > >
> > > > > > pip install apache-airflow[gcp]==1.10.10 --constraint
> > > > >
> > >
> >
> https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt
> > > > > >
> > > > > > I think this is simple enough to be used as installation method.
> I
> > > added
> > > > > it to the documentation and I think I am ok with dropping -pinned
> > > package
> > > > > altogether.
> > > > > >
> > > > > > J.
> > > > > >
> > > > > >
> > > > > > On Sun, Mar 22, 2020 at 10:15 AM Jarek Potiuk <
> > > Jarek.Potiuk@polidea.com>
> > > > > wrote:
> > > > > > >
> > > > > > > Yesterday we had another master breakage - this time from
> > > elasticsearch
> > > > > releasing MINOR version 7.6 breaking our builds (not it was MINOR
> > > version
> > > > > so should be compatible .... it was not for us). I fixed it quickly
> > > > > yesterday by limiting it to < 7.6 but for me - this is quite clear
> > that
> > > > > trying to rely on SemVer being followed by others is a futile
> effort
> > > (at
> > > > > least in python's world).
> > > > > > >
> > > > > > > The theory is nice, but it breaks in practice. And it's not
> > really
> > > a
> > > > > fault of the library maintainers. It's simply sometimes not so easy
> > to
> > > see
> > > > > how your APIs are used - and in Python, you cannot prevent using
> > stuff
> > > that
> > > > > you think is an internal detail. This is what happened in
> > elasticsearch
> > > > > case yesterday - apparently, our plugin was using an "internal" API
> > > > > unknowingly and some parameters from that API were dropped during
> > > > > refactoring of elasticsearch library.
> > > > > > >
> > > > > > > My observation (it's anecdotal though) is that the COVID-19
> > > situation
> > > > > made people have more time, fewer distractions, fewer things to do,
> > > and we
> > > > > have higher frequency of OSS packages being released recently so we
> > > should
> > > > > protect a bit from more often breakages.
> > > > > > >
> > > > > > > I think learning from yesterday is:
> > > > > > >
> > > > > > > * we should merge the requirements.txt solution quickly to
> > prevent
> > > > > further breakages (I am reading and testing it now) - I think
> > everyone
> > > > > agrees it's good to have it
> > > > > > > * I think we can continue discussing whether
> > apache-airflow-pinned
> > > > > package should be released or not. I can leave the code building
> the
> > > > > package but we can decide about it after some more discussion
> > > > > > >
> > > > > > > Does it sound good?
> > > > > > >
> > > > > > > J
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Mar 20, 2020 at 2:47 PM Jarek Potiuk <
> > > Jarek.Potiuk@polidea.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > And rebased it right now and fixed automated requirements
> > update.
> > > > > > > >
> > > > > > > > On Fri, Mar 20, 2020 at 2:28 PM Jarek Potiuk <
> > > Jarek.Potiuk@polidea.com>
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > Ah BTW. I just noticed that for some reason I pasted an old
> > PR
> > > > > earlier in the thread :(.
> > > > > > > > > This is the one with requirements.txt I am talking about:
> > > > > https://github.com/apache/airflow/pull/7730
> > > > > > > > >
> > > > > > > > > On Fri, Mar 20, 2020 at 2:26 PM Jarek Potiuk <
> > > > > Jarek.Potiuk@polidea.com> wrote:
> > > > > > > > > >
> > > > > > > > > > Nope. Not blocking. I can work with my branch just
> > > requirements.txt
> > > > > is enough for that :)
> > > > > > > > > >
> > > > > > > > > > I think the problem with semver is that it is loosely
> > > followed - we
> > > > > had a number of breakages in the past with minor version upgrades
> :(.
> > > > > > > > > >
> > > > > > > > > > J.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Mar 20, 2020 at 1:27 PM Kaxil Naik <
> > > kaxilnaik@gmail.com>
> > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the detailed explanation Jarek.
> > > > > > > > > > >
> > > > > > > > > > > How about we have an upper limit for all our
> > dependencies,
> > > example
> > > > > instead
> > > > > > > > > > > of "google-cloud-storage>=1.16", we have
> > > > > "google-cloud-storage>=1.16,<2.0" ?
> > > > > > > > > > >
> > > > > > > > > > > If a dependency breaks compatibility in minor versions,
> > we
> > > can't do
> > > > > > > > > > > anything about it but if they follow SemVer, we should
> be
> > > safe and
> > > > > the
> > > > > > > > > > > first-time installers would have a non-breaking
> package.
> > > WDYT?
> > > > > > > > > > >
> > > > > > > > > > > Btw I hope this is not blocking you in building a
> > > production image
> > > > > as I
> > > > > > > > > > > think requirements.txt is solving that? Please let me
> > know
> > > if it is
> > > > > > > > > > > blocking.
> > > > > > > > > > >
> > > > > > > > > > > PS: I am also just dumping my ideas to solve this
> issue.
> > > Love to
> > > > > hear what
> > > > > > > > > > > others think too.
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Kaxil
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Mar 19, 2020 at 2:43 PM Jarek Potiuk <
> > > > > Jarek.Potiuk@polidea.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I think we have similar understanding. But let me
> just
> > > clarify
> > > > > because I
> > > > > > > > > > > > think we think about we think about solving two
> > > different problems
> > > > > > > > > > > > My proposal is not solving all problems with
> > > dependencies - quite
> > > > > the
> > > > > > > > > > > > contrary, I want to solve just one specific
> > > "repeatability"
> > > > > problem - read
> > > > > > > > > > > > on :)..
> > > > > > > > > > > >
> > > > > > > > > > > > 1. A potential source of confusion: using "-pinned"
> for
> > > > > installation but
> > > > > > > > > > > > > using "non-pinned" for DAG development.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > This could be confusing indeed - but they are the
> same
> > > in fact -
> > > > > > > > > > > > just deps might be different over time.
> > > > > > > > > > > >
> > > > > > > > > > > > 2. Most of the users would still try to install
> > > > > "apache-airflow" package
> > > > > > > > > > > > > that might have been broken for example because of
> a
> > > > > dependency
> > > > > > > > > > > > release,
> > > > > > > > > > > > > either way, we would still have to suggest them to
> > use
> > > > > "pinned"
> > > > > > > > > > > > version
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > True. I thought we might describe it in the README
> and
> > > make it
> > > > > prominently
> > > > > > > > > > > > explained. Usually people look at the readme in PyPI
> > > when they are
> > > > > > > > > > > > installing
> > > > > > > > > > > > stuff and it does not work:
> > > > > https://pypi.org/project/apache-airflow/.
> > > > > > > > > > > >
> > > > > > > > > > > > Also - we could of course explain how to use
> > > requirements.txt
> > > > > from the
> > > > > > > > > > > > released
> > > > > > > > > > > > version when they are installing it. That would be an
> > > extra
> > > > > friction point
> > > > > > > > > > > > though
> > > > > > > > > > > > and maybe having "always installable" version of
> > airflow
> > > is a
> > > > > better
> > > > > > > > > > > > choice.
> > > > > > > > > > > >
> > > > > > > > > > > > 3. If they install "pinned" version, it is no longer
> a
> > > library
> > > > > again,
> > > > > > > > > > > > > that is users won't be able to use new NumPy
> release
> > or
> > > > > matplotlib for
> > > > > > > > > > > > > example. In which case we are just circling back to
> > > the same
> > > > > problem,
> > > > > > > > > > > > > "either we risk broken package" while releasing or
> we
> > > risk
> > > > > potentially
> > > > > > > > > > > > > incompatible versions.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Yep. But maybe it's just a question of naming. Maybe
> > > even we
> > > > > could name
> > > > > > > > > > > > this package differently to indicate that this
> version
> > > is a way
> > > > > to quickly
> > > > > > > > > > > > install
> > > > > > > > > > > > airflow but not to do any serious development with
> it.
> > > > > > > > > > > >
> > > > > > > > > > > > So speaking about THE problem I want to solve with
> the
> > > > > > > > > > > > requirements.txt and apache-airflow-pinned package:
> > > > > > > > > > > >
> > > > > > > > > > > > I really only want to solve "first-time-user"
> > experience
> > > here -
> > > > > nothing
> > > > > > > > > > > > more. I
> > > > > > > > > > > > definitely do not want to replace the current
> > > installation method
> > > > > for
> > > > > > > > > > > > experienced
> > > > > > > > > > > > users - for them using --constraint requirements.txt
> is
> > > exactly
> > > > > what they
> > > > > > > > > > > > need.
> > > > > > > > > > > > The only problem I am trying to solve with that is
> > > > > "repeatability" of
> > > > > > > > > > > > installation.
> > > > > > > > > > > >
> > > > > > > > > > > > Maybe "apache-airflow-quickinstall" or something like
> > > that would
> > > > > be better
> > > > > > > > > > > > than "apache-airflow-pinned" or
> > > > > "apache-airflow-repeatable-install" or
> > > > > > > > > > > > something like that. I think about it as a "flavour"
> of
> > > ariflow
> > > > > rather than
> > > > > > > > > > > > anything else. I even originally implemented it as
> > > [pinned] extra
> > > > > where I
> > > > > > > > > > > > pinned all requirements. Unfortunately I found that
> if
> > > you have
> > > > > > > > > > > > main requirement without limits, adding the same
> > > requirement as
> > > > > extra with
> > > > > > > > > > > > == does not make it pinned :(. That was my original
> > plan.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > > Btw I have been on "we should have pinned
> dependency"
> > > camp as
> > > > > Airflow
> > > > > > > > > > > > > should definitely install without breaking since
> > day-1
> > > but I
> > > > > think a
> > > > > > > > > > > > > separate "-pinned" package won't solve that issue.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Ah yeah we went the same route. I do not think we can
> > > solve the
> > > > > > > > > > > > "library vs. app" problem easily. This is a bit of
> > > > > "eat-and-have-cake"
> > > > > > > > > > > > at the same time. I know people have problems
> > > > > > > > > > > > with conflicting dependencies when they are trying to
> > > install
> > > > > libraries
> > > > > > > > > > > > with different requirements. And I am not even trying
> > to
> > > solve
> > > > > that
> > > > > > > > > > > > problem now. Not even close. This requires some other
> > > solution
> > > > > > > > > > > > (for example separate virtualenvs with different
> > > dependencies
> > > > > > > > > > > > build from wheels on per-task basis). But that's
> > > something much
> > > > > further
> > > > > > > > > > > > in the future (if at all).
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > WDYT? Also please do let me know if I have
> > > misunderstood
> > > > > something
> > > > > > > > > > > > > (definitely possible :D).
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > Kaxil
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > >
> > > > > > > > > > Jarek Potiuk
> > > > > > > > > > Polidea | Principal Software Engineer
> > > > > > > > > >
> > > > > > > > > > M: +48 660 796 129
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > > Jarek Potiuk
> > > > > > > > > Polidea | Principal Software Engineer
> > > > > > > > >
> > > > > > > > > M: +48 660 796 129
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > Jarek Potiuk
> > > > > > > > Polidea | Principal Software Engineer
> > > > > > > >
> > > > > > > > M: +48 660 796 129
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Jarek Potiuk
> > > > > > > Polidea | Principal Software Engineer
> > > > > > >
> > > > > > > M: +48 660 796 129
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Jarek Potiuk
> > > > > > Polidea | Principal Software Engineer
> > > > > >
> > > > > > M: +48 660 796 129
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Jarek Potiuk
> > > > > Polidea | Principal Software Engineer
> > > > >
> > > > > M: +48 660 796 129
> > > > >
> > >
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
> >
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Back to (some) dependency pinning
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
The resolving will become more strict in the future:
http://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html
Cheers, Fokko
Op wo 25 mrt. 2020 om 15:40 schreef Jarek Potiuk <Ja...@polidea.com>:
> OK. I think that one does not need voting then. I will proceed with my PR
> for that :).
>
> J.
>
> On Wed, Mar 25, 2020 at 3:38 PM Daniel Imberman <daniel.imberman@gmail.com
> >
> wrote:
>
> > Agreed. Kind of a “best we can do” considering the current nature of
> > python.
> > On Mar 24, 2020, 2:45 PM -0700, Driesprong, Fokko <fokko@driesprong.frl
> >,
> > wrote:
> > > Yes, I'd be in favor of not having two packages, and just pinning the
> > > versions then. In this case, all the versions will be pinned, so if a
> > user
> > > wants to install a newer version of elastic, they have to do it
> > explicitly.
> > > For Java, you have nice packages that will check if you break any
> public
> > > API, but for Python this is impossible :'(
> > >
> > > Cheers, Fokko
> > >
> > >
> > > Op di 24 mrt. 2020 om 11:11 schreef Jarek Potiuk <
> > Jarek.Potiuk@polidea.com>:
> > >
> > > > And yet another update: - after seeing how it works I will remove
> > > > requirement generation from pre-commit - now that it needs to be
> > > > generated separately for different versions of python it's a bit too
> > > > much overhead (you'd need to have more images downloaded for
> different
> > > > python versions for pre-commit). Instaad I will add breeze commands
> to
> > > > re-ggenerate the requirements (and bash scripts if you do not use
> > > > breeze), and anyone changing setup.py will have to do it (otherwise
> CI
> > > > builds will fail). I think this workflow will be great to keep our
> > > > requirements up-to-date and have a stable installation method.
> > > >
> > > > J.
> > > >
> > > >
> > > > On Mon, Mar 23, 2020 at 5:54 PM Jarek Potiuk <
> Jarek.Potiuk@polidea.com
> > >
> > > > wrote:
> > > > >
> > > > > Update - It seems that we won't need the -pinned version
> eventually.
> > I
> > > > realized that we need to have slightly different requirements for
> > different
> > > > python versions.
> > > > >
> > > > > I just added PR for that:
> > https://github.com/apache/airflow/pull/7841
> > > > >
> > > > > I also found out (during production image exercise) that we can
> > install
> > > > airflow predictably in a very simple way (once we release the
> > requirements
> > > > in 1.10.10):
> > > > >
> > > > > pip install apache-airflow[gcp]==1.10.10 --constraint
> > > >
> >
> https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt
> > > > >
> > > > > I think this is simple enough to be used as installation method. I
> > added
> > > > it to the documentation and I think I am ok with dropping -pinned
> > package
> > > > altogether.
> > > > >
> > > > > J.
> > > > >
> > > > >
> > > > > On Sun, Mar 22, 2020 at 10:15 AM Jarek Potiuk <
> > Jarek.Potiuk@polidea.com>
> > > > wrote:
> > > > > >
> > > > > > Yesterday we had another master breakage - this time from
> > elasticsearch
> > > > releasing MINOR version 7.6 breaking our builds (not it was MINOR
> > version
> > > > so should be compatible .... it was not for us). I fixed it quickly
> > > > yesterday by limiting it to < 7.6 but for me - this is quite clear
> that
> > > > trying to rely on SemVer being followed by others is a futile effort
> > (at
> > > > least in python's world).
> > > > > >
> > > > > > The theory is nice, but it breaks in practice. And it's not
> really
> > a
> > > > fault of the library maintainers. It's simply sometimes not so easy
> to
> > see
> > > > how your APIs are used - and in Python, you cannot prevent using
> stuff
> > that
> > > > you think is an internal detail. This is what happened in
> elasticsearch
> > > > case yesterday - apparently, our plugin was using an "internal" API
> > > > unknowingly and some parameters from that API were dropped during
> > > > refactoring of elasticsearch library.
> > > > > >
> > > > > > My observation (it's anecdotal though) is that the COVID-19
> > situation
> > > > made people have more time, fewer distractions, fewer things to do,
> > and we
> > > > have higher frequency of OSS packages being released recently so we
> > should
> > > > protect a bit from more often breakages.
> > > > > >
> > > > > > I think learning from yesterday is:
> > > > > >
> > > > > > * we should merge the requirements.txt solution quickly to
> prevent
> > > > further breakages (I am reading and testing it now) - I think
> everyone
> > > > agrees it's good to have it
> > > > > > * I think we can continue discussing whether
> apache-airflow-pinned
> > > > package should be released or not. I can leave the code building the
> > > > package but we can decide about it after some more discussion
> > > > > >
> > > > > > Does it sound good?
> > > > > >
> > > > > > J
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Mar 20, 2020 at 2:47 PM Jarek Potiuk <
> > Jarek.Potiuk@polidea.com>
> > > > wrote:
> > > > > > >
> > > > > > > And rebased it right now and fixed automated requirements
> update.
> > > > > > >
> > > > > > > On Fri, Mar 20, 2020 at 2:28 PM Jarek Potiuk <
> > Jarek.Potiuk@polidea.com>
> > > > wrote:
> > > > > > > >
> > > > > > > > Ah BTW. I just noticed that for some reason I pasted an old
> PR
> > > > earlier in the thread :(.
> > > > > > > > This is the one with requirements.txt I am talking about:
> > > > https://github.com/apache/airflow/pull/7730
> > > > > > > >
> > > > > > > > On Fri, Mar 20, 2020 at 2:26 PM Jarek Potiuk <
> > > > Jarek.Potiuk@polidea.com> wrote:
> > > > > > > > >
> > > > > > > > > Nope. Not blocking. I can work with my branch just
> > requirements.txt
> > > > is enough for that :)
> > > > > > > > >
> > > > > > > > > I think the problem with semver is that it is loosely
> > followed - we
> > > > had a number of breakages in the past with minor version upgrades :(.
> > > > > > > > >
> > > > > > > > > J.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Mar 20, 2020 at 1:27 PM Kaxil Naik <
> > kaxilnaik@gmail.com>
> > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Thanks for the detailed explanation Jarek.
> > > > > > > > > >
> > > > > > > > > > How about we have an upper limit for all our
> dependencies,
> > example
> > > > instead
> > > > > > > > > > of "google-cloud-storage>=1.16", we have
> > > > "google-cloud-storage>=1.16,<2.0" ?
> > > > > > > > > >
> > > > > > > > > > If a dependency breaks compatibility in minor versions,
> we
> > can't do
> > > > > > > > > > anything about it but if they follow SemVer, we should be
> > safe and
> > > > the
> > > > > > > > > > first-time installers would have a non-breaking package.
> > WDYT?
> > > > > > > > > >
> > > > > > > > > > Btw I hope this is not blocking you in building a
> > production image
> > > > as I
> > > > > > > > > > think requirements.txt is solving that? Please let me
> know
> > if it is
> > > > > > > > > > blocking.
> > > > > > > > > >
> > > > > > > > > > PS: I am also just dumping my ideas to solve this issue.
> > Love to
> > > > hear what
> > > > > > > > > > others think too.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > Kaxil
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Mar 19, 2020 at 2:43 PM Jarek Potiuk <
> > > > Jarek.Potiuk@polidea.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I think we have similar understanding. But let me just
> > clarify
> > > > because I
> > > > > > > > > > > think we think about we think about solving two
> > different problems
> > > > > > > > > > > My proposal is not solving all problems with
> > dependencies - quite
> > > > the
> > > > > > > > > > > contrary, I want to solve just one specific
> > "repeatability"
> > > > problem - read
> > > > > > > > > > > on :)..
> > > > > > > > > > >
> > > > > > > > > > > 1. A potential source of confusion: using "-pinned" for
> > > > installation but
> > > > > > > > > > > > using "non-pinned" for DAG development.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > This could be confusing indeed - but they are the same
> > in fact -
> > > > > > > > > > > just deps might be different over time.
> > > > > > > > > > >
> > > > > > > > > > > 2. Most of the users would still try to install
> > > > "apache-airflow" package
> > > > > > > > > > > > that might have been broken for example because of a
> > > > dependency
> > > > > > > > > > > release,
> > > > > > > > > > > > either way, we would still have to suggest them to
> use
> > > > "pinned"
> > > > > > > > > > > version
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > True. I thought we might describe it in the README and
> > make it
> > > > prominently
> > > > > > > > > > > explained. Usually people look at the readme in PyPI
> > when they are
> > > > > > > > > > > installing
> > > > > > > > > > > stuff and it does not work:
> > > > https://pypi.org/project/apache-airflow/.
> > > > > > > > > > >
> > > > > > > > > > > Also - we could of course explain how to use
> > requirements.txt
> > > > from the
> > > > > > > > > > > released
> > > > > > > > > > > version when they are installing it. That would be an
> > extra
> > > > friction point
> > > > > > > > > > > though
> > > > > > > > > > > and maybe having "always installable" version of
> airflow
> > is a
> > > > better
> > > > > > > > > > > choice.
> > > > > > > > > > >
> > > > > > > > > > > 3. If they install "pinned" version, it is no longer a
> > library
> > > > again,
> > > > > > > > > > > > that is users won't be able to use new NumPy release
> or
> > > > matplotlib for
> > > > > > > > > > > > example. In which case we are just circling back to
> > the same
> > > > problem,
> > > > > > > > > > > > "either we risk broken package" while releasing or we
> > risk
> > > > potentially
> > > > > > > > > > > > incompatible versions.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Yep. But maybe it's just a question of naming. Maybe
> > even we
> > > > could name
> > > > > > > > > > > this package differently to indicate that this version
> > is a way
> > > > to quickly
> > > > > > > > > > > install
> > > > > > > > > > > airflow but not to do any serious development with it.
> > > > > > > > > > >
> > > > > > > > > > > So speaking about THE problem I want to solve with the
> > > > > > > > > > > requirements.txt and apache-airflow-pinned package:
> > > > > > > > > > >
> > > > > > > > > > > I really only want to solve "first-time-user"
> experience
> > here -
> > > > nothing
> > > > > > > > > > > more. I
> > > > > > > > > > > definitely do not want to replace the current
> > installation method
> > > > for
> > > > > > > > > > > experienced
> > > > > > > > > > > users - for them using --constraint requirements.txt is
> > exactly
> > > > what they
> > > > > > > > > > > need.
> > > > > > > > > > > The only problem I am trying to solve with that is
> > > > "repeatability" of
> > > > > > > > > > > installation.
> > > > > > > > > > >
> > > > > > > > > > > Maybe "apache-airflow-quickinstall" or something like
> > that would
> > > > be better
> > > > > > > > > > > than "apache-airflow-pinned" or
> > > > "apache-airflow-repeatable-install" or
> > > > > > > > > > > something like that. I think about it as a "flavour" of
> > ariflow
> > > > rather than
> > > > > > > > > > > anything else. I even originally implemented it as
> > [pinned] extra
> > > > where I
> > > > > > > > > > > pinned all requirements. Unfortunately I found that if
> > you have
> > > > > > > > > > > main requirement without limits, adding the same
> > requirement as
> > > > extra with
> > > > > > > > > > > == does not make it pinned :(. That was my original
> plan.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > Btw I have been on "we should have pinned dependency"
> > camp as
> > > > Airflow
> > > > > > > > > > > > should definitely install without breaking since
> day-1
> > but I
> > > > think a
> > > > > > > > > > > > separate "-pinned" package won't solve that issue.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Ah yeah we went the same route. I do not think we can
> > solve the
> > > > > > > > > > > "library vs. app" problem easily. This is a bit of
> > > > "eat-and-have-cake"
> > > > > > > > > > > at the same time. I know people have problems
> > > > > > > > > > > with conflicting dependencies when they are trying to
> > install
> > > > libraries
> > > > > > > > > > > with different requirements. And I am not even trying
> to
> > solve
> > > > that
> > > > > > > > > > > problem now. Not even close. This requires some other
> > solution
> > > > > > > > > > > (for example separate virtualenvs with different
> > dependencies
> > > > > > > > > > > build from wheels on per-task basis). But that's
> > something much
> > > > further
> > > > > > > > > > > in the future (if at all).
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > WDYT? Also please do let me know if I have
> > misunderstood
> > > > something
> > > > > > > > > > > > (definitely possible :D).
> > > > > > > > > > > >
> > > > > > > > > > > > Regards,
> > > > > > > > > > > > Kaxil
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > > Jarek Potiuk
> > > > > > > > > Polidea | Principal Software Engineer
> > > > > > > > >
> > > > > > > > > M: +48 660 796 129
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > Jarek Potiuk
> > > > > > > > Polidea | Principal Software Engineer
> > > > > > > >
> > > > > > > > M: +48 660 796 129
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Jarek Potiuk
> > > > > > > Polidea | Principal Software Engineer
> > > > > > >
> > > > > > > M: +48 660 796 129
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Jarek Potiuk
> > > > > > Polidea | Principal Software Engineer
> > > > > >
> > > > > > M: +48 660 796 129
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Jarek Potiuk
> > > > > Polidea | Principal Software Engineer
> > > > >
> > > > > M: +48 660 796 129
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Jarek Potiuk
> > > > Polidea | Principal Software Engineer
> > > >
> > > > M: +48 660 796 129
> > > >
> >
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
Re: [DISCUSS] Back to (some) dependency pinning
Posted by Jarek Potiuk <Ja...@polidea.com>.
OK. I think that one does not need voting then. I will proceed with my PR
for that :).
J.
On Wed, Mar 25, 2020 at 3:38 PM Daniel Imberman <da...@gmail.com>
wrote:
> Agreed. Kind of a “best we can do” considering the current nature of
> python.
> On Mar 24, 2020, 2:45 PM -0700, Driesprong, Fokko <fo...@driesprong.frl>,
> wrote:
> > Yes, I'd be in favor of not having two packages, and just pinning the
> > versions then. In this case, all the versions will be pinned, so if a
> user
> > wants to install a newer version of elastic, they have to do it
> explicitly.
> > For Java, you have nice packages that will check if you break any public
> > API, but for Python this is impossible :'(
> >
> > Cheers, Fokko
> >
> >
> > Op di 24 mrt. 2020 om 11:11 schreef Jarek Potiuk <
> Jarek.Potiuk@polidea.com>:
> >
> > > And yet another update: - after seeing how it works I will remove
> > > requirement generation from pre-commit - now that it needs to be
> > > generated separately for different versions of python it's a bit too
> > > much overhead (you'd need to have more images downloaded for different
> > > python versions for pre-commit). Instaad I will add breeze commands to
> > > re-ggenerate the requirements (and bash scripts if you do not use
> > > breeze), and anyone changing setup.py will have to do it (otherwise CI
> > > builds will fail). I think this workflow will be great to keep our
> > > requirements up-to-date and have a stable installation method.
> > >
> > > J.
> > >
> > >
> > > On Mon, Mar 23, 2020 at 5:54 PM Jarek Potiuk <Jarek.Potiuk@polidea.com
> >
> > > wrote:
> > > >
> > > > Update - It seems that we won't need the -pinned version eventually.
> I
> > > realized that we need to have slightly different requirements for
> different
> > > python versions.
> > > >
> > > > I just added PR for that:
> https://github.com/apache/airflow/pull/7841
> > > >
> > > > I also found out (during production image exercise) that we can
> install
> > > airflow predictably in a very simple way (once we release the
> requirements
> > > in 1.10.10):
> > > >
> > > > pip install apache-airflow[gcp]==1.10.10 --constraint
> > >
> https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt
> > > >
> > > > I think this is simple enough to be used as installation method. I
> added
> > > it to the documentation and I think I am ok with dropping -pinned
> package
> > > altogether.
> > > >
> > > > J.
> > > >
> > > >
> > > > On Sun, Mar 22, 2020 at 10:15 AM Jarek Potiuk <
> Jarek.Potiuk@polidea.com>
> > > wrote:
> > > > >
> > > > > Yesterday we had another master breakage - this time from
> elasticsearch
> > > releasing MINOR version 7.6 breaking our builds (not it was MINOR
> version
> > > so should be compatible .... it was not for us). I fixed it quickly
> > > yesterday by limiting it to < 7.6 but for me - this is quite clear that
> > > trying to rely on SemVer being followed by others is a futile effort
> (at
> > > least in python's world).
> > > > >
> > > > > The theory is nice, but it breaks in practice. And it's not really
> a
> > > fault of the library maintainers. It's simply sometimes not so easy to
> see
> > > how your APIs are used - and in Python, you cannot prevent using stuff
> that
> > > you think is an internal detail. This is what happened in elasticsearch
> > > case yesterday - apparently, our plugin was using an "internal" API
> > > unknowingly and some parameters from that API were dropped during
> > > refactoring of elasticsearch library.
> > > > >
> > > > > My observation (it's anecdotal though) is that the COVID-19
> situation
> > > made people have more time, fewer distractions, fewer things to do,
> and we
> > > have higher frequency of OSS packages being released recently so we
> should
> > > protect a bit from more often breakages.
> > > > >
> > > > > I think learning from yesterday is:
> > > > >
> > > > > * we should merge the requirements.txt solution quickly to prevent
> > > further breakages (I am reading and testing it now) - I think everyone
> > > agrees it's good to have it
> > > > > * I think we can continue discussing whether apache-airflow-pinned
> > > package should be released or not. I can leave the code building the
> > > package but we can decide about it after some more discussion
> > > > >
> > > > > Does it sound good?
> > > > >
> > > > > J
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Mar 20, 2020 at 2:47 PM Jarek Potiuk <
> Jarek.Potiuk@polidea.com>
> > > wrote:
> > > > > >
> > > > > > And rebased it right now and fixed automated requirements update.
> > > > > >
> > > > > > On Fri, Mar 20, 2020 at 2:28 PM Jarek Potiuk <
> Jarek.Potiuk@polidea.com>
> > > wrote:
> > > > > > >
> > > > > > > Ah BTW. I just noticed that for some reason I pasted an old PR
> > > earlier in the thread :(.
> > > > > > > This is the one with requirements.txt I am talking about:
> > > https://github.com/apache/airflow/pull/7730
> > > > > > >
> > > > > > > On Fri, Mar 20, 2020 at 2:26 PM Jarek Potiuk <
> > > Jarek.Potiuk@polidea.com> wrote:
> > > > > > > >
> > > > > > > > Nope. Not blocking. I can work with my branch just
> requirements.txt
> > > is enough for that :)
> > > > > > > >
> > > > > > > > I think the problem with semver is that it is loosely
> followed - we
> > > had a number of breakages in the past with minor version upgrades :(.
> > > > > > > >
> > > > > > > > J.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Mar 20, 2020 at 1:27 PM Kaxil Naik <
> kaxilnaik@gmail.com>
> > > wrote:
> > > > > > > > >
> > > > > > > > > Thanks for the detailed explanation Jarek.
> > > > > > > > >
> > > > > > > > > How about we have an upper limit for all our dependencies,
> example
> > > instead
> > > > > > > > > of "google-cloud-storage>=1.16", we have
> > > "google-cloud-storage>=1.16,<2.0" ?
> > > > > > > > >
> > > > > > > > > If a dependency breaks compatibility in minor versions, we
> can't do
> > > > > > > > > anything about it but if they follow SemVer, we should be
> safe and
> > > the
> > > > > > > > > first-time installers would have a non-breaking package.
> WDYT?
> > > > > > > > >
> > > > > > > > > Btw I hope this is not blocking you in building a
> production image
> > > as I
> > > > > > > > > think requirements.txt is solving that? Please let me know
> if it is
> > > > > > > > > blocking.
> > > > > > > > >
> > > > > > > > > PS: I am also just dumping my ideas to solve this issue.
> Love to
> > > hear what
> > > > > > > > > others think too.
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Kaxil
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Mar 19, 2020 at 2:43 PM Jarek Potiuk <
> > > Jarek.Potiuk@polidea.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I think we have similar understanding. But let me just
> clarify
> > > because I
> > > > > > > > > > think we think about we think about solving two
> different problems
> > > > > > > > > > My proposal is not solving all problems with
> dependencies - quite
> > > the
> > > > > > > > > > contrary, I want to solve just one specific
> "repeatability"
> > > problem - read
> > > > > > > > > > on :)..
> > > > > > > > > >
> > > > > > > > > > 1. A potential source of confusion: using "-pinned" for
> > > installation but
> > > > > > > > > > > using "non-pinned" for DAG development.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > This could be confusing indeed - but they are the same
> in fact -
> > > > > > > > > > just deps might be different over time.
> > > > > > > > > >
> > > > > > > > > > 2. Most of the users would still try to install
> > > "apache-airflow" package
> > > > > > > > > > > that might have been broken for example because of a
> > > dependency
> > > > > > > > > > release,
> > > > > > > > > > > either way, we would still have to suggest them to use
> > > "pinned"
> > > > > > > > > > version
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > True. I thought we might describe it in the README and
> make it
> > > prominently
> > > > > > > > > > explained. Usually people look at the readme in PyPI
> when they are
> > > > > > > > > > installing
> > > > > > > > > > stuff and it does not work:
> > > https://pypi.org/project/apache-airflow/.
> > > > > > > > > >
> > > > > > > > > > Also - we could of course explain how to use
> requirements.txt
> > > from the
> > > > > > > > > > released
> > > > > > > > > > version when they are installing it. That would be an
> extra
> > > friction point
> > > > > > > > > > though
> > > > > > > > > > and maybe having "always installable" version of airflow
> is a
> > > better
> > > > > > > > > > choice.
> > > > > > > > > >
> > > > > > > > > > 3. If they install "pinned" version, it is no longer a
> library
> > > again,
> > > > > > > > > > > that is users won't be able to use new NumPy release or
> > > matplotlib for
> > > > > > > > > > > example. In which case we are just circling back to
> the same
> > > problem,
> > > > > > > > > > > "either we risk broken package" while releasing or we
> risk
> > > potentially
> > > > > > > > > > > incompatible versions.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Yep. But maybe it's just a question of naming. Maybe
> even we
> > > could name
> > > > > > > > > > this package differently to indicate that this version
> is a way
> > > to quickly
> > > > > > > > > > install
> > > > > > > > > > airflow but not to do any serious development with it.
> > > > > > > > > >
> > > > > > > > > > So speaking about THE problem I want to solve with the
> > > > > > > > > > requirements.txt and apache-airflow-pinned package:
> > > > > > > > > >
> > > > > > > > > > I really only want to solve "first-time-user" experience
> here -
> > > nothing
> > > > > > > > > > more. I
> > > > > > > > > > definitely do not want to replace the current
> installation method
> > > for
> > > > > > > > > > experienced
> > > > > > > > > > users - for them using --constraint requirements.txt is
> exactly
> > > what they
> > > > > > > > > > need.
> > > > > > > > > > The only problem I am trying to solve with that is
> > > "repeatability" of
> > > > > > > > > > installation.
> > > > > > > > > >
> > > > > > > > > > Maybe "apache-airflow-quickinstall" or something like
> that would
> > > be better
> > > > > > > > > > than "apache-airflow-pinned" or
> > > "apache-airflow-repeatable-install" or
> > > > > > > > > > something like that. I think about it as a "flavour" of
> ariflow
> > > rather than
> > > > > > > > > > anything else. I even originally implemented it as
> [pinned] extra
> > > where I
> > > > > > > > > > pinned all requirements. Unfortunately I found that if
> you have
> > > > > > > > > > main requirement without limits, adding the same
> requirement as
> > > extra with
> > > > > > > > > > == does not make it pinned :(. That was my original plan.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > Btw I have been on "we should have pinned dependency"
> camp as
> > > Airflow
> > > > > > > > > > > should definitely install without breaking since day-1
> but I
> > > think a
> > > > > > > > > > > separate "-pinned" package won't solve that issue.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Ah yeah we went the same route. I do not think we can
> solve the
> > > > > > > > > > "library vs. app" problem easily. This is a bit of
> > > "eat-and-have-cake"
> > > > > > > > > > at the same time. I know people have problems
> > > > > > > > > > with conflicting dependencies when they are trying to
> install
> > > libraries
> > > > > > > > > > with different requirements. And I am not even trying to
> solve
> > > that
> > > > > > > > > > problem now. Not even close. This requires some other
> solution
> > > > > > > > > > (for example separate virtualenvs with different
> dependencies
> > > > > > > > > > build from wheels on per-task basis). But that's
> something much
> > > further
> > > > > > > > > > in the future (if at all).
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > WDYT? Also please do let me know if I have
> misunderstood
> > > something
> > > > > > > > > > > (definitely possible :D).
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Kaxil
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > Jarek Potiuk
> > > > > > > > Polidea | Principal Software Engineer
> > > > > > > >
> > > > > > > > M: +48 660 796 129
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Jarek Potiuk
> > > > > > > Polidea | Principal Software Engineer
> > > > > > >
> > > > > > > M: +48 660 796 129
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Jarek Potiuk
> > > > > > Polidea | Principal Software Engineer
> > > > > >
> > > > > > M: +48 660 796 129
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Jarek Potiuk
> > > > > Polidea | Principal Software Engineer
> > > > >
> > > > > M: +48 660 796 129
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Jarek Potiuk
> > > > Polidea | Principal Software Engineer
> > > >
> > > > M: +48 660 796 129
> > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Jarek Potiuk
> > > Polidea | Principal Software Engineer
> > >
> > > M: +48 660 796 129
> > >
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Back to (some) dependency pinning
Posted by Daniel Imberman <da...@gmail.com>.
Agreed. Kind of a “best we can do” considering the current nature of python.
On Mar 24, 2020, 2:45 PM -0700, Driesprong, Fokko <fo...@driesprong.frl>, wrote:
> Yes, I'd be in favor of not having two packages, and just pinning the
> versions then. In this case, all the versions will be pinned, so if a user
> wants to install a newer version of elastic, they have to do it explicitly.
> For Java, you have nice packages that will check if you break any public
> API, but for Python this is impossible :'(
>
> Cheers, Fokko
>
>
> Op di 24 mrt. 2020 om 11:11 schreef Jarek Potiuk <Ja...@polidea.com>:
>
> > And yet another update: - after seeing how it works I will remove
> > requirement generation from pre-commit - now that it needs to be
> > generated separately for different versions of python it's a bit too
> > much overhead (you'd need to have more images downloaded for different
> > python versions for pre-commit). Instaad I will add breeze commands to
> > re-ggenerate the requirements (and bash scripts if you do not use
> > breeze), and anyone changing setup.py will have to do it (otherwise CI
> > builds will fail). I think this workflow will be great to keep our
> > requirements up-to-date and have a stable installation method.
> >
> > J.
> >
> >
> > On Mon, Mar 23, 2020 at 5:54 PM Jarek Potiuk <Ja...@polidea.com>
> > wrote:
> > >
> > > Update - It seems that we won't need the -pinned version eventually. I
> > realized that we need to have slightly different requirements for different
> > python versions.
> > >
> > > I just added PR for that: https://github.com/apache/airflow/pull/7841
> > >
> > > I also found out (during production image exercise) that we can install
> > airflow predictably in a very simple way (once we release the requirements
> > in 1.10.10):
> > >
> > > pip install apache-airflow[gcp]==1.10.10 --constraint
> > https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt
> > >
> > > I think this is simple enough to be used as installation method. I added
> > it to the documentation and I think I am ok with dropping -pinned package
> > altogether.
> > >
> > > J.
> > >
> > >
> > > On Sun, Mar 22, 2020 at 10:15 AM Jarek Potiuk <Ja...@polidea.com>
> > wrote:
> > > >
> > > > Yesterday we had another master breakage - this time from elasticsearch
> > releasing MINOR version 7.6 breaking our builds (not it was MINOR version
> > so should be compatible .... it was not for us). I fixed it quickly
> > yesterday by limiting it to < 7.6 but for me - this is quite clear that
> > trying to rely on SemVer being followed by others is a futile effort (at
> > least in python's world).
> > > >
> > > > The theory is nice, but it breaks in practice. And it's not really a
> > fault of the library maintainers. It's simply sometimes not so easy to see
> > how your APIs are used - and in Python, you cannot prevent using stuff that
> > you think is an internal detail. This is what happened in elasticsearch
> > case yesterday - apparently, our plugin was using an "internal" API
> > unknowingly and some parameters from that API were dropped during
> > refactoring of elasticsearch library.
> > > >
> > > > My observation (it's anecdotal though) is that the COVID-19 situation
> > made people have more time, fewer distractions, fewer things to do, and we
> > have higher frequency of OSS packages being released recently so we should
> > protect a bit from more often breakages.
> > > >
> > > > I think learning from yesterday is:
> > > >
> > > > * we should merge the requirements.txt solution quickly to prevent
> > further breakages (I am reading and testing it now) - I think everyone
> > agrees it's good to have it
> > > > * I think we can continue discussing whether apache-airflow-pinned
> > package should be released or not. I can leave the code building the
> > package but we can decide about it after some more discussion
> > > >
> > > > Does it sound good?
> > > >
> > > > J
> > > >
> > > >
> > > >
> > > > On Fri, Mar 20, 2020 at 2:47 PM Jarek Potiuk <Ja...@polidea.com>
> > wrote:
> > > > >
> > > > > And rebased it right now and fixed automated requirements update.
> > > > >
> > > > > On Fri, Mar 20, 2020 at 2:28 PM Jarek Potiuk <Ja...@polidea.com>
> > wrote:
> > > > > >
> > > > > > Ah BTW. I just noticed that for some reason I pasted an old PR
> > earlier in the thread :(.
> > > > > > This is the one with requirements.txt I am talking about:
> > https://github.com/apache/airflow/pull/7730
> > > > > >
> > > > > > On Fri, Mar 20, 2020 at 2:26 PM Jarek Potiuk <
> > Jarek.Potiuk@polidea.com> wrote:
> > > > > > >
> > > > > > > Nope. Not blocking. I can work with my branch just requirements.txt
> > is enough for that :)
> > > > > > >
> > > > > > > I think the problem with semver is that it is loosely followed - we
> > had a number of breakages in the past with minor version upgrades :(.
> > > > > > >
> > > > > > > J.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Mar 20, 2020 at 1:27 PM Kaxil Naik <ka...@gmail.com>
> > wrote:
> > > > > > > >
> > > > > > > > Thanks for the detailed explanation Jarek.
> > > > > > > >
> > > > > > > > How about we have an upper limit for all our dependencies, example
> > instead
> > > > > > > > of "google-cloud-storage>=1.16", we have
> > "google-cloud-storage>=1.16,<2.0" ?
> > > > > > > >
> > > > > > > > If a dependency breaks compatibility in minor versions, we can't do
> > > > > > > > anything about it but if they follow SemVer, we should be safe and
> > the
> > > > > > > > first-time installers would have a non-breaking package. WDYT?
> > > > > > > >
> > > > > > > > Btw I hope this is not blocking you in building a production image
> > as I
> > > > > > > > think requirements.txt is solving that? Please let me know if it is
> > > > > > > > blocking.
> > > > > > > >
> > > > > > > > PS: I am also just dumping my ideas to solve this issue. Love to
> > hear what
> > > > > > > > others think too.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Kaxil
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Mar 19, 2020 at 2:43 PM Jarek Potiuk <
> > Jarek.Potiuk@polidea.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I think we have similar understanding. But let me just clarify
> > because I
> > > > > > > > > think we think about we think about solving two different problems
> > > > > > > > > My proposal is not solving all problems with dependencies - quite
> > the
> > > > > > > > > contrary, I want to solve just one specific "repeatability"
> > problem - read
> > > > > > > > > on :)..
> > > > > > > > >
> > > > > > > > > 1. A potential source of confusion: using "-pinned" for
> > installation but
> > > > > > > > > > using "non-pinned" for DAG development.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > This could be confusing indeed - but they are the same in fact -
> > > > > > > > > just deps might be different over time.
> > > > > > > > >
> > > > > > > > > 2. Most of the users would still try to install
> > "apache-airflow" package
> > > > > > > > > > that might have been broken for example because of a
> > dependency
> > > > > > > > > release,
> > > > > > > > > > either way, we would still have to suggest them to use
> > "pinned"
> > > > > > > > > version
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > True. I thought we might describe it in the README and make it
> > prominently
> > > > > > > > > explained. Usually people look at the readme in PyPI when they are
> > > > > > > > > installing
> > > > > > > > > stuff and it does not work:
> > https://pypi.org/project/apache-airflow/.
> > > > > > > > >
> > > > > > > > > Also - we could of course explain how to use requirements.txt
> > from the
> > > > > > > > > released
> > > > > > > > > version when they are installing it. That would be an extra
> > friction point
> > > > > > > > > though
> > > > > > > > > and maybe having "always installable" version of airflow is a
> > better
> > > > > > > > > choice.
> > > > > > > > >
> > > > > > > > > 3. If they install "pinned" version, it is no longer a library
> > again,
> > > > > > > > > > that is users won't be able to use new NumPy release or
> > matplotlib for
> > > > > > > > > > example. In which case we are just circling back to the same
> > problem,
> > > > > > > > > > "either we risk broken package" while releasing or we risk
> > potentially
> > > > > > > > > > incompatible versions.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Yep. But maybe it's just a question of naming. Maybe even we
> > could name
> > > > > > > > > this package differently to indicate that this version is a way
> > to quickly
> > > > > > > > > install
> > > > > > > > > airflow but not to do any serious development with it.
> > > > > > > > >
> > > > > > > > > So speaking about THE problem I want to solve with the
> > > > > > > > > requirements.txt and apache-airflow-pinned package:
> > > > > > > > >
> > > > > > > > > I really only want to solve "first-time-user" experience here -
> > nothing
> > > > > > > > > more. I
> > > > > > > > > definitely do not want to replace the current installation method
> > for
> > > > > > > > > experienced
> > > > > > > > > users - for them using --constraint requirements.txt is exactly
> > what they
> > > > > > > > > need.
> > > > > > > > > The only problem I am trying to solve with that is
> > "repeatability" of
> > > > > > > > > installation.
> > > > > > > > >
> > > > > > > > > Maybe "apache-airflow-quickinstall" or something like that would
> > be better
> > > > > > > > > than "apache-airflow-pinned" or
> > "apache-airflow-repeatable-install" or
> > > > > > > > > something like that. I think about it as a "flavour" of ariflow
> > rather than
> > > > > > > > > anything else. I even originally implemented it as [pinned] extra
> > where I
> > > > > > > > > pinned all requirements. Unfortunately I found that if you have
> > > > > > > > > main requirement without limits, adding the same requirement as
> > extra with
> > > > > > > > > == does not make it pinned :(. That was my original plan.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > Btw I have been on "we should have pinned dependency" camp as
> > Airflow
> > > > > > > > > > should definitely install without breaking since day-1 but I
> > think a
> > > > > > > > > > separate "-pinned" package won't solve that issue.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Ah yeah we went the same route. I do not think we can solve the
> > > > > > > > > "library vs. app" problem easily. This is a bit of
> > "eat-and-have-cake"
> > > > > > > > > at the same time. I know people have problems
> > > > > > > > > with conflicting dependencies when they are trying to install
> > libraries
> > > > > > > > > with different requirements. And I am not even trying to solve
> > that
> > > > > > > > > problem now. Not even close. This requires some other solution
> > > > > > > > > (for example separate virtualenvs with different dependencies
> > > > > > > > > build from wheels on per-task basis). But that's something much
> > further
> > > > > > > > > in the future (if at all).
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > WDYT? Also please do let me know if I have misunderstood
> > something
> > > > > > > > > > (definitely possible :D).
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > Kaxil
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Jarek Potiuk
> > > > > > > Polidea | Principal Software Engineer
> > > > > > >
> > > > > > > M: +48 660 796 129
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Jarek Potiuk
> > > > > > Polidea | Principal Software Engineer
> > > > > >
> > > > > > M: +48 660 796 129
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Jarek Potiuk
> > > > > Polidea | Principal Software Engineer
> > > > >
> > > > > M: +48 660 796 129
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Jarek Potiuk
> > > > Polidea | Principal Software Engineer
> > > >
> > > > M: +48 660 796 129
> > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Jarek Potiuk
> > > Polidea | Principal Software Engineer
> > >
> > > M: +48 660 796 129
> > >
> > >
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea | Principal Software Engineer
> >
> > M: +48 660 796 129
> >
Re: [DISCUSS] Back to (some) dependency pinning
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Yes, I'd be in favor of not having two packages, and just pinning the
versions then. In this case, all the versions will be pinned, so if a user
wants to install a newer version of elastic, they have to do it explicitly.
For Java, you have nice packages that will check if you break any public
API, but for Python this is impossible :'(
Cheers, Fokko
Op di 24 mrt. 2020 om 11:11 schreef Jarek Potiuk <Ja...@polidea.com>:
> And yet another update: - after seeing how it works I will remove
> requirement generation from pre-commit - now that it needs to be
> generated separately for different versions of python it's a bit too
> much overhead (you'd need to have more images downloaded for different
> python versions for pre-commit). Instaad I will add breeze commands to
> re-ggenerate the requirements (and bash scripts if you do not use
> breeze), and anyone changing setup.py will have to do it (otherwise CI
> builds will fail). I think this workflow will be great to keep our
> requirements up-to-date and have a stable installation method.
>
> J.
>
>
> On Mon, Mar 23, 2020 at 5:54 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
> >
> > Update - It seems that we won't need the -pinned version eventually. I
> realized that we need to have slightly different requirements for different
> python versions.
> >
> > I just added PR for that: https://github.com/apache/airflow/pull/7841
> >
> > I also found out (during production image exercise) that we can install
> airflow predictably in a very simple way (once we release the requirements
> in 1.10.10):
> >
> > pip install apache-airflow[gcp]==1.10.10 --constraint
> https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt
> >
> > I think this is simple enough to be used as installation method. I added
> it to the documentation and I think I am ok with dropping -pinned package
> altogether.
> >
> > J.
> >
> >
> > On Sun, Mar 22, 2020 at 10:15 AM Jarek Potiuk <Ja...@polidea.com>
> wrote:
> >>
> >> Yesterday we had another master breakage - this time from elasticsearch
> releasing MINOR version 7.6 breaking our builds (not it was MINOR version
> so should be compatible .... it was not for us). I fixed it quickly
> yesterday by limiting it to < 7.6 but for me - this is quite clear that
> trying to rely on SemVer being followed by others is a futile effort (at
> least in python's world).
> >>
> >> The theory is nice, but it breaks in practice. And it's not really a
> fault of the library maintainers. It's simply sometimes not so easy to see
> how your APIs are used - and in Python, you cannot prevent using stuff that
> you think is an internal detail. This is what happened in elasticsearch
> case yesterday - apparently, our plugin was using an "internal" API
> unknowingly and some parameters from that API were dropped during
> refactoring of elasticsearch library.
> >>
> >> My observation (it's anecdotal though) is that the COVID-19 situation
> made people have more time, fewer distractions, fewer things to do, and we
> have higher frequency of OSS packages being released recently so we should
> protect a bit from more often breakages.
> >>
> >> I think learning from yesterday is:
> >>
> >> * we should merge the requirements.txt solution quickly to prevent
> further breakages (I am reading and testing it now) - I think everyone
> agrees it's good to have it
> >> * I think we can continue discussing whether apache-airflow-pinned
> package should be released or not. I can leave the code building the
> package but we can decide about it after some more discussion
> >>
> >> Does it sound good?
> >>
> >> J
> >>
> >>
> >>
> >> On Fri, Mar 20, 2020 at 2:47 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
> >>>
> >>> And rebased it right now and fixed automated requirements update.
> >>>
> >>> On Fri, Mar 20, 2020 at 2:28 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
> >>>>
> >>>> Ah BTW. I just noticed that for some reason I pasted an old PR
> earlier in the thread :(.
> >>>> This is the one with requirements.txt I am talking about:
> https://github.com/apache/airflow/pull/7730
> >>>>
> >>>> On Fri, Mar 20, 2020 at 2:26 PM Jarek Potiuk <
> Jarek.Potiuk@polidea.com> wrote:
> >>>>>
> >>>>> Nope. Not blocking. I can work with my branch just requirements.txt
> is enough for that :)
> >>>>>
> >>>>> I think the problem with semver is that it is loosely followed - we
> had a number of breakages in the past with minor version upgrades :(.
> >>>>>
> >>>>> J.
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Fri, Mar 20, 2020 at 1:27 PM Kaxil Naik <ka...@gmail.com>
> wrote:
> >>>>>>
> >>>>>> Thanks for the detailed explanation Jarek.
> >>>>>>
> >>>>>> How about we have an upper limit for all our dependencies, example
> instead
> >>>>>> of "google-cloud-storage>=1.16", we have
> "google-cloud-storage>=1.16,<2.0" ?
> >>>>>>
> >>>>>> If a dependency breaks compatibility in minor versions, we can't do
> >>>>>> anything about it but if they follow SemVer, we should be safe and
> the
> >>>>>> first-time installers would have a non-breaking package. WDYT?
> >>>>>>
> >>>>>> Btw I hope this is not blocking you in building a production image
> as I
> >>>>>> think requirements.txt is solving that? Please let me know if it is
> >>>>>> blocking.
> >>>>>>
> >>>>>> PS: I am also just dumping my ideas to solve this issue. Love to
> hear what
> >>>>>> others think too.
> >>>>>>
> >>>>>> Regards,
> >>>>>> Kaxil
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Mar 19, 2020 at 2:43 PM Jarek Potiuk <
> Jarek.Potiuk@polidea.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>> > I think we have similar understanding. But let me just clarify
> because I
> >>>>>> > think we think about we think about solving two different problems
> >>>>>> > My proposal is not solving all problems with dependencies - quite
> the
> >>>>>> > contrary, I want to solve just one specific "repeatability"
> problem - read
> >>>>>> > on :)..
> >>>>>> >
> >>>>>> > 1. A potential source of confusion: using "-pinned" for
> installation but
> >>>>>> > > using "non-pinned" for DAG development.
> >>>>>> > >
> >>>>>> >
> >>>>>> > This could be confusing indeed - but they are the same in fact -
> >>>>>> > just deps might be different over time.
> >>>>>> >
> >>>>>> > 2. Most of the users would still try to install
> "apache-airflow" package
> >>>>>> > > that might have been broken for example because of a
> dependency
> >>>>>> > release,
> >>>>>> > > either way, we would still have to suggest them to use
> "pinned"
> >>>>>> > version
> >>>>>> > >
> >>>>>> >
> >>>>>> > True. I thought we might describe it in the README and make it
> prominently
> >>>>>> > explained. Usually people look at the readme in PyPI when they are
> >>>>>> > installing
> >>>>>> > stuff and it does not work:
> https://pypi.org/project/apache-airflow/.
> >>>>>> >
> >>>>>> > Also - we could of course explain how to use requirements.txt
> from the
> >>>>>> > released
> >>>>>> > version when they are installing it. That would be an extra
> friction point
> >>>>>> > though
> >>>>>> > and maybe having "always installable" version of airflow is a
> better
> >>>>>> > choice.
> >>>>>> >
> >>>>>> > 3. If they install "pinned" version, it is no longer a library
> again,
> >>>>>> > > that is users won't be able to use new NumPy release or
> matplotlib for
> >>>>>> > > example. In which case we are just circling back to the same
> problem,
> >>>>>> > > "either we risk broken package" while releasing or we risk
> potentially
> >>>>>> > > incompatible versions.
> >>>>>> > >
> >>>>>> >
> >>>>>> > Yep. But maybe it's just a question of naming. Maybe even we
> could name
> >>>>>> > this package differently to indicate that this version is a way
> to quickly
> >>>>>> > install
> >>>>>> > airflow but not to do any serious development with it.
> >>>>>> >
> >>>>>> > So speaking about THE problem I want to solve with the
> >>>>>> > requirements.txt and apache-airflow-pinned package:
> >>>>>> >
> >>>>>> > I really only want to solve "first-time-user" experience here -
> nothing
> >>>>>> > more. I
> >>>>>> > definitely do not want to replace the current installation method
> for
> >>>>>> > experienced
> >>>>>> > users - for them using --constraint requirements.txt is exactly
> what they
> >>>>>> > need.
> >>>>>> > The only problem I am trying to solve with that is
> "repeatability" of
> >>>>>> > installation.
> >>>>>> >
> >>>>>> > Maybe "apache-airflow-quickinstall" or something like that would
> be better
> >>>>>> > than "apache-airflow-pinned" or
> "apache-airflow-repeatable-install" or
> >>>>>> > something like that. I think about it as a "flavour" of ariflow
> rather than
> >>>>>> > anything else. I even originally implemented it as [pinned] extra
> where I
> >>>>>> > pinned all requirements. Unfortunately I found that if you have
> >>>>>> > main requirement without limits, adding the same requirement as
> extra with
> >>>>>> > == does not make it pinned :(. That was my original plan.
> >>>>>> >
> >>>>>> >
> >>>>>> > > Btw I have been on "we should have pinned dependency" camp as
> Airflow
> >>>>>> > > should definitely install without breaking since day-1 but I
> think a
> >>>>>> > > separate "-pinned" package won't solve that issue.
> >>>>>> > >
> >>>>>> >
> >>>>>> > Ah yeah we went the same route. I do not think we can solve the
> >>>>>> > "library vs. app" problem easily. This is a bit of
> "eat-and-have-cake"
> >>>>>> > at the same time. I know people have problems
> >>>>>> > with conflicting dependencies when they are trying to install
> libraries
> >>>>>> > with different requirements. And I am not even trying to solve
> that
> >>>>>> > problem now. Not even close. This requires some other solution
> >>>>>> > (for example separate virtualenvs with different dependencies
> >>>>>> > build from wheels on per-task basis). But that's something much
> further
> >>>>>> > in the future (if at all).
> >>>>>> >
> >>>>>> >
> >>>>>> > >
> >>>>>> > > WDYT? Also please do let me know if I have misunderstood
> something
> >>>>>> > > (definitely possible :D).
> >>>>>> > >
> >>>>>> > > Regards,
> >>>>>> > > Kaxil
> >>>>>> >
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>>
> >>>>> Jarek Potiuk
> >>>>> Polidea | Principal Software Engineer
> >>>>>
> >>>>> M: +48 660 796 129
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>>
> >>>> Jarek Potiuk
> >>>> Polidea | Principal Software Engineer
> >>>>
> >>>> M: +48 660 796 129
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>>
> >>> Jarek Potiuk
> >>> Polidea | Principal Software Engineer
> >>>
> >>> M: +48 660 796 129
> >>>
> >>>
> >>
> >>
> >> --
> >>
> >> Jarek Potiuk
> >> Polidea | Principal Software Engineer
> >>
> >> M: +48 660 796 129
> >>
> >>
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea | Principal Software Engineer
> >
> > M: +48 660 796 129
> >
> >
>
>
> --
>
> Jarek Potiuk
> Polidea | Principal Software Engineer
>
> M: +48 660 796 129
>
Re: [DISCUSS] Back to (some) dependency pinning
Posted by Jarek Potiuk <Ja...@polidea.com>.
And yet another update: - after seeing how it works I will remove
requirement generation from pre-commit - now that it needs to be
generated separately for different versions of python it's a bit too
much overhead (you'd need to have more images downloaded for different
python versions for pre-commit). Instaad I will add breeze commands to
re-ggenerate the requirements (and bash scripts if you do not use
breeze), and anyone changing setup.py will have to do it (otherwise CI
builds will fail). I think this workflow will be great to keep our
requirements up-to-date and have a stable installation method.
J.
On Mon, Mar 23, 2020 at 5:54 PM Jarek Potiuk <Ja...@polidea.com> wrote:
>
> Update - It seems that we won't need the -pinned version eventually. I realized that we need to have slightly different requirements for different python versions.
>
> I just added PR for that: https://github.com/apache/airflow/pull/7841
>
> I also found out (during production image exercise) that we can install airflow predictably in a very simple way (once we release the requirements in 1.10.10):
>
> pip install apache-airflow[gcp]==1.10.10 --constraint https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt
>
> I think this is simple enough to be used as installation method. I added it to the documentation and I think I am ok with dropping -pinned package altogether.
>
> J.
>
>
> On Sun, Mar 22, 2020 at 10:15 AM Jarek Potiuk <Ja...@polidea.com> wrote:
>>
>> Yesterday we had another master breakage - this time from elasticsearch releasing MINOR version 7.6 breaking our builds (not it was MINOR version so should be compatible .... it was not for us). I fixed it quickly yesterday by limiting it to < 7.6 but for me - this is quite clear that trying to rely on SemVer being followed by others is a futile effort (at least in python's world).
>>
>> The theory is nice, but it breaks in practice. And it's not really a fault of the library maintainers. It's simply sometimes not so easy to see how your APIs are used - and in Python, you cannot prevent using stuff that you think is an internal detail. This is what happened in elasticsearch case yesterday - apparently, our plugin was using an "internal" API unknowingly and some parameters from that API were dropped during refactoring of elasticsearch library.
>>
>> My observation (it's anecdotal though) is that the COVID-19 situation made people have more time, fewer distractions, fewer things to do, and we have higher frequency of OSS packages being released recently so we should protect a bit from more often breakages.
>>
>> I think learning from yesterday is:
>>
>> * we should merge the requirements.txt solution quickly to prevent further breakages (I am reading and testing it now) - I think everyone agrees it's good to have it
>> * I think we can continue discussing whether apache-airflow-pinned package should be released or not. I can leave the code building the package but we can decide about it after some more discussion
>>
>> Does it sound good?
>>
>> J
>>
>>
>>
>> On Fri, Mar 20, 2020 at 2:47 PM Jarek Potiuk <Ja...@polidea.com> wrote:
>>>
>>> And rebased it right now and fixed automated requirements update.
>>>
>>> On Fri, Mar 20, 2020 at 2:28 PM Jarek Potiuk <Ja...@polidea.com> wrote:
>>>>
>>>> Ah BTW. I just noticed that for some reason I pasted an old PR earlier in the thread :(.
>>>> This is the one with requirements.txt I am talking about: https://github.com/apache/airflow/pull/7730
>>>>
>>>> On Fri, Mar 20, 2020 at 2:26 PM Jarek Potiuk <Ja...@polidea.com> wrote:
>>>>>
>>>>> Nope. Not blocking. I can work with my branch just requirements.txt is enough for that :)
>>>>>
>>>>> I think the problem with semver is that it is loosely followed - we had a number of breakages in the past with minor version upgrades :(.
>>>>>
>>>>> J.
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Mar 20, 2020 at 1:27 PM Kaxil Naik <ka...@gmail.com> wrote:
>>>>>>
>>>>>> Thanks for the detailed explanation Jarek.
>>>>>>
>>>>>> How about we have an upper limit for all our dependencies, example instead
>>>>>> of "google-cloud-storage>=1.16", we have "google-cloud-storage>=1.16,<2.0" ?
>>>>>>
>>>>>> If a dependency breaks compatibility in minor versions, we can't do
>>>>>> anything about it but if they follow SemVer, we should be safe and the
>>>>>> first-time installers would have a non-breaking package. WDYT?
>>>>>>
>>>>>> Btw I hope this is not blocking you in building a production image as I
>>>>>> think requirements.txt is solving that? Please let me know if it is
>>>>>> blocking.
>>>>>>
>>>>>> PS: I am also just dumping my ideas to solve this issue. Love to hear what
>>>>>> others think too.
>>>>>>
>>>>>> Regards,
>>>>>> Kaxil
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 19, 2020 at 2:43 PM Jarek Potiuk <Ja...@polidea.com>
>>>>>> wrote:
>>>>>>
>>>>>> > I think we have similar understanding. But let me just clarify because I
>>>>>> > think we think about we think about solving two different problems
>>>>>> > My proposal is not solving all problems with dependencies - quite the
>>>>>> > contrary, I want to solve just one specific "repeatability" problem - read
>>>>>> > on :)..
>>>>>> >
>>>>>> > 1. A potential source of confusion: using "-pinned" for installation but
>>>>>> > > using "non-pinned" for DAG development.
>>>>>> > >
>>>>>> >
>>>>>> > This could be confusing indeed - but they are the same in fact -
>>>>>> > just deps might be different over time.
>>>>>> >
>>>>>> > 2. Most of the users would still try to install "apache-airflow" package
>>>>>> > > that might have been broken for example because of a dependency
>>>>>> > release,
>>>>>> > > either way, we would still have to suggest them to use "pinned"
>>>>>> > version
>>>>>> > >
>>>>>> >
>>>>>> > True. I thought we might describe it in the README and make it prominently
>>>>>> > explained. Usually people look at the readme in PyPI when they are
>>>>>> > installing
>>>>>> > stuff and it does not work: https://pypi.org/project/apache-airflow/.
>>>>>> >
>>>>>> > Also - we could of course explain how to use requirements.txt from the
>>>>>> > released
>>>>>> > version when they are installing it. That would be an extra friction point
>>>>>> > though
>>>>>> > and maybe having "always installable" version of airflow is a better
>>>>>> > choice.
>>>>>> >
>>>>>> > 3. If they install "pinned" version, it is no longer a library again,
>>>>>> > > that is users won't be able to use new NumPy release or matplotlib for
>>>>>> > > example. In which case we are just circling back to the same problem,
>>>>>> > > "either we risk broken package" while releasing or we risk potentially
>>>>>> > > incompatible versions.
>>>>>> > >
>>>>>> >
>>>>>> > Yep. But maybe it's just a question of naming. Maybe even we could name
>>>>>> > this package differently to indicate that this version is a way to quickly
>>>>>> > install
>>>>>> > airflow but not to do any serious development with it.
>>>>>> >
>>>>>> > So speaking about THE problem I want to solve with the
>>>>>> > requirements.txt and apache-airflow-pinned package:
>>>>>> >
>>>>>> > I really only want to solve "first-time-user" experience here - nothing
>>>>>> > more. I
>>>>>> > definitely do not want to replace the current installation method for
>>>>>> > experienced
>>>>>> > users - for them using --constraint requirements.txt is exactly what they
>>>>>> > need.
>>>>>> > The only problem I am trying to solve with that is "repeatability" of
>>>>>> > installation.
>>>>>> >
>>>>>> > Maybe "apache-airflow-quickinstall" or something like that would be better
>>>>>> > than "apache-airflow-pinned" or "apache-airflow-repeatable-install" or
>>>>>> > something like that. I think about it as a "flavour" of ariflow rather than
>>>>>> > anything else. I even originally implemented it as [pinned] extra where I
>>>>>> > pinned all requirements. Unfortunately I found that if you have
>>>>>> > main requirement without limits, adding the same requirement as extra with
>>>>>> > == does not make it pinned :(. That was my original plan.
>>>>>> >
>>>>>> >
>>>>>> > > Btw I have been on "we should have pinned dependency" camp as Airflow
>>>>>> > > should definitely install without breaking since day-1 but I think a
>>>>>> > > separate "-pinned" package won't solve that issue.
>>>>>> > >
>>>>>> >
>>>>>> > Ah yeah we went the same route. I do not think we can solve the
>>>>>> > "library vs. app" problem easily. This is a bit of "eat-and-have-cake"
>>>>>> > at the same time. I know people have problems
>>>>>> > with conflicting dependencies when they are trying to install libraries
>>>>>> > with different requirements. And I am not even trying to solve that
>>>>>> > problem now. Not even close. This requires some other solution
>>>>>> > (for example separate virtualenvs with different dependencies
>>>>>> > build from wheels on per-task basis). But that's something much further
>>>>>> > in the future (if at all).
>>>>>> >
>>>>>> >
>>>>>> > >
>>>>>> > > WDYT? Also please do let me know if I have misunderstood something
>>>>>> > > (definitely possible :D).
>>>>>> > >
>>>>>> > > Regards,
>>>>>> > > Kaxil
>>>>>> >
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Jarek Potiuk
>>>>> Polidea | Principal Software Engineer
>>>>>
>>>>> M: +48 660 796 129
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Jarek Potiuk
>>>> Polidea | Principal Software Engineer
>>>>
>>>> M: +48 660 796 129
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Jarek Potiuk
>>> Polidea | Principal Software Engineer
>>>
>>> M: +48 660 796 129
>>>
>>>
>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea | Principal Software Engineer
>>
>> M: +48 660 796 129
>>
>>
>
>
> --
>
> Jarek Potiuk
> Polidea | Principal Software Engineer
>
> M: +48 660 796 129
>
>
--
Jarek Potiuk
Polidea | Principal Software Engineer
M: +48 660 796 129
Re: [DISCUSS] Back to (some) dependency pinning
Posted by Jarek Potiuk <Ja...@polidea.com>.
Update - It seems that we won't need the -pinned version eventually. I
realized that we need to have slightly different requirements for different
python versions.
I just added PR for that: https://github.com/apache/airflow/pull/7841
I also found out (during production image exercise) that we can install
airflow predictably in a very simple way (once we release the requirements
in 1.10.10):
pip install apache-airflow[gcp]==1.10.10 --constraint
https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt
I think this is simple enough to be used as installation method. I added it
to the documentation and I think I am ok with dropping -pinned package
altogether.
J.
On Sun, Mar 22, 2020 at 10:15 AM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Yesterday we had another master breakage - this time from elasticsearch
> releasing MINOR version 7.6 breaking our builds (not it was MINOR version
> so should be compatible .... it was not for us). I fixed it quickly
> yesterday by limiting it to < 7.6 but for me - this is quite clear that
> trying to rely on SemVer being followed by others is a futile effort (at
> least in python's world).
>
> The theory is nice, but it breaks in practice. And it's not really a fault
> of the library maintainers. It's simply sometimes not so easy to see how
> your APIs are used - and in Python, you cannot prevent using stuff that you
> think is an internal detail. This is what happened in elasticsearch case
> yesterday - apparently, our plugin was using an "internal" API unknowingly
> and some parameters from that API were dropped during refactoring of
> elasticsearch library.
>
> My observation (it's anecdotal though) is that the COVID-19 situation made
> people have more time, fewer distractions, fewer things to do, and we
> have higher frequency of OSS packages being released recently so we
> should protect a bit from more often breakages.
>
> I think learning from yesterday is:
>
> * we should merge the requirements.txt solution quickly to prevent further
> breakages (I am reading and testing it now) - I think everyone agrees it's
> good to have it
> * I think we can continue discussing whether apache-airflow-pinned package
> should be released or not. I can leave the code building the package but we
> can decide about it after some more discussion
>
> Does it sound good?
>
> J
>
>
>
> On Fri, Mar 20, 2020 at 2:47 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> And rebased it right now and fixed automated requirements update.
>>
>> On Fri, Mar 20, 2020 at 2:28 PM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>>> Ah BTW. I just noticed that for some reason I pasted an old PR earlier
>>> in the thread :(.
>>> This is the one with requirements.txt I am talking about:
>>> https://github.com/apache/airflow/pull/7730
>>>
>>> On Fri, Mar 20, 2020 at 2:26 PM Jarek Potiuk <Ja...@polidea.com>
>>> wrote:
>>>
>>>> Nope. Not blocking. I can work with my branch just requirements.txt is
>>>> enough for that :)
>>>>
>>>> I think the problem with semver is that it is loosely followed - we had
>>>> a number of breakages in the past with minor version upgrades :(.
>>>>
>>>> J.
>>>>
>>>>
>>>>
>>>> On Fri, Mar 20, 2020 at 1:27 PM Kaxil Naik <ka...@gmail.com> wrote:
>>>>
>>>>> Thanks for the detailed explanation Jarek.
>>>>>
>>>>> How about we have an upper limit for all our dependencies, example
>>>>> instead
>>>>> of "google-cloud-storage>=1.16", we have
>>>>> "google-cloud-storage>=1.16,<2.0" ?
>>>>>
>>>>> If a dependency breaks compatibility in minor versions, we can't do
>>>>> anything about it but if they follow SemVer, we should be safe and the
>>>>> first-time installers would have a non-breaking package. WDYT?
>>>>>
>>>>> Btw I hope this is not blocking you in building a production image as I
>>>>> think requirements.txt is solving that? Please let me know if it is
>>>>> blocking.
>>>>>
>>>>> PS: I am also just dumping my ideas to solve this issue. Love to hear
>>>>> what
>>>>> others think too.
>>>>>
>>>>> Regards,
>>>>> Kaxil
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Mar 19, 2020 at 2:43 PM Jarek Potiuk <Jarek.Potiuk@polidea.com
>>>>> >
>>>>> wrote:
>>>>>
>>>>> > I think we have similar understanding. But let me just clarify
>>>>> because I
>>>>> > think we think about we think about solving two different problems
>>>>> > My proposal is not solving all problems with dependencies - quite the
>>>>> > contrary, I want to solve just one specific "repeatability" problem
>>>>> - read
>>>>> > on :)..
>>>>> >
>>>>> > 1. A potential source of confusion: using "-pinned" for
>>>>> installation but
>>>>> > > using "non-pinned" for DAG development.
>>>>> > >
>>>>> >
>>>>> > This could be confusing indeed - but they are the same in fact -
>>>>> > just deps might be different over time.
>>>>> >
>>>>> > 2. Most of the users would still try to install "apache-airflow"
>>>>> package
>>>>> > > that might have been broken for example because of a dependency
>>>>> > release,
>>>>> > > either way, we would still have to suggest them to use "pinned"
>>>>> > version
>>>>> > >
>>>>> >
>>>>> > True. I thought we might describe it in the README and make it
>>>>> prominently
>>>>> > explained. Usually people look at the readme in PyPI when they are
>>>>> > installing
>>>>> > stuff and it does not work: https://pypi.org/project/apache-airflow/
>>>>> .
>>>>> >
>>>>> > Also - we could of course explain how to use requirements.txt from
>>>>> the
>>>>> > released
>>>>> > version when they are installing it. That would be an extra friction
>>>>> point
>>>>> > though
>>>>> > and maybe having "always installable" version of airflow is a better
>>>>> > choice.
>>>>> >
>>>>> > 3. If they install "pinned" version, it is no longer a library
>>>>> again,
>>>>> > > that is users won't be able to use new NumPy release or
>>>>> matplotlib for
>>>>> > > example. In which case we are just circling back to the same
>>>>> problem,
>>>>> > > "either we risk broken package" while releasing or we risk
>>>>> potentially
>>>>> > > incompatible versions.
>>>>> > >
>>>>> >
>>>>> > Yep. But maybe it's just a question of naming. Maybe even we could
>>>>> name
>>>>> > this package differently to indicate that this version is a way to
>>>>> quickly
>>>>> > install
>>>>> > airflow but not to do any serious development with it.
>>>>> >
>>>>> > So speaking about THE problem I want to solve with the
>>>>> > requirements.txt and apache-airflow-pinned package:
>>>>> >
>>>>> > I really only want to solve "first-time-user" experience here -
>>>>> nothing
>>>>> > more. I
>>>>> > definitely do not want to replace the current installation method for
>>>>> > experienced
>>>>> > users - for them using --constraint requirements.txt is exactly what
>>>>> they
>>>>> > need.
>>>>> > The only problem I am trying to solve with that is "repeatability" of
>>>>> > installation.
>>>>> >
>>>>> > Maybe "apache-airflow-quickinstall" or something like that would be
>>>>> better
>>>>> > than "apache-airflow-pinned" or "apache-airflow-repeatable-install"
>>>>> or
>>>>> > something like that. I think about it as a "flavour" of ariflow
>>>>> rather than
>>>>> > anything else. I even originally implemented it as [pinned] extra
>>>>> where I
>>>>> > pinned all requirements. Unfortunately I found that if you have
>>>>> > main requirement without limits, adding the same requirement as
>>>>> extra with
>>>>> > == does not make it pinned :(. That was my original plan.
>>>>> >
>>>>> >
>>>>> > > Btw I have been on "we should have pinned dependency" camp as
>>>>> Airflow
>>>>> > > should definitely install without breaking since day-1 but I think
>>>>> a
>>>>> > > separate "-pinned" package won't solve that issue.
>>>>> > >
>>>>> >
>>>>> > Ah yeah we went the same route. I do not think we can solve the
>>>>> > "library vs. app" problem easily. This is a bit of
>>>>> "eat-and-have-cake"
>>>>> > at the same time. I know people have problems
>>>>> > with conflicting dependencies when they are trying to install
>>>>> libraries
>>>>> > with different requirements. And I am not even trying to solve that
>>>>> > problem now. Not even close. This requires some other solution
>>>>> > (for example separate virtualenvs with different dependencies
>>>>> > build from wheels on per-task basis). But that's something much
>>>>> further
>>>>> > in the future (if at all).
>>>>> >
>>>>> >
>>>>> > >
>>>>> > > WDYT? Also please do let me know if I have misunderstood something
>>>>> > > (definitely possible :D).
>>>>> > >
>>>>> > > Regards,
>>>>> > > Kaxil
>>>>> >
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Jarek Potiuk
>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>
>>>> M: +48 660 796 129 <+48660796129>
>>>> [image: Polidea] <https://www.polidea.com/>
>>>>
>>>>
>>>
>>> --
>>>
>>> Jarek Potiuk
>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>
>>> M: +48 660 796 129 <+48660796129>
>>> [image: Polidea] <https://www.polidea.com/>
>>>
>>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Back to (some) dependency pinning
Posted by Jarek Potiuk <Ja...@polidea.com>.
Yesterday we had another master breakage - this time from elasticsearch
releasing MINOR version 7.6 breaking our builds (not it was MINOR version
so should be compatible .... it was not for us). I fixed it quickly
yesterday by limiting it to < 7.6 but for me - this is quite clear that
trying to rely on SemVer being followed by others is a futile effort (at
least in python's world).
The theory is nice, but it breaks in practice. And it's not really a fault
of the library maintainers. It's simply sometimes not so easy to see how
your APIs are used - and in Python, you cannot prevent using stuff that you
think is an internal detail. This is what happened in elasticsearch case
yesterday - apparently, our plugin was using an "internal" API unknowingly
and some parameters from that API were dropped during refactoring of
elasticsearch library.
My observation (it's anecdotal though) is that the COVID-19 situation made
people have more time, fewer distractions, fewer things to do, and we
have higher frequency of OSS packages being released recently so we
should protect a bit from more often breakages.
I think learning from yesterday is:
* we should merge the requirements.txt solution quickly to prevent further
breakages (I am reading and testing it now) - I think everyone agrees it's
good to have it
* I think we can continue discussing whether apache-airflow-pinned package
should be released or not. I can leave the code building the package but we
can decide about it after some more discussion
Does it sound good?
J
On Fri, Mar 20, 2020 at 2:47 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> And rebased it right now and fixed automated requirements update.
>
> On Fri, Mar 20, 2020 at 2:28 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> Ah BTW. I just noticed that for some reason I pasted an old PR earlier in
>> the thread :(.
>> This is the one with requirements.txt I am talking about:
>> https://github.com/apache/airflow/pull/7730
>>
>> On Fri, Mar 20, 2020 at 2:26 PM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>>> Nope. Not blocking. I can work with my branch just requirements.txt is
>>> enough for that :)
>>>
>>> I think the problem with semver is that it is loosely followed - we had
>>> a number of breakages in the past with minor version upgrades :(.
>>>
>>> J.
>>>
>>>
>>>
>>> On Fri, Mar 20, 2020 at 1:27 PM Kaxil Naik <ka...@gmail.com> wrote:
>>>
>>>> Thanks for the detailed explanation Jarek.
>>>>
>>>> How about we have an upper limit for all our dependencies, example
>>>> instead
>>>> of "google-cloud-storage>=1.16", we have
>>>> "google-cloud-storage>=1.16,<2.0" ?
>>>>
>>>> If a dependency breaks compatibility in minor versions, we can't do
>>>> anything about it but if they follow SemVer, we should be safe and the
>>>> first-time installers would have a non-breaking package. WDYT?
>>>>
>>>> Btw I hope this is not blocking you in building a production image as I
>>>> think requirements.txt is solving that? Please let me know if it is
>>>> blocking.
>>>>
>>>> PS: I am also just dumping my ideas to solve this issue. Love to hear
>>>> what
>>>> others think too.
>>>>
>>>> Regards,
>>>> Kaxil
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Mar 19, 2020 at 2:43 PM Jarek Potiuk <Ja...@polidea.com>
>>>> wrote:
>>>>
>>>> > I think we have similar understanding. But let me just clarify
>>>> because I
>>>> > think we think about we think about solving two different problems
>>>> > My proposal is not solving all problems with dependencies - quite the
>>>> > contrary, I want to solve just one specific "repeatability" problem -
>>>> read
>>>> > on :)..
>>>> >
>>>> > 1. A potential source of confusion: using "-pinned" for
>>>> installation but
>>>> > > using "non-pinned" for DAG development.
>>>> > >
>>>> >
>>>> > This could be confusing indeed - but they are the same in fact -
>>>> > just deps might be different over time.
>>>> >
>>>> > 2. Most of the users would still try to install "apache-airflow"
>>>> package
>>>> > > that might have been broken for example because of a dependency
>>>> > release,
>>>> > > either way, we would still have to suggest them to use "pinned"
>>>> > version
>>>> > >
>>>> >
>>>> > True. I thought we might describe it in the README and make it
>>>> prominently
>>>> > explained. Usually people look at the readme in PyPI when they are
>>>> > installing
>>>> > stuff and it does not work: https://pypi.org/project/apache-airflow/.
>>>> >
>>>> > Also - we could of course explain how to use requirements.txt from the
>>>> > released
>>>> > version when they are installing it. That would be an extra friction
>>>> point
>>>> > though
>>>> > and maybe having "always installable" version of airflow is a better
>>>> > choice.
>>>> >
>>>> > 3. If they install "pinned" version, it is no longer a library
>>>> again,
>>>> > > that is users won't be able to use new NumPy release or
>>>> matplotlib for
>>>> > > example. In which case we are just circling back to the same
>>>> problem,
>>>> > > "either we risk broken package" while releasing or we risk
>>>> potentially
>>>> > > incompatible versions.
>>>> > >
>>>> >
>>>> > Yep. But maybe it's just a question of naming. Maybe even we could
>>>> name
>>>> > this package differently to indicate that this version is a way to
>>>> quickly
>>>> > install
>>>> > airflow but not to do any serious development with it.
>>>> >
>>>> > So speaking about THE problem I want to solve with the
>>>> > requirements.txt and apache-airflow-pinned package:
>>>> >
>>>> > I really only want to solve "first-time-user" experience here -
>>>> nothing
>>>> > more. I
>>>> > definitely do not want to replace the current installation method for
>>>> > experienced
>>>> > users - for them using --constraint requirements.txt is exactly what
>>>> they
>>>> > need.
>>>> > The only problem I am trying to solve with that is "repeatability" of
>>>> > installation.
>>>> >
>>>> > Maybe "apache-airflow-quickinstall" or something like that would be
>>>> better
>>>> > than "apache-airflow-pinned" or "apache-airflow-repeatable-install" or
>>>> > something like that. I think about it as a "flavour" of ariflow
>>>> rather than
>>>> > anything else. I even originally implemented it as [pinned] extra
>>>> where I
>>>> > pinned all requirements. Unfortunately I found that if you have
>>>> > main requirement without limits, adding the same requirement as extra
>>>> with
>>>> > == does not make it pinned :(. That was my original plan.
>>>> >
>>>> >
>>>> > > Btw I have been on "we should have pinned dependency" camp as
>>>> Airflow
>>>> > > should definitely install without breaking since day-1 but I think a
>>>> > > separate "-pinned" package won't solve that issue.
>>>> > >
>>>> >
>>>> > Ah yeah we went the same route. I do not think we can solve the
>>>> > "library vs. app" problem easily. This is a bit of "eat-and-have-cake"
>>>> > at the same time. I know people have problems
>>>> > with conflicting dependencies when they are trying to install
>>>> libraries
>>>> > with different requirements. And I am not even trying to solve that
>>>> > problem now. Not even close. This requires some other solution
>>>> > (for example separate virtualenvs with different dependencies
>>>> > build from wheels on per-task basis). But that's something much
>>>> further
>>>> > in the future (if at all).
>>>> >
>>>> >
>>>> > >
>>>> > > WDYT? Also please do let me know if I have misunderstood something
>>>> > > (definitely possible :D).
>>>> > >
>>>> > > Regards,
>>>> > > Kaxil
>>>> >
>>>>
>>>
>>>
>>> --
>>>
>>> Jarek Potiuk
>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>
>>> M: +48 660 796 129 <+48660796129>
>>> [image: Polidea] <https://www.polidea.com/>
>>>
>>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Back to (some) dependency pinning
Posted by Jarek Potiuk <Ja...@polidea.com>.
And rebased it right now and fixed automated requirements update.
On Fri, Mar 20, 2020 at 2:28 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Ah BTW. I just noticed that for some reason I pasted an old PR earlier in
> the thread :(.
> This is the one with requirements.txt I am talking about:
> https://github.com/apache/airflow/pull/7730
>
> On Fri, Mar 20, 2020 at 2:26 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> Nope. Not blocking. I can work with my branch just requirements.txt is
>> enough for that :)
>>
>> I think the problem with semver is that it is loosely followed - we had a
>> number of breakages in the past with minor version upgrades :(.
>>
>> J.
>>
>>
>>
>> On Fri, Mar 20, 2020 at 1:27 PM Kaxil Naik <ka...@gmail.com> wrote:
>>
>>> Thanks for the detailed explanation Jarek.
>>>
>>> How about we have an upper limit for all our dependencies, example
>>> instead
>>> of "google-cloud-storage>=1.16", we have
>>> "google-cloud-storage>=1.16,<2.0" ?
>>>
>>> If a dependency breaks compatibility in minor versions, we can't do
>>> anything about it but if they follow SemVer, we should be safe and the
>>> first-time installers would have a non-breaking package. WDYT?
>>>
>>> Btw I hope this is not blocking you in building a production image as I
>>> think requirements.txt is solving that? Please let me know if it is
>>> blocking.
>>>
>>> PS: I am also just dumping my ideas to solve this issue. Love to hear
>>> what
>>> others think too.
>>>
>>> Regards,
>>> Kaxil
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Mar 19, 2020 at 2:43 PM Jarek Potiuk <Ja...@polidea.com>
>>> wrote:
>>>
>>> > I think we have similar understanding. But let me just clarify because
>>> I
>>> > think we think about we think about solving two different problems
>>> > My proposal is not solving all problems with dependencies - quite the
>>> > contrary, I want to solve just one specific "repeatability" problem -
>>> read
>>> > on :)..
>>> >
>>> > 1. A potential source of confusion: using "-pinned" for
>>> installation but
>>> > > using "non-pinned" for DAG development.
>>> > >
>>> >
>>> > This could be confusing indeed - but they are the same in fact -
>>> > just deps might be different over time.
>>> >
>>> > 2. Most of the users would still try to install "apache-airflow"
>>> package
>>> > > that might have been broken for example because of a dependency
>>> > release,
>>> > > either way, we would still have to suggest them to use "pinned"
>>> > version
>>> > >
>>> >
>>> > True. I thought we might describe it in the README and make it
>>> prominently
>>> > explained. Usually people look at the readme in PyPI when they are
>>> > installing
>>> > stuff and it does not work: https://pypi.org/project/apache-airflow/.
>>> >
>>> > Also - we could of course explain how to use requirements.txt from the
>>> > released
>>> > version when they are installing it. That would be an extra friction
>>> point
>>> > though
>>> > and maybe having "always installable" version of airflow is a better
>>> > choice.
>>> >
>>> > 3. If they install "pinned" version, it is no longer a library
>>> again,
>>> > > that is users won't be able to use new NumPy release or
>>> matplotlib for
>>> > > example. In which case we are just circling back to the same
>>> problem,
>>> > > "either we risk broken package" while releasing or we risk
>>> potentially
>>> > > incompatible versions.
>>> > >
>>> >
>>> > Yep. But maybe it's just a question of naming. Maybe even we could name
>>> > this package differently to indicate that this version is a way to
>>> quickly
>>> > install
>>> > airflow but not to do any serious development with it.
>>> >
>>> > So speaking about THE problem I want to solve with the
>>> > requirements.txt and apache-airflow-pinned package:
>>> >
>>> > I really only want to solve "first-time-user" experience here - nothing
>>> > more. I
>>> > definitely do not want to replace the current installation method for
>>> > experienced
>>> > users - for them using --constraint requirements.txt is exactly what
>>> they
>>> > need.
>>> > The only problem I am trying to solve with that is "repeatability" of
>>> > installation.
>>> >
>>> > Maybe "apache-airflow-quickinstall" or something like that would be
>>> better
>>> > than "apache-airflow-pinned" or "apache-airflow-repeatable-install" or
>>> > something like that. I think about it as a "flavour" of ariflow rather
>>> than
>>> > anything else. I even originally implemented it as [pinned] extra
>>> where I
>>> > pinned all requirements. Unfortunately I found that if you have
>>> > main requirement without limits, adding the same requirement as extra
>>> with
>>> > == does not make it pinned :(. That was my original plan.
>>> >
>>> >
>>> > > Btw I have been on "we should have pinned dependency" camp as Airflow
>>> > > should definitely install without breaking since day-1 but I think a
>>> > > separate "-pinned" package won't solve that issue.
>>> > >
>>> >
>>> > Ah yeah we went the same route. I do not think we can solve the
>>> > "library vs. app" problem easily. This is a bit of "eat-and-have-cake"
>>> > at the same time. I know people have problems
>>> > with conflicting dependencies when they are trying to install libraries
>>> > with different requirements. And I am not even trying to solve that
>>> > problem now. Not even close. This requires some other solution
>>> > (for example separate virtualenvs with different dependencies
>>> > build from wheels on per-task basis). But that's something much further
>>> > in the future (if at all).
>>> >
>>> >
>>> > >
>>> > > WDYT? Also please do let me know if I have misunderstood something
>>> > > (definitely possible :D).
>>> > >
>>> > > Regards,
>>> > > Kaxil
>>> >
>>>
>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Back to (some) dependency pinning
Posted by Jarek Potiuk <Ja...@polidea.com>.
Ah BTW. I just noticed that for some reason I pasted an old PR earlier in
the thread :(.
This is the one with requirements.txt I am talking about:
https://github.com/apache/airflow/pull/7730
On Fri, Mar 20, 2020 at 2:26 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Nope. Not blocking. I can work with my branch just requirements.txt is
> enough for that :)
>
> I think the problem with semver is that it is loosely followed - we had a
> number of breakages in the past with minor version upgrades :(.
>
> J.
>
>
>
> On Fri, Mar 20, 2020 at 1:27 PM Kaxil Naik <ka...@gmail.com> wrote:
>
>> Thanks for the detailed explanation Jarek.
>>
>> How about we have an upper limit for all our dependencies, example instead
>> of "google-cloud-storage>=1.16", we have
>> "google-cloud-storage>=1.16,<2.0" ?
>>
>> If a dependency breaks compatibility in minor versions, we can't do
>> anything about it but if they follow SemVer, we should be safe and the
>> first-time installers would have a non-breaking package. WDYT?
>>
>> Btw I hope this is not blocking you in building a production image as I
>> think requirements.txt is solving that? Please let me know if it is
>> blocking.
>>
>> PS: I am also just dumping my ideas to solve this issue. Love to hear what
>> others think too.
>>
>> Regards,
>> Kaxil
>>
>>
>>
>>
>>
>> On Thu, Mar 19, 2020 at 2:43 PM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>> > I think we have similar understanding. But let me just clarify because I
>> > think we think about we think about solving two different problems
>> > My proposal is not solving all problems with dependencies - quite the
>> > contrary, I want to solve just one specific "repeatability" problem -
>> read
>> > on :)..
>> >
>> > 1. A potential source of confusion: using "-pinned" for installation
>> but
>> > > using "non-pinned" for DAG development.
>> > >
>> >
>> > This could be confusing indeed - but they are the same in fact -
>> > just deps might be different over time.
>> >
>> > 2. Most of the users would still try to install "apache-airflow"
>> package
>> > > that might have been broken for example because of a dependency
>> > release,
>> > > either way, we would still have to suggest them to use "pinned"
>> > version
>> > >
>> >
>> > True. I thought we might describe it in the README and make it
>> prominently
>> > explained. Usually people look at the readme in PyPI when they are
>> > installing
>> > stuff and it does not work: https://pypi.org/project/apache-airflow/.
>> >
>> > Also - we could of course explain how to use requirements.txt from the
>> > released
>> > version when they are installing it. That would be an extra friction
>> point
>> > though
>> > and maybe having "always installable" version of airflow is a better
>> > choice.
>> >
>> > 3. If they install "pinned" version, it is no longer a library again,
>> > > that is users won't be able to use new NumPy release or matplotlib
>> for
>> > > example. In which case we are just circling back to the same
>> problem,
>> > > "either we risk broken package" while releasing or we risk
>> potentially
>> > > incompatible versions.
>> > >
>> >
>> > Yep. But maybe it's just a question of naming. Maybe even we could name
>> > this package differently to indicate that this version is a way to
>> quickly
>> > install
>> > airflow but not to do any serious development with it.
>> >
>> > So speaking about THE problem I want to solve with the
>> > requirements.txt and apache-airflow-pinned package:
>> >
>> > I really only want to solve "first-time-user" experience here - nothing
>> > more. I
>> > definitely do not want to replace the current installation method for
>> > experienced
>> > users - for them using --constraint requirements.txt is exactly what
>> they
>> > need.
>> > The only problem I am trying to solve with that is "repeatability" of
>> > installation.
>> >
>> > Maybe "apache-airflow-quickinstall" or something like that would be
>> better
>> > than "apache-airflow-pinned" or "apache-airflow-repeatable-install" or
>> > something like that. I think about it as a "flavour" of ariflow rather
>> than
>> > anything else. I even originally implemented it as [pinned] extra where
>> I
>> > pinned all requirements. Unfortunately I found that if you have
>> > main requirement without limits, adding the same requirement as extra
>> with
>> > == does not make it pinned :(. That was my original plan.
>> >
>> >
>> > > Btw I have been on "we should have pinned dependency" camp as Airflow
>> > > should definitely install without breaking since day-1 but I think a
>> > > separate "-pinned" package won't solve that issue.
>> > >
>> >
>> > Ah yeah we went the same route. I do not think we can solve the
>> > "library vs. app" problem easily. This is a bit of "eat-and-have-cake"
>> > at the same time. I know people have problems
>> > with conflicting dependencies when they are trying to install libraries
>> > with different requirements. And I am not even trying to solve that
>> > problem now. Not even close. This requires some other solution
>> > (for example separate virtualenvs with different dependencies
>> > build from wheels on per-task basis). But that's something much further
>> > in the future (if at all).
>> >
>> >
>> > >
>> > > WDYT? Also please do let me know if I have misunderstood something
>> > > (definitely possible :D).
>> > >
>> > > Regards,
>> > > Kaxil
>> >
>>
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Back to (some) dependency pinning
Posted by Jarek Potiuk <Ja...@polidea.com>.
Nope. Not blocking. I can work with my branch just requirements.txt is
enough for that :)
I think the problem with semver is that it is loosely followed - we had a
number of breakages in the past with minor version upgrades :(.
J.
On Fri, Mar 20, 2020 at 1:27 PM Kaxil Naik <ka...@gmail.com> wrote:
> Thanks for the detailed explanation Jarek.
>
> How about we have an upper limit for all our dependencies, example instead
> of "google-cloud-storage>=1.16", we have "google-cloud-storage>=1.16,<2.0"
> ?
>
> If a dependency breaks compatibility in minor versions, we can't do
> anything about it but if they follow SemVer, we should be safe and the
> first-time installers would have a non-breaking package. WDYT?
>
> Btw I hope this is not blocking you in building a production image as I
> think requirements.txt is solving that? Please let me know if it is
> blocking.
>
> PS: I am also just dumping my ideas to solve this issue. Love to hear what
> others think too.
>
> Regards,
> Kaxil
>
>
>
>
>
> On Thu, Mar 19, 2020 at 2:43 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
> > I think we have similar understanding. But let me just clarify because I
> > think we think about we think about solving two different problems
> > My proposal is not solving all problems with dependencies - quite the
> > contrary, I want to solve just one specific "repeatability" problem -
> read
> > on :)..
> >
> > 1. A potential source of confusion: using "-pinned" for installation
> but
> > > using "non-pinned" for DAG development.
> > >
> >
> > This could be confusing indeed - but they are the same in fact -
> > just deps might be different over time.
> >
> > 2. Most of the users would still try to install "apache-airflow"
> package
> > > that might have been broken for example because of a dependency
> > release,
> > > either way, we would still have to suggest them to use "pinned"
> > version
> > >
> >
> > True. I thought we might describe it in the README and make it
> prominently
> > explained. Usually people look at the readme in PyPI when they are
> > installing
> > stuff and it does not work: https://pypi.org/project/apache-airflow/.
> >
> > Also - we could of course explain how to use requirements.txt from the
> > released
> > version when they are installing it. That would be an extra friction
> point
> > though
> > and maybe having "always installable" version of airflow is a better
> > choice.
> >
> > 3. If they install "pinned" version, it is no longer a library again,
> > > that is users won't be able to use new NumPy release or matplotlib
> for
> > > example. In which case we are just circling back to the same
> problem,
> > > "either we risk broken package" while releasing or we risk
> potentially
> > > incompatible versions.
> > >
> >
> > Yep. But maybe it's just a question of naming. Maybe even we could name
> > this package differently to indicate that this version is a way to
> quickly
> > install
> > airflow but not to do any serious development with it.
> >
> > So speaking about THE problem I want to solve with the
> > requirements.txt and apache-airflow-pinned package:
> >
> > I really only want to solve "first-time-user" experience here - nothing
> > more. I
> > definitely do not want to replace the current installation method for
> > experienced
> > users - for them using --constraint requirements.txt is exactly what they
> > need.
> > The only problem I am trying to solve with that is "repeatability" of
> > installation.
> >
> > Maybe "apache-airflow-quickinstall" or something like that would be
> better
> > than "apache-airflow-pinned" or "apache-airflow-repeatable-install" or
> > something like that. I think about it as a "flavour" of ariflow rather
> than
> > anything else. I even originally implemented it as [pinned] extra where I
> > pinned all requirements. Unfortunately I found that if you have
> > main requirement without limits, adding the same requirement as extra
> with
> > == does not make it pinned :(. That was my original plan.
> >
> >
> > > Btw I have been on "we should have pinned dependency" camp as Airflow
> > > should definitely install without breaking since day-1 but I think a
> > > separate "-pinned" package won't solve that issue.
> > >
> >
> > Ah yeah we went the same route. I do not think we can solve the
> > "library vs. app" problem easily. This is a bit of "eat-and-have-cake"
> > at the same time. I know people have problems
> > with conflicting dependencies when they are trying to install libraries
> > with different requirements. And I am not even trying to solve that
> > problem now. Not even close. This requires some other solution
> > (for example separate virtualenvs with different dependencies
> > build from wheels on per-task basis). But that's something much further
> > in the future (if at all).
> >
> >
> > >
> > > WDYT? Also please do let me know if I have misunderstood something
> > > (definitely possible :D).
> > >
> > > Regards,
> > > Kaxil
> >
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Back to (some) dependency pinning
Posted by Kaxil Naik <ka...@gmail.com>.
Thanks for the detailed explanation Jarek.
How about we have an upper limit for all our dependencies, example instead
of "google-cloud-storage>=1.16", we have "google-cloud-storage>=1.16,<2.0" ?
If a dependency breaks compatibility in minor versions, we can't do
anything about it but if they follow SemVer, we should be safe and the
first-time installers would have a non-breaking package. WDYT?
Btw I hope this is not blocking you in building a production image as I
think requirements.txt is solving that? Please let me know if it is
blocking.
PS: I am also just dumping my ideas to solve this issue. Love to hear what
others think too.
Regards,
Kaxil
On Thu, Mar 19, 2020 at 2:43 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> I think we have similar understanding. But let me just clarify because I
> think we think about we think about solving two different problems
> My proposal is not solving all problems with dependencies - quite the
> contrary, I want to solve just one specific "repeatability" problem - read
> on :)..
>
> 1. A potential source of confusion: using "-pinned" for installation but
> > using "non-pinned" for DAG development.
> >
>
> This could be confusing indeed - but they are the same in fact -
> just deps might be different over time.
>
> 2. Most of the users would still try to install "apache-airflow" package
> > that might have been broken for example because of a dependency
> release,
> > either way, we would still have to suggest them to use "pinned"
> version
> >
>
> True. I thought we might describe it in the README and make it prominently
> explained. Usually people look at the readme in PyPI when they are
> installing
> stuff and it does not work: https://pypi.org/project/apache-airflow/.
>
> Also - we could of course explain how to use requirements.txt from the
> released
> version when they are installing it. That would be an extra friction point
> though
> and maybe having "always installable" version of airflow is a better
> choice.
>
> 3. If they install "pinned" version, it is no longer a library again,
> > that is users won't be able to use new NumPy release or matplotlib for
> > example. In which case we are just circling back to the same problem,
> > "either we risk broken package" while releasing or we risk potentially
> > incompatible versions.
> >
>
> Yep. But maybe it's just a question of naming. Maybe even we could name
> this package differently to indicate that this version is a way to quickly
> install
> airflow but not to do any serious development with it.
>
> So speaking about THE problem I want to solve with the
> requirements.txt and apache-airflow-pinned package:
>
> I really only want to solve "first-time-user" experience here - nothing
> more. I
> definitely do not want to replace the current installation method for
> experienced
> users - for them using --constraint requirements.txt is exactly what they
> need.
> The only problem I am trying to solve with that is "repeatability" of
> installation.
>
> Maybe "apache-airflow-quickinstall" or something like that would be better
> than "apache-airflow-pinned" or "apache-airflow-repeatable-install" or
> something like that. I think about it as a "flavour" of ariflow rather than
> anything else. I even originally implemented it as [pinned] extra where I
> pinned all requirements. Unfortunately I found that if you have
> main requirement without limits, adding the same requirement as extra with
> == does not make it pinned :(. That was my original plan.
>
>
> > Btw I have been on "we should have pinned dependency" camp as Airflow
> > should definitely install without breaking since day-1 but I think a
> > separate "-pinned" package won't solve that issue.
> >
>
> Ah yeah we went the same route. I do not think we can solve the
> "library vs. app" problem easily. This is a bit of "eat-and-have-cake"
> at the same time. I know people have problems
> with conflicting dependencies when they are trying to install libraries
> with different requirements. And I am not even trying to solve that
> problem now. Not even close. This requires some other solution
> (for example separate virtualenvs with different dependencies
> build from wheels on per-task basis). But that's something much further
> in the future (if at all).
>
>
> >
> > WDYT? Also please do let me know if I have misunderstood something
> > (definitely possible :D).
> >
> > Regards,
> > Kaxil
>
Re: [DISCUSS] Back to (some) dependency pinning
Posted by Jarek Potiuk <Ja...@polidea.com>.
I think we have similar understanding. But let me just clarify because I
think we think about we think about solving two different problems
My proposal is not solving all problems with dependencies - quite the
contrary, I want to solve just one specific "repeatability" problem - read
on :)..
1. A potential source of confusion: using "-pinned" for installation but
> using "non-pinned" for DAG development.
>
This could be confusing indeed - but they are the same in fact -
just deps might be different over time.
2. Most of the users would still try to install "apache-airflow" package
> that might have been broken for example because of a dependency release,
> either way, we would still have to suggest them to use "pinned" version
>
True. I thought we might describe it in the README and make it prominently
explained. Usually people look at the readme in PyPI when they are
installing
stuff and it does not work: https://pypi.org/project/apache-airflow/.
Also - we could of course explain how to use requirements.txt from the
released
version when they are installing it. That would be an extra friction point
though
and maybe having "always installable" version of airflow is a better choice.
3. If they install "pinned" version, it is no longer a library again,
> that is users won't be able to use new NumPy release or matplotlib for
> example. In which case we are just circling back to the same problem,
> "either we risk broken package" while releasing or we risk potentially
> incompatible versions.
>
Yep. But maybe it's just a question of naming. Maybe even we could name
this package differently to indicate that this version is a way to quickly
install
airflow but not to do any serious development with it.
So speaking about THE problem I want to solve with the
requirements.txt and apache-airflow-pinned package:
I really only want to solve "first-time-user" experience here - nothing
more. I
definitely do not want to replace the current installation method for
experienced
users - for them using --constraint requirements.txt is exactly what they
need.
The only problem I am trying to solve with that is "repeatability" of
installation.
Maybe "apache-airflow-quickinstall" or something like that would be better
than "apache-airflow-pinned" or "apache-airflow-repeatable-install" or
something like that. I think about it as a "flavour" of ariflow rather than
anything else. I even originally implemented it as [pinned] extra where I
pinned all requirements. Unfortunately I found that if you have
main requirement without limits, adding the same requirement as extra with
== does not make it pinned :(. That was my original plan.
> Btw I have been on "we should have pinned dependency" camp as Airflow
> should definitely install without breaking since day-1 but I think a
> separate "-pinned" package won't solve that issue.
>
Ah yeah we went the same route. I do not think we can solve the
"library vs. app" problem easily. This is a bit of "eat-and-have-cake"
at the same time. I know people have problems
with conflicting dependencies when they are trying to install libraries
with different requirements. And I am not even trying to solve that
problem now. Not even close. This requires some other solution
(for example separate virtualenvs with different dependencies
build from wheels on per-task basis). But that's something much further
in the future (if at all).
>
> WDYT? Also please do let me know if I have misunderstood something
> (definitely possible :D).
>
> Regards,
> Kaxil
Re: [DISCUSS] Back to (some) dependency pinning
Posted by Kaxil Naik <ka...@gmail.com>.
I think the "-pinned" version might have following
1. A potential source of confusion: using "-pinned" for installation but
using "non-pinned" for DAG development.
2. Most of the users would still try to install "apache-airflow" package
that might have been broken for example because of a dependency release,
either way, we would still have to suggest them to use "pinned" version
3. If they install "pinned" version, it is no longer a library again,
that is users won't be able to use new NumPy release or matplotlib for
example. In which case we are just circling back to the same problem,
"either we risk broken package" while releasing or we risk potentially
incompatible versions.
Hence I feel a good compromise would be a requirements.txt file if the
airflow breaks on installation.
Btw I have been on "we should have pinned dependency" camp as Airflow
should definitely install without breaking since day-1 but I think a
separate "-pinned" package won't solve that issue.
WDYT? Also please do let me know if I have misunderstood something
(definitely possible :D).
Regards,
Kaxil
On Thu, Mar 19, 2020 at 1:42 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Agree Kaxil that requirements.txt is crucial. The requirements.txt is
> already part of this change
> https://github.com/apache/airflow/pull/4938 (and it is essential to
> the whole concept).
>
> From what I see no-one complains about requirements.txt (the nice
> thing is that we will have it automatically updated based on our
> pre-commit + Ci image) and we have a nice workflow how to make sure it
> is automatically updated along the way of development (basically
> automatically by whoever who changes setup.py). Also it will help us
> with any automated vulnerability scanning (we will be able to deploy
> github's dependabot more easily then).
>
> Let me just summarize where we are (I think):
>
> I think the only contention point/doubt is whether we need the
> *-pinned package at all. I don't need that package for production
> image (I just need the requirements.txt). But for me there are two
> arguments pro/con:
>
> Pro:
> - Having the pinned package will make it easy for everyone to be able
> to consistently install airflow to begin with and test it. Right now
> it might put-off the users if they do 'pip install
> apache-airflow==1.10.8' and it does not install and they have to
> google how. Over time it happens with any released Airflow version
> sooner or later (with 1.10.8 it happened in a matter of hours). Due to
> that I think Airflow might be perceived as not-really-stable.
>
> Con:
> - We will have two versions of airflow released ("apache-airflow" and
> "apache-airflow-pinned"). I don't think it makes a lot of overhead for
> testing etc. (we can automatically compare and verify installation for
> both and see that they are equivalent at the time of release). But
> maybe I am wrong about that one :)
>
> I am thinking about starting a vote on that. WDYT? Did we have enough
> discussion?
>
> J.
>
>
> On Thu, Mar 19, 2020 at 1:03 PM Kaxil Naik <ka...@gmail.com> wrote:
> >
> > I think we can have a requirements.txt (freezed when the package is
> > released, similar to yarn.lock) instead of releasing a separate
> > apache-airflow-pinned package.
> >
> > Regards,
> > Kaxil
> >
> > On Tue, Mar 17, 2020 at 7:38 PM Ash Berlin-Taylor <as...@apache.org>
> wrote:
> >
> > > I think irrespective of what we do about releasing a pinned version,
> using
> > > this approach so our prod image is repeatable sounds good!
> > >
> > > On 17 March 2020 19:17:59 GMT, Jarek Potiuk <Ja...@polidea.com>
> > > wrote:
> > > >Any other comments?
> > > >
> > > >I'd love to hear your thoughts. It's the one thing that maybe not
> keeps
> > > >me
> > > >from prod image, But I would love to know if I can rely on the
> > > >requirements.txt being part of the source code so that I can use it
> > > >when building the prod image..
> > > >
> > > >J.
> > > >
> > > >
> > > >On Mon, Mar 16, 2020 at 12:16 PM Jarek Potiuk
> > > ><Ja...@polidea.com>
> > > >wrote:
> > > >
> > > >>
> > > >> On Mon, Mar 16, 2020 at 11:16 AM Driesprong, Fokko
> > > ><fo...@driesprong.frl>
> > > >> wrote:
> > > >>
> > > >>> Personally I don't like to have two versions in the PyPi repo. This
> > > >also
> > > >>> complicates the releases, since we need to test, release and verify
> > > >two
> > > >>> versions of Airflow
> > > >>>
> > > >>
> > > >> Not necessarily. Release is automated (and I run both package builds
> > > >in CI
> > > >> now).
> > > >> Regarding testing - at the moment of release 'apache-airflow` and
> > > >> 'apache-airflow-pinned' are 1-1 equivalent. Their behaviour starts
> to
> > > >be
> > > >> different when something out of our control happens with
> dependencies
> > > >after
> > > >> the release. We can even have automated tests and install both
> > > >> 'apache-airflow' and 'apache-airlfow-pinned' at release time and
> > > >compare
> > > >> the virtualenvs on binary level to verify that they are equivalent.
> > > >This
> > > >> way we can just test one of them.
> > > >>
> > > >>
> > > >>> I'm afraid that this might confuse users.
> > > >>>
> > > >>
> > > >> I think that the current situation is much more confusing for the
> > > >users.
> > > >> In order to install <1.10.8 for example you need to know that you
> > > >should
> > > >> run 'pip install apache-airflow==1.10.8 Werkzeug<1.0'. If you want
> to
> > > >> install 1.10.2 you need to add other constraints. And tomorrow
> > > >another
> > > >> constraint might be needed to install 1.10.9. This is also a hard
> > > >blocker
> > > >> for the official, production image of Apache Airflow. One of the
> > > >> constraints to make an official Docker image is repeatability
> > > >> https://github.com/docker-library/official-images#repeatability .
> We
> > > >> don't have repeatability now.
> > > >>
> > > >> We need to consider what our alternatives are. I think our problem
> is
> > > >that:
> > > >>
> > > >> 1) we want to have it open for operator development
> > > >> 2) we want the user (and official docker build) to have a reliable
> > > >way to
> > > >> install the released version. This already happened and will happen
> > > >that
> > > >> out of a sudden released version of airflow will stop installing
> > > >without
> > > >> magic 'Werkzeug <= 1.0' or similar.
> > > >>
> > > >> We already have good solution for 1) with open dependencies . I see
> > > >two
> > > >> solutions to that:
> > > >>
> > > >> *Solution 1) Ask the users to use requirements.txt as constraint
> file
> > > >> manually:*
> > > >>
> > > >> a) manually download requirements.txt from the right release in
> > > >github
> > > >> b) pip install apache-airflow==1.10.9 --constraint requirements.txt
> > > >>
> > > >> Or
> > > >>
> > > >> *Solution 2) Use "-pinned" package (effectively does the same):*
> > > >>
> > > >> pip install apache-airflow-pinned==1.10.9
> > > >>
> > > >> I find the second one much less confusing and straightforward..
> > > >>
> > > >>
> > > >>
> > > >> I am all ears if we have any other solution? Can we discuss some
> > > >potential
> > > >> options here?
> > > >>
> > > >>
> > > >> Besides that, it feels a bit like we're reinventing certain
> > > >mechanisms that
> > > >>> are already in tooling like Poetry.
> > > >>>
> > > >>
> > > >> Unfortunately according to my checks - Poetry does not handle our
> > > >case
> > > >> where we are both library and application at the same time. Form
> what
> > > >I
> > > >> checked poetry uses .lock file and publishes the "pinned" version
> > > >always in
> > > >> such case. Hacking it to support both cases would be
> super-difficult
> > > >and
> > > >> defeat the purpose of using ready-to-use-tool, I looked at it and
> > > >could not
> > > >> find a way to achieve both 1) and 2). If you think we can do it, It
> > > >would
> > > >> be great to discuss the approach we can take. If we can agree that
> we
> > > >only
> > > >> publish pinned version, I would be super happy and switch to poetry
> > > >> immediately - but we cannot do it I am afraid. There was a long
> > > >discussion
> > > >> about it and we cannot afford pinning by default, unfortunately.
> This
> > > >has
> > > >> already been discussed several times and unless this assumption has
> > > >changed
> > > >> - we cannot use poetry or similar.
> > > >>
> > > >> My 2ct.
> > > >>>
> > > >> --
> > > >>
> > > >> Jarek Potiuk
> > > >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > >>
> > > >> M: +48 660 796 129 <+48660796129>
> > > >> [image: Polidea] <https://www.polidea.com/>
> > > >>
> > > >>
> > > >
> > > >--
> > > >
> > > >Jarek Potiuk
> > > >Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > >
> > > >M: +48 660 796 129 <+48660796129>
> > > >[image: Polidea] <https://www.polidea.com/>
> > >
>
>
>
> --
>
> Jarek Potiuk
> Polidea | Principal Software Engineer
>
> M: +48 660 796 129
>
Re: [DISCUSS] Back to (some) dependency pinning
Posted by Jarek Potiuk <Ja...@polidea.com>.
Agree Kaxil that requirements.txt is crucial. The requirements.txt is
already part of this change
https://github.com/apache/airflow/pull/4938 (and it is essential to
the whole concept).
From what I see no-one complains about requirements.txt (the nice
thing is that we will have it automatically updated based on our
pre-commit + Ci image) and we have a nice workflow how to make sure it
is automatically updated along the way of development (basically
automatically by whoever who changes setup.py). Also it will help us
with any automated vulnerability scanning (we will be able to deploy
github's dependabot more easily then).
Let me just summarize where we are (I think):
I think the only contention point/doubt is whether we need the
*-pinned package at all. I don't need that package for production
image (I just need the requirements.txt). But for me there are two
arguments pro/con:
Pro:
- Having the pinned package will make it easy for everyone to be able
to consistently install airflow to begin with and test it. Right now
it might put-off the users if they do 'pip install
apache-airflow==1.10.8' and it does not install and they have to
google how. Over time it happens with any released Airflow version
sooner or later (with 1.10.8 it happened in a matter of hours). Due to
that I think Airflow might be perceived as not-really-stable.
Con:
- We will have two versions of airflow released ("apache-airflow" and
"apache-airflow-pinned"). I don't think it makes a lot of overhead for
testing etc. (we can automatically compare and verify installation for
both and see that they are equivalent at the time of release). But
maybe I am wrong about that one :)
I am thinking about starting a vote on that. WDYT? Did we have enough
discussion?
J.
On Thu, Mar 19, 2020 at 1:03 PM Kaxil Naik <ka...@gmail.com> wrote:
>
> I think we can have a requirements.txt (freezed when the package is
> released, similar to yarn.lock) instead of releasing a separate
> apache-airflow-pinned package.
>
> Regards,
> Kaxil
>
> On Tue, Mar 17, 2020 at 7:38 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>
> > I think irrespective of what we do about releasing a pinned version, using
> > this approach so our prod image is repeatable sounds good!
> >
> > On 17 March 2020 19:17:59 GMT, Jarek Potiuk <Ja...@polidea.com>
> > wrote:
> > >Any other comments?
> > >
> > >I'd love to hear your thoughts. It's the one thing that maybe not keeps
> > >me
> > >from prod image, But I would love to know if I can rely on the
> > >requirements.txt being part of the source code so that I can use it
> > >when building the prod image..
> > >
> > >J.
> > >
> > >
> > >On Mon, Mar 16, 2020 at 12:16 PM Jarek Potiuk
> > ><Ja...@polidea.com>
> > >wrote:
> > >
> > >>
> > >> On Mon, Mar 16, 2020 at 11:16 AM Driesprong, Fokko
> > ><fo...@driesprong.frl>
> > >> wrote:
> > >>
> > >>> Personally I don't like to have two versions in the PyPi repo. This
> > >also
> > >>> complicates the releases, since we need to test, release and verify
> > >two
> > >>> versions of Airflow
> > >>>
> > >>
> > >> Not necessarily. Release is automated (and I run both package builds
> > >in CI
> > >> now).
> > >> Regarding testing - at the moment of release 'apache-airflow` and
> > >> 'apache-airflow-pinned' are 1-1 equivalent. Their behaviour starts to
> > >be
> > >> different when something out of our control happens with dependencies
> > >after
> > >> the release. We can even have automated tests and install both
> > >> 'apache-airflow' and 'apache-airlfow-pinned' at release time and
> > >compare
> > >> the virtualenvs on binary level to verify that they are equivalent.
> > >This
> > >> way we can just test one of them.
> > >>
> > >>
> > >>> I'm afraid that this might confuse users.
> > >>>
> > >>
> > >> I think that the current situation is much more confusing for the
> > >users.
> > >> In order to install <1.10.8 for example you need to know that you
> > >should
> > >> run 'pip install apache-airflow==1.10.8 Werkzeug<1.0'. If you want to
> > >> install 1.10.2 you need to add other constraints. And tomorrow
> > >another
> > >> constraint might be needed to install 1.10.9. This is also a hard
> > >blocker
> > >> for the official, production image of Apache Airflow. One of the
> > >> constraints to make an official Docker image is repeatability
> > >> https://github.com/docker-library/official-images#repeatability . We
> > >> don't have repeatability now.
> > >>
> > >> We need to consider what our alternatives are. I think our problem is
> > >that:
> > >>
> > >> 1) we want to have it open for operator development
> > >> 2) we want the user (and official docker build) to have a reliable
> > >way to
> > >> install the released version. This already happened and will happen
> > >that
> > >> out of a sudden released version of airflow will stop installing
> > >without
> > >> magic 'Werkzeug <= 1.0' or similar.
> > >>
> > >> We already have good solution for 1) with open dependencies . I see
> > >two
> > >> solutions to that:
> > >>
> > >> *Solution 1) Ask the users to use requirements.txt as constraint file
> > >> manually:*
> > >>
> > >> a) manually download requirements.txt from the right release in
> > >github
> > >> b) pip install apache-airflow==1.10.9 --constraint requirements.txt
> > >>
> > >> Or
> > >>
> > >> *Solution 2) Use "-pinned" package (effectively does the same):*
> > >>
> > >> pip install apache-airflow-pinned==1.10.9
> > >>
> > >> I find the second one much less confusing and straightforward..
> > >>
> > >>
> > >>
> > >> I am all ears if we have any other solution? Can we discuss some
> > >potential
> > >> options here?
> > >>
> > >>
> > >> Besides that, it feels a bit like we're reinventing certain
> > >mechanisms that
> > >>> are already in tooling like Poetry.
> > >>>
> > >>
> > >> Unfortunately according to my checks - Poetry does not handle our
> > >case
> > >> where we are both library and application at the same time. Form what
> > >I
> > >> checked poetry uses .lock file and publishes the "pinned" version
> > >always in
> > >> such case. Hacking it to support both cases would be super-difficult
> > >and
> > >> defeat the purpose of using ready-to-use-tool, I looked at it and
> > >could not
> > >> find a way to achieve both 1) and 2). If you think we can do it, It
> > >would
> > >> be great to discuss the approach we can take. If we can agree that we
> > >only
> > >> publish pinned version, I would be super happy and switch to poetry
> > >> immediately - but we cannot do it I am afraid. There was a long
> > >discussion
> > >> about it and we cannot afford pinning by default, unfortunately. This
> > >has
> > >> already been discussed several times and unless this assumption has
> > >changed
> > >> - we cannot use poetry or similar.
> > >>
> > >> My 2ct.
> > >>>
> > >> --
> > >>
> > >> Jarek Potiuk
> > >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >>
> > >> M: +48 660 796 129 <+48660796129>
> > >> [image: Polidea] <https://www.polidea.com/>
> > >>
> > >>
> > >
> > >--
> > >
> > >Jarek Potiuk
> > >Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >
> > >M: +48 660 796 129 <+48660796129>
> > >[image: Polidea] <https://www.polidea.com/>
> >
--
Jarek Potiuk
Polidea | Principal Software Engineer
M: +48 660 796 129
Re: [DISCUSS] Back to (some) dependency pinning
Posted by Kaxil Naik <ka...@gmail.com>.
I think we can have a requirements.txt (freezed when the package is
released, similar to yarn.lock) instead of releasing a separate
apache-airflow-pinned package.
Regards,
Kaxil
On Tue, Mar 17, 2020 at 7:38 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> I think irrespective of what we do about releasing a pinned version, using
> this approach so our prod image is repeatable sounds good!
>
> On 17 March 2020 19:17:59 GMT, Jarek Potiuk <Ja...@polidea.com>
> wrote:
> >Any other comments?
> >
> >I'd love to hear your thoughts. It's the one thing that maybe not keeps
> >me
> >from prod image, But I would love to know if I can rely on the
> >requirements.txt being part of the source code so that I can use it
> >when building the prod image..
> >
> >J.
> >
> >
> >On Mon, Mar 16, 2020 at 12:16 PM Jarek Potiuk
> ><Ja...@polidea.com>
> >wrote:
> >
> >>
> >> On Mon, Mar 16, 2020 at 11:16 AM Driesprong, Fokko
> ><fo...@driesprong.frl>
> >> wrote:
> >>
> >>> Personally I don't like to have two versions in the PyPi repo. This
> >also
> >>> complicates the releases, since we need to test, release and verify
> >two
> >>> versions of Airflow
> >>>
> >>
> >> Not necessarily. Release is automated (and I run both package builds
> >in CI
> >> now).
> >> Regarding testing - at the moment of release 'apache-airflow` and
> >> 'apache-airflow-pinned' are 1-1 equivalent. Their behaviour starts to
> >be
> >> different when something out of our control happens with dependencies
> >after
> >> the release. We can even have automated tests and install both
> >> 'apache-airflow' and 'apache-airlfow-pinned' at release time and
> >compare
> >> the virtualenvs on binary level to verify that they are equivalent.
> >This
> >> way we can just test one of them.
> >>
> >>
> >>> I'm afraid that this might confuse users.
> >>>
> >>
> >> I think that the current situation is much more confusing for the
> >users.
> >> In order to install <1.10.8 for example you need to know that you
> >should
> >> run 'pip install apache-airflow==1.10.8 Werkzeug<1.0'. If you want to
> >> install 1.10.2 you need to add other constraints. And tomorrow
> >another
> >> constraint might be needed to install 1.10.9. This is also a hard
> >blocker
> >> for the official, production image of Apache Airflow. One of the
> >> constraints to make an official Docker image is repeatability
> >> https://github.com/docker-library/official-images#repeatability . We
> >> don't have repeatability now.
> >>
> >> We need to consider what our alternatives are. I think our problem is
> >that:
> >>
> >> 1) we want to have it open for operator development
> >> 2) we want the user (and official docker build) to have a reliable
> >way to
> >> install the released version. This already happened and will happen
> >that
> >> out of a sudden released version of airflow will stop installing
> >without
> >> magic 'Werkzeug <= 1.0' or similar.
> >>
> >> We already have good solution for 1) with open dependencies . I see
> >two
> >> solutions to that:
> >>
> >> *Solution 1) Ask the users to use requirements.txt as constraint file
> >> manually:*
> >>
> >> a) manually download requirements.txt from the right release in
> >github
> >> b) pip install apache-airflow==1.10.9 --constraint requirements.txt
> >>
> >> Or
> >>
> >> *Solution 2) Use "-pinned" package (effectively does the same):*
> >>
> >> pip install apache-airflow-pinned==1.10.9
> >>
> >> I find the second one much less confusing and straightforward..
> >>
> >>
> >>
> >> I am all ears if we have any other solution? Can we discuss some
> >potential
> >> options here?
> >>
> >>
> >> Besides that, it feels a bit like we're reinventing certain
> >mechanisms that
> >>> are already in tooling like Poetry.
> >>>
> >>
> >> Unfortunately according to my checks - Poetry does not handle our
> >case
> >> where we are both library and application at the same time. Form what
> >I
> >> checked poetry uses .lock file and publishes the "pinned" version
> >always in
> >> such case. Hacking it to support both cases would be super-difficult
> >and
> >> defeat the purpose of using ready-to-use-tool, I looked at it and
> >could not
> >> find a way to achieve both 1) and 2). If you think we can do it, It
> >would
> >> be great to discuss the approach we can take. If we can agree that we
> >only
> >> publish pinned version, I would be super happy and switch to poetry
> >> immediately - but we cannot do it I am afraid. There was a long
> >discussion
> >> about it and we cannot afford pinning by default, unfortunately. This
> >has
> >> already been discussed several times and unless this assumption has
> >changed
> >> - we cannot use poetry or similar.
> >>
> >> My 2ct.
> >>>
> >> --
> >>
> >> Jarek Potiuk
> >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>
> >> M: +48 660 796 129 <+48660796129>
> >> [image: Polidea] <https://www.polidea.com/>
> >>
> >>
> >
> >--
> >
> >Jarek Potiuk
> >Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> >M: +48 660 796 129 <+48660796129>
> >[image: Polidea] <https://www.polidea.com/>
>
Re: [DISCUSS] Back to (some) dependency pinning
Posted by Ash Berlin-Taylor <as...@apache.org>.
I think irrespective of what we do about releasing a pinned version, using this approach so our prod image is repeatable sounds good!
On 17 March 2020 19:17:59 GMT, Jarek Potiuk <Ja...@polidea.com> wrote:
>Any other comments?
>
>I'd love to hear your thoughts. It's the one thing that maybe not keeps
>me
>from prod image, But I would love to know if I can rely on the
>requirements.txt being part of the source code so that I can use it
>when building the prod image..
>
>J.
>
>
>On Mon, Mar 16, 2020 at 12:16 PM Jarek Potiuk
><Ja...@polidea.com>
>wrote:
>
>>
>> On Mon, Mar 16, 2020 at 11:16 AM Driesprong, Fokko
><fo...@driesprong.frl>
>> wrote:
>>
>>> Personally I don't like to have two versions in the PyPi repo. This
>also
>>> complicates the releases, since we need to test, release and verify
>two
>>> versions of Airflow
>>>
>>
>> Not necessarily. Release is automated (and I run both package builds
>in CI
>> now).
>> Regarding testing - at the moment of release 'apache-airflow` and
>> 'apache-airflow-pinned' are 1-1 equivalent. Their behaviour starts to
>be
>> different when something out of our control happens with dependencies
>after
>> the release. We can even have automated tests and install both
>> 'apache-airflow' and 'apache-airlfow-pinned' at release time and
>compare
>> the virtualenvs on binary level to verify that they are equivalent.
>This
>> way we can just test one of them.
>>
>>
>>> I'm afraid that this might confuse users.
>>>
>>
>> I think that the current situation is much more confusing for the
>users.
>> In order to install <1.10.8 for example you need to know that you
>should
>> run 'pip install apache-airflow==1.10.8 Werkzeug<1.0'. If you want to
>> install 1.10.2 you need to add other constraints. And tomorrow
>another
>> constraint might be needed to install 1.10.9. This is also a hard
>blocker
>> for the official, production image of Apache Airflow. One of the
>> constraints to make an official Docker image is repeatability
>> https://github.com/docker-library/official-images#repeatability . We
>> don't have repeatability now.
>>
>> We need to consider what our alternatives are. I think our problem is
>that:
>>
>> 1) we want to have it open for operator development
>> 2) we want the user (and official docker build) to have a reliable
>way to
>> install the released version. This already happened and will happen
>that
>> out of a sudden released version of airflow will stop installing
>without
>> magic 'Werkzeug <= 1.0' or similar.
>>
>> We already have good solution for 1) with open dependencies . I see
>two
>> solutions to that:
>>
>> *Solution 1) Ask the users to use requirements.txt as constraint file
>> manually:*
>>
>> a) manually download requirements.txt from the right release in
>github
>> b) pip install apache-airflow==1.10.9 --constraint requirements.txt
>>
>> Or
>>
>> *Solution 2) Use "-pinned" package (effectively does the same):*
>>
>> pip install apache-airflow-pinned==1.10.9
>>
>> I find the second one much less confusing and straightforward..
>>
>>
>>
>> I am all ears if we have any other solution? Can we discuss some
>potential
>> options here?
>>
>>
>> Besides that, it feels a bit like we're reinventing certain
>mechanisms that
>>> are already in tooling like Poetry.
>>>
>>
>> Unfortunately according to my checks - Poetry does not handle our
>case
>> where we are both library and application at the same time. Form what
>I
>> checked poetry uses .lock file and publishes the "pinned" version
>always in
>> such case. Hacking it to support both cases would be super-difficult
>and
>> defeat the purpose of using ready-to-use-tool, I looked at it and
>could not
>> find a way to achieve both 1) and 2). If you think we can do it, It
>would
>> be great to discuss the approach we can take. If we can agree that we
>only
>> publish pinned version, I would be super happy and switch to poetry
>> immediately - but we cannot do it I am afraid. There was a long
>discussion
>> about it and we cannot afford pinning by default, unfortunately. This
>has
>> already been discussed several times and unless this assumption has
>changed
>> - we cannot use poetry or similar.
>>
>> My 2ct.
>>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>
>--
>
>Jarek Potiuk
>Polidea <https://www.polidea.com/> | Principal Software Engineer
>
>M: +48 660 796 129 <+48660796129>
>[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Back to (some) dependency pinning
Posted by Jarek Potiuk <Ja...@polidea.com>.
Any other comments?
I'd love to hear your thoughts. It's the one thing that maybe not keeps me
from prod image, But I would love to know if I can rely on the
requirements.txt being part of the source code so that I can use it
when building the prod image..
J.
On Mon, Mar 16, 2020 at 12:16 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
>
> On Mon, Mar 16, 2020 at 11:16 AM Driesprong, Fokko <fo...@driesprong.frl>
> wrote:
>
>> Personally I don't like to have two versions in the PyPi repo. This also
>> complicates the releases, since we need to test, release and verify two
>> versions of Airflow
>>
>
> Not necessarily. Release is automated (and I run both package builds in CI
> now).
> Regarding testing - at the moment of release 'apache-airflow` and
> 'apache-airflow-pinned' are 1-1 equivalent. Their behaviour starts to be
> different when something out of our control happens with dependencies after
> the release. We can even have automated tests and install both
> 'apache-airflow' and 'apache-airlfow-pinned' at release time and compare
> the virtualenvs on binary level to verify that they are equivalent. This
> way we can just test one of them.
>
>
>> I'm afraid that this might confuse users.
>>
>
> I think that the current situation is much more confusing for the users.
> In order to install <1.10.8 for example you need to know that you should
> run 'pip install apache-airflow==1.10.8 Werkzeug<1.0'. If you want to
> install 1.10.2 you need to add other constraints. And tomorrow another
> constraint might be needed to install 1.10.9. This is also a hard blocker
> for the official, production image of Apache Airflow. One of the
> constraints to make an official Docker image is repeatability
> https://github.com/docker-library/official-images#repeatability . We
> don't have repeatability now.
>
> We need to consider what our alternatives are. I think our problem is that:
>
> 1) we want to have it open for operator development
> 2) we want the user (and official docker build) to have a reliable way to
> install the released version. This already happened and will happen that
> out of a sudden released version of airflow will stop installing without
> magic 'Werkzeug <= 1.0' or similar.
>
> We already have good solution for 1) with open dependencies . I see two
> solutions to that:
>
> *Solution 1) Ask the users to use requirements.txt as constraint file
> manually:*
>
> a) manually download requirements.txt from the right release in github
> b) pip install apache-airflow==1.10.9 --constraint requirements.txt
>
> Or
>
> *Solution 2) Use "-pinned" package (effectively does the same):*
>
> pip install apache-airflow-pinned==1.10.9
>
> I find the second one much less confusing and straightforward..
>
>
>
> I am all ears if we have any other solution? Can we discuss some potential
> options here?
>
>
> Besides that, it feels a bit like we're reinventing certain mechanisms that
>> are already in tooling like Poetry.
>>
>
> Unfortunately according to my checks - Poetry does not handle our case
> where we are both library and application at the same time. Form what I
> checked poetry uses .lock file and publishes the "pinned" version always in
> such case. Hacking it to support both cases would be super-difficult and
> defeat the purpose of using ready-to-use-tool, I looked at it and could not
> find a way to achieve both 1) and 2). If you think we can do it, It would
> be great to discuss the approach we can take. If we can agree that we only
> publish pinned version, I would be super happy and switch to poetry
> immediately - but we cannot do it I am afraid. There was a long discussion
> about it and we cannot afford pinning by default, unfortunately. This has
> already been discussed several times and unless this assumption has changed
> - we cannot use poetry or similar.
>
> My 2ct.
>>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Back to (some) dependency pinning
Posted by Jarek Potiuk <Ja...@polidea.com>.
On Mon, Mar 16, 2020 at 11:16 AM Driesprong, Fokko <fo...@driesprong.frl>
wrote:
> Personally I don't like to have two versions in the PyPi repo. This also
> complicates the releases, since we need to test, release and verify two
> versions of Airflow
>
Not necessarily. Release is automated (and I run both package builds in CI
now).
Regarding testing - at the moment of release 'apache-airflow` and
'apache-airflow-pinned' are 1-1 equivalent. Their behaviour starts to be
different when something out of our control happens with dependencies after
the release. We can even have automated tests and install both
'apache-airflow' and 'apache-airlfow-pinned' at release time and compare
the virtualenvs on binary level to verify that they are equivalent. This
way we can just test one of them.
> I'm afraid that this might confuse users.
>
I think that the current situation is much more confusing for the users. In
order to install <1.10.8 for example you need to know that you should run
'pip install apache-airflow==1.10.8 Werkzeug<1.0'. If you want to install
1.10.2 you need to add other constraints. And tomorrow another constraint
might be needed to install 1.10.9. This is also a hard blocker for the
official, production image of Apache Airflow. One of the constraints to
make an official Docker image is repeatability
https://github.com/docker-library/official-images#repeatability . We don't
have repeatability now.
We need to consider what our alternatives are. I think our problem is that:
1) we want to have it open for operator development
2) we want the user (and official docker build) to have a reliable way to
install the released version. This already happened and will happen that
out of a sudden released version of airflow will stop installing without
magic 'Werkzeug <= 1.0' or similar.
We already have good solution for 1) with open dependencies . I see two
solutions to that:
*Solution 1) Ask the users to use requirements.txt as constraint file
manually:*
a) manually download requirements.txt from the right release in github
b) pip install apache-airflow==1.10.9 --constraint requirements.txt
Or
*Solution 2) Use "-pinned" package (effectively does the same):*
pip install apache-airflow-pinned==1.10.9
I find the second one much less confusing and straightforward..
I am all ears if we have any other solution? Can we discuss some potential
options here?
Besides that, it feels a bit like we're reinventing certain mechanisms that
> are already in tooling like Poetry.
>
Unfortunately according to my checks - Poetry does not handle our case
where we are both library and application at the same time. Form what I
checked poetry uses .lock file and publishes the "pinned" version always in
such case. Hacking it to support both cases would be super-difficult and
defeat the purpose of using ready-to-use-tool, I looked at it and could not
find a way to achieve both 1) and 2). If you think we can do it, It would
be great to discuss the approach we can take. If we can agree that we only
publish pinned version, I would be super happy and switch to poetry
immediately - but we cannot do it I am afraid. There was a long discussion
about it and we cannot afford pinning by default, unfortunately. This has
already been discussed several times and unless this assumption has changed
- we cannot use poetry or similar.
My 2ct.
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Back to (some) dependency pinning
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Personally I don't like to have two versions in the PyPi repo. This also
complicates the releases, since we need to test, release and verify two
versions of Airflow. I'm afraid that this might confuse users.
Besides that, it feels a bit like we're reinventing certain mechanisms that
are already in tooling like Poetry.
My 2ct.
Cheers, Fokko
Op ma 16 mrt. 2020 om 11:04 schreef Jarek Potiuk <Ja...@polidea.com>:
> Does anyone think this one needs voting? I think it's an incremental change
> (but one that helps having stable installation mechanism on installing any
> released version of Airflow).
>
> J.
>
>
> On Sun, Mar 15, 2020 at 1:17 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
> > I think I have finally found a good and simple solution that I will also
> > be able to use in the production image.
> >
> > It boils down to adding "-apache-airflow-pinned" package additionally to
> > "apache-airflow". This package is the same as apache-airflow but it has
> > pinned dependencies (so it will never fail to install because of
> transitive
> > dependencies). The original 'apache-airflow' package is untouched - it
> > works as previously and sometimes it will fail when some transitive
> > dependencies will change.
> >
> > With this change we also have self-managed standard python
> > requirements.txt file (every time you update setup.py with dependencies,
> it
> > will be regenerated via pre-commit when you rebuild the breeze image).
> So
> > this will make it nice for IDE integration (a lot of ides will
> > automatically manage new requirements from requirements.txt file).
> >
> > It is designed to be self-managing.
> >
> > The PR is here: https://github.com/apache/airflow/pull/7730 - looking
> > forward to reviews and feedback.
> >
> > I also described it in CONTRIBUTING.rst so I just copy the description
> > here:
> >
> > Pinned Airflow version and requirements.txt file
> > <
> https://github.com/PolideaInternal/airflow/blob/AIRFLOW-7067-pinned-version-of-airflow/CONTRIBUTING.rst#id18
> >
> >
> > Airflow is not a standard python project. Most of the python projects
> fall
> > into one of two types - application or library. As described in [Should I
> > pin my python dependencies versions](
> >
> https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions
> )
> > decision whether to pin (freeze) requirements for a python project
> depdends
> > on the type. For applications, dependencies should be pinned, but for
> > libraries, they should be open.
> >
> > For application, pinning the dependencies makes it more stable to install
> > in the future - because new (even transitive) dependencies might cause
> > installation to fail. For libraries - the dependencies should be open to
> > allow several different libraries with the same requirements to be
> > installed at the same time.
> >
> > The problem is that Apache Airflow is a bit of both - application to
> > install and library to be used when you are developing your own operators
> > and DAGs.
> >
> > This - seemingly unsolvable - puzzle is solved as follows:
> >
> > - by default when you install apache-airflow package - the
> > dependencies are as open as possible while still allowing the
> > apache-airflow to install. This means that 'apache-airflow' package
> might
> > fail to install in case a direct or transitive dependency is released
> that
> > breaks the installation. In such case when installing apache-airflow,
> > you might need to provide additional constraints (for example pip
> > install apache-airflow==1.10.2 Werkzeug<1.0.0)
> > - we have requirements.txt file generated automatically based on the
> > set of all latest working and tested requirement versions. You can
> also use
> > that file as a constraints file when installing apache airflow -
> either
> > from the sources pip install -e . --constraint requirements.txt or
> > from the pypi package pip install apache-airflow --constraint
> > requirements.txt. Note that this will also work with extras for
> > example pip install .[gcp] --constraint requirements.txt or pip
> > install apache-airflow[gcp] --constraint requirements.txt
> > - we also release apache-airflow-pinned package which has all
> > dependencies frozen. This is guaranteed to work and be installable no
> > matter if some of the direct or transitive dependencies are released.
> If
> > you want to only install airflow and start using it - install
> > apache-airflow-pinned package.
> >
> > The requirements.txt file is maintained automatically as we update
> > dependencies and verified at CI time via pre-commit check/generation. It
> > reflects the current set of dependencies installed in the CI image of
> > Apache Airflow. The same set of requirements will be used to produce the
> > production image.
> >
> > J.
> >
> >
> > On Sat, Nov 30, 2019 at 1:33 PM Jarek Potiuk <Ja...@polidea.com>
> > wrote:
> >
> >> Just to update the plan and have some summary for those who did not read
> >> my TL;DR mail (and include probot integration). Here is a shorter
> summary
> >> of the proposal I have:
> >>
> >> 1. Keep the current process of *pip install -e[<extras>]* unchanged -
> >> taking into account the limits from setup.py
> >> 2. Add *requirements.txt* file with a fixed set of requirements
> >> (always *== -* including transitive deps). Those will be *managed by
> >> probot*, similarly to dependabot.
> >> 3. PRs in CI will use the requirements.txt file as a constraint file
> >> (known to work) when running tests to avoid transitive dependency
> problems
> >> 4. Daily CRON job installs all deps from scratch without
> *requirements.txt
> >> - * to detect early problems with current set of dependencies. An
> >> example of that from last night [1]: the CRON job failed and I
> >> could prepare the fix [2] while PRs of contributors continued to
> work.
> >> 5. Airflow installation for development will be guaranteed to work
> >> with *pip install -e . [<extras>] --constraints requirements.txt
> *from
> >> master - no matter the state of transitive dependencies
> >> 6. We will modify setup.py to also have a way to take the
> >> constraints, to guarantee installation works. Something like: *pip
> >> install apache-airflow==1.10.6[<extras>,constrained] *
> >>
> >> I'd love to hear more comments and ask you about opinion on that
> proposal.
> >>
> >> [1] Failed CRON job from last night - triggered likely by "from scratch"
> >> dependencies: https://travis-ci.org/apache/airflow/builds/618832805
> >> [2] Fix to the last night CRON job failure:
> >> https://github.com/apache/airflow/pull/6691
> >> [4] AIP-8 - Split Hooks/Operators to separate packages:
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827303
> >> [3] AIP-21 - Changes in import paths
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> >>
> >> *@Dan Davydow* -> Fully agree we should have something like that. I
> >> think this should be part of AIP-8 [3] when we get there. I hope we can
> >> learn a lot about those when we get AIP-21 [4] fully implemented and
> >> backporting packages generated and used for people as preparation for
> >> Airflow 2.0 migration (I hope it will happen soon - we are very close to
> >> have automation in place to move all the packages from contrib - worked
> on
> >> by Kamil Breguła). Having such (probot managed) set of requirements + a
> >> way to test it automatically is - I think - an important prerequisite to
> >> start designing it (and work out a reasonable approach for
> cross-dependency
> >> problems).
> >>
> >> J.
> >>
> >> On Mon, Nov 18, 2019 at 6:18 PM Dan Davydov
> <dd...@twitter.com.invalid>
> >> wrote:
> >>
> >>> My 2 cents on the long-term plan:
> >>> Once Airflow has Dag Isolation (i.e. DAG python dependencies are
> >>> completely
> >>> decoupled from Airflow), we should pin the core Airflow deps, and
> package
> >>> operators separately with version ranges (for the reasons Ash mentioned
> >>> about libraries vs applications).
> >>>
> >>> On Sat, Nov 16, 2019 at 6:22 AM Jarek Potiuk <Jarek.Potiuk@polidea.com
> >
> >>> wrote:
> >>>
> >>> > After some discussions and some tests by myself and Astronomer
> docker,
> >>> I
> >>> > think I have finally a complete proposal for good, consistent
> solution
> >>> for
> >>> > our dependency management.
> >>> >
> >>> > It uses well-known standards, does not introduce any new tools (like
> >>> > poetry) and I think it serves all the use cases we have very well.
> >>> >
> >>> > I wanted to summarise it here before I create it as an official AIP.
> I
> >>> will
> >>> > need to finalise the approach to complete the work on the official
> >>> > production image.
> >>> >
> >>> > Here are some basic assumptions:
> >>> >
> >>> > 1) We keep the current process of setup.py which is used to install
> >>> Airflow
> >>> > for development *'pip install -e .[extras]'*. It's nice and modular.
> >>> It has
> >>> > open-dependencies (some of them constrained due to compatibilities).
> >>> This
> >>> > will break from time to time for fresh install when some transitive
> >>> > dependencies are changed (or like with yesterday's pymssql drama).
> No
> >>> > changes whatsoever for people who continue installing it this way.
> The
> >>> same
> >>> > setup.py is used to install airflow in production (but see point 7
> >>> below
> >>> > for constrained builds).
> >>> >
> >>> > 2) On top of setup.py, we will have a standard *requirements.txt*
> file
> >>> as
> >>> > well. This will be a snapshot of current "working" dependencies
> >>> (including
> >>> > all transitive dependencies!). This file might be generated
> >>> automatically
> >>> > by pip freeze (with some scripting) and possibly put as part of
> >>> pre-commit
> >>> > checks to make sure it is self-maintainable when new dependencies are
> >>> > added. With the current Breeze + Pre-commit setup we can do it fully
> >>> > automatically, so that it will happen behind the scenes (and it will
> be
> >>> > verified in CI). This requirements.txt file will be a set of "KNOWN
> >>> TO BE
> >>> > GOOD" requirements. This requirements.txt file will have ONLY
> >>> > 'dependency==VERSION' - specific versions of each requirement . Nice
> >>> > side-effect of the requirements.txt file is that if you use IDEs like
> >>> > Pycharm, VSCode, it will automatically suggest installing
> >>> missing/outdated
> >>> > requirements in your virtualenv so you do not have to remember about
> >>> it.
> >>> >
> >>> > 3) By having *requirements.txt *we can add a dependabot
> >>> > <https://dependabot.com/> that will upgrade dependencies in this
> file
> >>> > automatically (it works in a way that it will create a PR every time
> >>> one of
> >>> > the dependencies is released). It works nicely with some of our
> other
> >>> > projects - see for example Oozie-2-Airflow here
> >>> > <
> >>> https://github.com/GoogleCloudPlatform/oozie-to-airflow/commits/master
> >
> >>> .
> >>> > It has the benefit that it prepares super-nice, isolated commits/PRs.
> >>> Those
> >>> > PRs (example here
> >>> > <https://github.com/GoogleCloudPlatform/oozie-to-airflow/pull/438>)
> >>> have
> >>> > all the information about changes and commits. Those are nice and
> >>> > actionable in terms of checking if we have breaking changes before
> >>> merging
> >>> > it (example below). The nice thing about dependabot is that it will
> >>> create
> >>> > a PR and our CI will test it automatically. So committers will have
> to
> >>> just
> >>> > double-check, take a look at the release notes and will be able to
> >>> merge it
> >>> > straight away. I will have to check if dependabot can be used for
> >>> Apache
> >>> > repos though (dependabot is owned by Github). It's an optional step
> >>> and we
> >>> > can do it manually though if we cannot use dependabot (but it
> requires
> >>> some
> >>> > more complex scripting).
> >>> >
> >>> > 4) We will use the requirements.txt file in CI in PRs so that we know
> >>> that
> >>> > CI always uses a "known to be good" set of requirements. Every PR
> will
> >>> have
> >>> > to have good set of requirements to be merged. No more transitive
> >>> > dependency changes.
> >>> >
> >>> > 5) We have a special CRON job run in CI daily and this job will use
> >>> '*pip
> >>> > install -e . [all]*' (not using the requirements.txt) - this way we
> >>> will
> >>> > have a mechanism to detect early problems with transitive
> dependencies
> >>> -
> >>> > without breaking PR builds.
> >>> >
> >>> > 6) We can use the requirements.txt file as constraints file as well
> >>> (!).
> >>> > For example you should be able to run '*pip install -e . [gcp]
> >>> > --constraints requirements.txt*' and it will install the "known to be
> >>> good"
> >>> > versions but only for those extras we want.
> >>> >
> >>> > 7) I believe I can customise setup.py to add extra "*constrained*"
> >>> extra
> >>> > which will also use the information from the requirements.txt. This
> >>> way the
> >>> > users will be able to do `*pip install
> >>> > apache-airflow==1.10.6[gcp,aws,constrained]*` and have "known to be
> >>> good"
> >>> > installation of Airflow - even if some transitive dependencies break
> >>> the
> >>> > "clean" install. I am not 100% sure if it is possible exactly this
> >>> way, but
> >>> > I think in the worst case it will be `*pip install
> >>> > apache-airflow=1.10.6[constrained,gcp-constrained,aws-constrained]*`
> -
> >>> we
> >>> > can generate a "-constrained" version of each of the extras
> >>> automatically
> >>> > based on the requirements.txt. I already had a POC for that. And then
> >>> it
> >>> > should be possible to upgrade each of the dependencies on its own.
> >>> >
> >>> > 8) Last but not least - we will use the requirements.txt file to
> build
> >>> the
> >>> > production image. One of the requirements for a good official image
> >>> > <
> >>> >
> >>>
> https://github.com/docker-library/official-images/blob/master/README.md#repeatability
> >>> > >
> >>> > is repeatability - which boils down to dependency-pinning. Currently
> we
> >>> > have no mechanism to enforce that. With requirements.txt this will be
> >>> > possible.
> >>> >
> >>> > Let me know what you think!
> >>> >
> >>> > J.
> >>> >
> >>> >
> >>> > Appendix: example dependabot-generated commit:
> >>> >
> >>> > Bump pylint from 2.4.3 to 2.4.4
> >>> >
> >>> > Bumps [pylint](https://github.com/PyCQA/pylint) from 2.4.3 to 2.4.4.
> >>> > - [Release notes](https://github.com/PyCQA/pylint/releases)
> >>> > - [Changelog](https://github.com/PyCQA/pylint/blob/master/ChangeLog)
> >>> > - [Commits](PyCQA/pylint@pylint-2.4.3...pylint-2.4.4
> >>> > <https://github.com/PyCQA/pylint/compare/pylint-2.4.3...pylint-2.4.4
> >)
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > On Fri, Aug 2, 2019 at 8:34 AM Felix Uellendall
> <feluelle@pm.me.invalid
> >>> >
> >>> > wrote:
> >>> >
> >>> > > I understand what Ash and Jarek are saying. I actually use airflow
> >>> with
> >>> > > custom plugins to have a end-product that fully satisfy our needs
> and
> >>> > when
> >>> > > writing new hooks and operators I don't want to see "airflow uses
> >>> > > requirement foo=1 but you have foo=2" - but actually that sometimes
> >>> also
> >>> > > just works. So why not let the developer decide if he want to risk
> >>> > breaking
> >>> > > it?
> >>> > > I like the idea of firstly defining groups for core and non-core
> >>> > > dependencies. I think only need to manage the core ones to always
> >>> work
> >>> > will
> >>> > > be a lot easier.
> >>> > > So I would suggest for core ones we have ranges including minor
> >>> releases
> >>> > > and non-core ones getting pinned by default installation (and ci).
> >>> > >
> >>> > > I think the more dependencies we add (which comes automatically as
> >>> > airflow
> >>> > > is growing) the more we are willing to set to more specific version
> >>> > ranges
> >>> > > and maybe even pin them at the end.
> >>> > >
> >>> > > Felix
> >>> > >
> >>> > > Sent from ProtonMail mobile
> >>> > >
> >>> > > -------- Original Message --------
> >>> > > On Aug 2, 2019, 06:10, Jarek Potiuk wrote:
> >>> > >
> >>> > > > Ash is totally right - that's exactly the difficulty we face.
> >>> Airflow
> >>> > is
> >>> > > > both a library and end product and this makes the usual advice
> >>> (pin if
> >>> > > you
> >>> > > > are end-product, don't pin if you are library) not really useful.
> >>> From
> >>> > > the
> >>> > > > very beginning of my adventures with Airflow I was for pinning of
> >>> > > > everything (and using dependabot or similar - I use it for other
> >>> > > projects),
> >>> > > > but over time I realised that this is very short-sighted
> approach.
> >>> it
> >>> > > does
> >>> > > > not take into account the "library" point of view.. Depending
> which
> >>> > user
> >>> > > of
> >>> > > > airflow you are, you have contradicting requirements. If you are
> >>> user
> >>> > of
> >>> > > > airflow who just want to use it as "end product" you want
> pinning.
> >>> If
> >>> > you
> >>> > > > want to develop your own operators or extend existing ones - you
> >>> use
> >>> > > > airflow as "library" and you do not want pinning.
> >>> > > >
> >>> > > > I also proposed at the beginning of that thread that we split
> core
> >>> > > > requirements (and pin it) and non-core ones (and don't pin it).
> >>> But it
> >>> > > > ain't easy to separate those two sets in a clear way
> unfortunately.
> >>> > > >
> >>> > > > That's why the idea of choosing at the installation time (and not
> >>> at
> >>> > > build
> >>> > > > time) whether you want to install "loose" or "frozen"
> dependencies
> >>> is
> >>> > so
> >>> > > > appealing.
> >>> > > > Possibly the best solution could be that you 'pip install
> airflow'
> >>> and
> >>> > > you
> >>> > > > get the pinned versions and then some other way to get the loose
> >>> one.
> >>> > > But I
> >>> > > > think we are a bit on the mercy of pip - this does not seem to be
> >>> > > possible.
> >>> > > >
> >>> > > > Then - it looks like using extras to add "pinning" mechanism is
> the
> >>> > next
> >>> > > > best idea.
> >>> > > >
> >>> > > > I am not afraid about complexity. We can fully automate
> generating
> >>> > those
> >>> > > > pinned requirements. I already have some ideas how we can make
> sure
> >>> > that
> >>> > > we
> >>> > > > keep those requirements in sync while developing and how they can
> >>> end
> >>> > up
> >>> > > > frozen in release. I would like to run a POC on that but in short
> >>> it is
> >>> > > > another "by-product" of the CI image we have now. Our CI image is
> >>> the
> >>> > > > perfect source of frozen requirements - we know those
> requirements
> >>> in
> >>> > CI
> >>> > > > image are ok and we can use them to generate the standard
> >>> > > > "requirements.txt" file and keep it updated via some local update
> >>> > script
> >>> > > > (and pre-commit hooks) + we can verify that they are updated in
> >>> the CI.
> >>> > > We
> >>> > > > can then write custom setup.py that will use that
> requirements.txt
> >>> and
> >>> > > the
> >>> > > > existing "extras" and generate "pinned" extras automatically.
> That
> >>> > sounds
> >>> > > > like fully doable and with very limited maintenance effort.
> >>> > > >
> >>> > > > J.
> >>> > > >
> >>> > > > On Thu, Aug 1, [2019](tel:2019) at 10:45 PM Qingping Hou <
> >>> > qph@scribd.com>
> >>> > > wrote:
> >>> > > >
> >>> > > >> On Thu, Aug 1, [2019](tel:2019) at 1:33 PM Chen Tong <
> >>> > cixuuz@gmail.com>
> >>> > > wrote:
> >>> > > >> > It is sometimes hard to distinguish if it is a library or an
> >>> > > application.
> >>> > > >> > Take operator as an example, a non-tech people may think it
> as a
> >>> > > >> well-built
> >>> > > >> > application while an engineer may consider it as a library and
> >>> > extends
> >>> > > >> > functionalities on it.
> >>> > > >> >
> >>> > > >>
> >>> > > >> Yeah, I agree. Personally, I would consider operator to be a
> >>> library
> >>> > > >> due to the expectation that other people will import them in
> >>> their own
> >>> > > >> projects/source tree.
> >>> > > >>
> >>> > > >> For things like REST endpoint handlers and perhaps scheduler, it
> >>> seems
> >>> > > >> safe to assume all changes and improvements will happen within
> >>> Airflow
> >>> > > >> source tree. In that case, it's safe to classify that part of
> >>> code as
> >>> > > >> application and freeze all its dependencies. The current plugin
> >>> system
> >>> > > >> might make this slightly complicated because people can still
> >>> extend
> >>> > > >> the core with custom code. But again, in an ideal world, plugins
> >>> > > >> should be self-contained and communicating with core through a
> >>> well
> >>> > > >> defined interface ;)
> >>> > > >>
> >>> > > >
> >>> > > > --
> >>> > > >
> >>> > > > Jarek Potiuk
> >>> > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>> > > >
> >>> > > > M: [+48 660 796 129](tel:+48660796129)
> >>> > <[+48660796129](tel:+48660796129)>
> >>> > > > [image: Polidea] <https://www.polidea.com/>
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> >
> >>> > Jarek Potiuk
> >>> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>> >
> >>> > M: +48 660 796 129 <+48660796129>
> >>> > [image: Polidea] <https://www.polidea.com/>
> >>> >
> >>>
> >>
> >>
> >> --
> >>
> >> Jarek Potiuk
> >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>
> >> M: +48 660 796 129 <+48660796129>
> >> [image: Polidea] <https://www.polidea.com/>
> >>
> >>
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
> >
> >
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
Re: [DISCUSS] Back to (some) dependency pinning
Posted by Jarek Potiuk <Ja...@polidea.com>.
Does anyone think this one needs voting? I think it's an incremental change
(but one that helps having stable installation mechanism on installing any
released version of Airflow).
J.
On Sun, Mar 15, 2020 at 1:17 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> I think I have finally found a good and simple solution that I will also
> be able to use in the production image.
>
> It boils down to adding "-apache-airflow-pinned" package additionally to
> "apache-airflow". This package is the same as apache-airflow but it has
> pinned dependencies (so it will never fail to install because of transitive
> dependencies). The original 'apache-airflow' package is untouched - it
> works as previously and sometimes it will fail when some transitive
> dependencies will change.
>
> With this change we also have self-managed standard python
> requirements.txt file (every time you update setup.py with dependencies, it
> will be regenerated via pre-commit when you rebuild the breeze image). So
> this will make it nice for IDE integration (a lot of ides will
> automatically manage new requirements from requirements.txt file).
>
> It is designed to be self-managing.
>
> The PR is here: https://github.com/apache/airflow/pull/7730 - looking
> forward to reviews and feedback.
>
> I also described it in CONTRIBUTING.rst so I just copy the description
> here:
>
> Pinned Airflow version and requirements.txt file
> <https://github.com/PolideaInternal/airflow/blob/AIRFLOW-7067-pinned-version-of-airflow/CONTRIBUTING.rst#id18>
>
> Airflow is not a standard python project. Most of the python projects fall
> into one of two types - application or library. As described in [Should I
> pin my python dependencies versions](
> https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions)
> decision whether to pin (freeze) requirements for a python project depdends
> on the type. For applications, dependencies should be pinned, but for
> libraries, they should be open.
>
> For application, pinning the dependencies makes it more stable to install
> in the future - because new (even transitive) dependencies might cause
> installation to fail. For libraries - the dependencies should be open to
> allow several different libraries with the same requirements to be
> installed at the same time.
>
> The problem is that Apache Airflow is a bit of both - application to
> install and library to be used when you are developing your own operators
> and DAGs.
>
> This - seemingly unsolvable - puzzle is solved as follows:
>
> - by default when you install apache-airflow package - the
> dependencies are as open as possible while still allowing the
> apache-airflow to install. This means that 'apache-airflow' package might
> fail to install in case a direct or transitive dependency is released that
> breaks the installation. In such case when installing apache-airflow,
> you might need to provide additional constraints (for example pip
> install apache-airflow==1.10.2 Werkzeug<1.0.0)
> - we have requirements.txt file generated automatically based on the
> set of all latest working and tested requirement versions. You can also use
> that file as a constraints file when installing apache airflow - either
> from the sources pip install -e . --constraint requirements.txt or
> from the pypi package pip install apache-airflow --constraint
> requirements.txt. Note that this will also work with extras for
> example pip install .[gcp] --constraint requirements.txt or pip
> install apache-airflow[gcp] --constraint requirements.txt
> - we also release apache-airflow-pinned package which has all
> dependencies frozen. This is guaranteed to work and be installable no
> matter if some of the direct or transitive dependencies are released. If
> you want to only install airflow and start using it - install
> apache-airflow-pinned package.
>
> The requirements.txt file is maintained automatically as we update
> dependencies and verified at CI time via pre-commit check/generation. It
> reflects the current set of dependencies installed in the CI image of
> Apache Airflow. The same set of requirements will be used to produce the
> production image.
>
> J.
>
>
> On Sat, Nov 30, 2019 at 1:33 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> Just to update the plan and have some summary for those who did not read
>> my TL;DR mail (and include probot integration). Here is a shorter summary
>> of the proposal I have:
>>
>> 1. Keep the current process of *pip install -e[<extras>]* unchanged -
>> taking into account the limits from setup.py
>> 2. Add *requirements.txt* file with a fixed set of requirements
>> (always *== -* including transitive deps). Those will be *managed by
>> probot*, similarly to dependabot.
>> 3. PRs in CI will use the requirements.txt file as a constraint file
>> (known to work) when running tests to avoid transitive dependency problems
>> 4. Daily CRON job installs all deps from scratch without *requirements.txt
>> - * to detect early problems with current set of dependencies. An
>> example of that from last night [1]: the CRON job failed and I
>> could prepare the fix [2] while PRs of contributors continued to work.
>> 5. Airflow installation for development will be guaranteed to work
>> with *pip install -e . [<extras>] --constraints requirements.txt *from
>> master - no matter the state of transitive dependencies
>> 6. We will modify setup.py to also have a way to take the
>> constraints, to guarantee installation works. Something like: *pip
>> install apache-airflow==1.10.6[<extras>,constrained] *
>>
>> I'd love to hear more comments and ask you about opinion on that proposal.
>>
>> [1] Failed CRON job from last night - triggered likely by "from scratch"
>> dependencies: https://travis-ci.org/apache/airflow/builds/618832805
>> [2] Fix to the last night CRON job failure:
>> https://github.com/apache/airflow/pull/6691
>> [4] AIP-8 - Split Hooks/Operators to separate packages:
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827303
>> [3] AIP-21 - Changes in import paths
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
>>
>> *@Dan Davydow* -> Fully agree we should have something like that. I
>> think this should be part of AIP-8 [3] when we get there. I hope we can
>> learn a lot about those when we get AIP-21 [4] fully implemented and
>> backporting packages generated and used for people as preparation for
>> Airflow 2.0 migration (I hope it will happen soon - we are very close to
>> have automation in place to move all the packages from contrib - worked on
>> by Kamil Breguła). Having such (probot managed) set of requirements + a
>> way to test it automatically is - I think - an important prerequisite to
>> start designing it (and work out a reasonable approach for cross-dependency
>> problems).
>>
>> J.
>>
>> On Mon, Nov 18, 2019 at 6:18 PM Dan Davydov <dd...@twitter.com.invalid>
>> wrote:
>>
>>> My 2 cents on the long-term plan:
>>> Once Airflow has Dag Isolation (i.e. DAG python dependencies are
>>> completely
>>> decoupled from Airflow), we should pin the core Airflow deps, and package
>>> operators separately with version ranges (for the reasons Ash mentioned
>>> about libraries vs applications).
>>>
>>> On Sat, Nov 16, 2019 at 6:22 AM Jarek Potiuk <Ja...@polidea.com>
>>> wrote:
>>>
>>> > After some discussions and some tests by myself and Astronomer docker,
>>> I
>>> > think I have finally a complete proposal for good, consistent solution
>>> for
>>> > our dependency management.
>>> >
>>> > It uses well-known standards, does not introduce any new tools (like
>>> > poetry) and I think it serves all the use cases we have very well.
>>> >
>>> > I wanted to summarise it here before I create it as an official AIP. I
>>> will
>>> > need to finalise the approach to complete the work on the official
>>> > production image.
>>> >
>>> > Here are some basic assumptions:
>>> >
>>> > 1) We keep the current process of setup.py which is used to install
>>> Airflow
>>> > for development *'pip install -e .[extras]'*. It's nice and modular.
>>> It has
>>> > open-dependencies (some of them constrained due to compatibilities).
>>> This
>>> > will break from time to time for fresh install when some transitive
>>> > dependencies are changed (or like with yesterday's pymssql drama). No
>>> > changes whatsoever for people who continue installing it this way. The
>>> same
>>> > setup.py is used to install airflow in production (but see point 7
>>> below
>>> > for constrained builds).
>>> >
>>> > 2) On top of setup.py, we will have a standard *requirements.txt* file
>>> as
>>> > well. This will be a snapshot of current "working" dependencies
>>> (including
>>> > all transitive dependencies!). This file might be generated
>>> automatically
>>> > by pip freeze (with some scripting) and possibly put as part of
>>> pre-commit
>>> > checks to make sure it is self-maintainable when new dependencies are
>>> > added. With the current Breeze + Pre-commit setup we can do it fully
>>> > automatically, so that it will happen behind the scenes (and it will be
>>> > verified in CI). This requirements.txt file will be a set of "KNOWN
>>> TO BE
>>> > GOOD" requirements. This requirements.txt file will have ONLY
>>> > 'dependency==VERSION' - specific versions of each requirement . Nice
>>> > side-effect of the requirements.txt file is that if you use IDEs like
>>> > Pycharm, VSCode, it will automatically suggest installing
>>> missing/outdated
>>> > requirements in your virtualenv so you do not have to remember about
>>> it.
>>> >
>>> > 3) By having *requirements.txt *we can add a dependabot
>>> > <https://dependabot.com/> that will upgrade dependencies in this file
>>> > automatically (it works in a way that it will create a PR every time
>>> one of
>>> > the dependencies is released). It works nicely with some of our other
>>> > projects - see for example Oozie-2-Airflow here
>>> > <
>>> https://github.com/GoogleCloudPlatform/oozie-to-airflow/commits/master>
>>> .
>>> > It has the benefit that it prepares super-nice, isolated commits/PRs.
>>> Those
>>> > PRs (example here
>>> > <https://github.com/GoogleCloudPlatform/oozie-to-airflow/pull/438>)
>>> have
>>> > all the information about changes and commits. Those are nice and
>>> > actionable in terms of checking if we have breaking changes before
>>> merging
>>> > it (example below). The nice thing about dependabot is that it will
>>> create
>>> > a PR and our CI will test it automatically. So committers will have to
>>> just
>>> > double-check, take a look at the release notes and will be able to
>>> merge it
>>> > straight away. I will have to check if dependabot can be used for
>>> Apache
>>> > repos though (dependabot is owned by Github). It's an optional step
>>> and we
>>> > can do it manually though if we cannot use dependabot (but it requires
>>> some
>>> > more complex scripting).
>>> >
>>> > 4) We will use the requirements.txt file in CI in PRs so that we know
>>> that
>>> > CI always uses a "known to be good" set of requirements. Every PR will
>>> have
>>> > to have good set of requirements to be merged. No more transitive
>>> > dependency changes.
>>> >
>>> > 5) We have a special CRON job run in CI daily and this job will use
>>> '*pip
>>> > install -e . [all]*' (not using the requirements.txt) - this way we
>>> will
>>> > have a mechanism to detect early problems with transitive dependencies
>>> -
>>> > without breaking PR builds.
>>> >
>>> > 6) We can use the requirements.txt file as constraints file as well
>>> (!).
>>> > For example you should be able to run '*pip install -e . [gcp]
>>> > --constraints requirements.txt*' and it will install the "known to be
>>> good"
>>> > versions but only for those extras we want.
>>> >
>>> > 7) I believe I can customise setup.py to add extra "*constrained*"
>>> extra
>>> > which will also use the information from the requirements.txt. This
>>> way the
>>> > users will be able to do `*pip install
>>> > apache-airflow==1.10.6[gcp,aws,constrained]*` and have "known to be
>>> good"
>>> > installation of Airflow - even if some transitive dependencies break
>>> the
>>> > "clean" install. I am not 100% sure if it is possible exactly this
>>> way, but
>>> > I think in the worst case it will be `*pip install
>>> > apache-airflow=1.10.6[constrained,gcp-constrained,aws-constrained]*` -
>>> we
>>> > can generate a "-constrained" version of each of the extras
>>> automatically
>>> > based on the requirements.txt. I already had a POC for that. And then
>>> it
>>> > should be possible to upgrade each of the dependencies on its own.
>>> >
>>> > 8) Last but not least - we will use the requirements.txt file to build
>>> the
>>> > production image. One of the requirements for a good official image
>>> > <
>>> >
>>> https://github.com/docker-library/official-images/blob/master/README.md#repeatability
>>> > >
>>> > is repeatability - which boils down to dependency-pinning. Currently we
>>> > have no mechanism to enforce that. With requirements.txt this will be
>>> > possible.
>>> >
>>> > Let me know what you think!
>>> >
>>> > J.
>>> >
>>> >
>>> > Appendix: example dependabot-generated commit:
>>> >
>>> > Bump pylint from 2.4.3 to 2.4.4
>>> >
>>> > Bumps [pylint](https://github.com/PyCQA/pylint) from 2.4.3 to 2.4.4.
>>> > - [Release notes](https://github.com/PyCQA/pylint/releases)
>>> > - [Changelog](https://github.com/PyCQA/pylint/blob/master/ChangeLog)
>>> > - [Commits](PyCQA/pylint@pylint-2.4.3...pylint-2.4.4
>>> > <https://github.com/PyCQA/pylint/compare/pylint-2.4.3...pylint-2.4.4>)
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Fri, Aug 2, 2019 at 8:34 AM Felix Uellendall <feluelle@pm.me.invalid
>>> >
>>> > wrote:
>>> >
>>> > > I understand what Ash and Jarek are saying. I actually use airflow
>>> with
>>> > > custom plugins to have a end-product that fully satisfy our needs and
>>> > when
>>> > > writing new hooks and operators I don't want to see "airflow uses
>>> > > requirement foo=1 but you have foo=2" - but actually that sometimes
>>> also
>>> > > just works. So why not let the developer decide if he want to risk
>>> > breaking
>>> > > it?
>>> > > I like the idea of firstly defining groups for core and non-core
>>> > > dependencies. I think only need to manage the core ones to always
>>> work
>>> > will
>>> > > be a lot easier.
>>> > > So I would suggest for core ones we have ranges including minor
>>> releases
>>> > > and non-core ones getting pinned by default installation (and ci).
>>> > >
>>> > > I think the more dependencies we add (which comes automatically as
>>> > airflow
>>> > > is growing) the more we are willing to set to more specific version
>>> > ranges
>>> > > and maybe even pin them at the end.
>>> > >
>>> > > Felix
>>> > >
>>> > > Sent from ProtonMail mobile
>>> > >
>>> > > -------- Original Message --------
>>> > > On Aug 2, 2019, 06:10, Jarek Potiuk wrote:
>>> > >
>>> > > > Ash is totally right - that's exactly the difficulty we face.
>>> Airflow
>>> > is
>>> > > > both a library and end product and this makes the usual advice
>>> (pin if
>>> > > you
>>> > > > are end-product, don't pin if you are library) not really useful.
>>> From
>>> > > the
>>> > > > very beginning of my adventures with Airflow I was for pinning of
>>> > > > everything (and using dependabot or similar - I use it for other
>>> > > projects),
>>> > > > but over time I realised that this is very short-sighted approach.
>>> it
>>> > > does
>>> > > > not take into account the "library" point of view.. Depending which
>>> > user
>>> > > of
>>> > > > airflow you are, you have contradicting requirements. If you are
>>> user
>>> > of
>>> > > > airflow who just want to use it as "end product" you want pinning.
>>> If
>>> > you
>>> > > > want to develop your own operators or extend existing ones - you
>>> use
>>> > > > airflow as "library" and you do not want pinning.
>>> > > >
>>> > > > I also proposed at the beginning of that thread that we split core
>>> > > > requirements (and pin it) and non-core ones (and don't pin it).
>>> But it
>>> > > > ain't easy to separate those two sets in a clear way unfortunately.
>>> > > >
>>> > > > That's why the idea of choosing at the installation time (and not
>>> at
>>> > > build
>>> > > > time) whether you want to install "loose" or "frozen" dependencies
>>> is
>>> > so
>>> > > > appealing.
>>> > > > Possibly the best solution could be that you 'pip install airflow'
>>> and
>>> > > you
>>> > > > get the pinned versions and then some other way to get the loose
>>> one.
>>> > > But I
>>> > > > think we are a bit on the mercy of pip - this does not seem to be
>>> > > possible.
>>> > > >
>>> > > > Then - it looks like using extras to add "pinning" mechanism is the
>>> > next
>>> > > > best idea.
>>> > > >
>>> > > > I am not afraid about complexity. We can fully automate generating
>>> > those
>>> > > > pinned requirements. I already have some ideas how we can make sure
>>> > that
>>> > > we
>>> > > > keep those requirements in sync while developing and how they can
>>> end
>>> > up
>>> > > > frozen in release. I would like to run a POC on that but in short
>>> it is
>>> > > > another "by-product" of the CI image we have now. Our CI image is
>>> the
>>> > > > perfect source of frozen requirements - we know those requirements
>>> in
>>> > CI
>>> > > > image are ok and we can use them to generate the standard
>>> > > > "requirements.txt" file and keep it updated via some local update
>>> > script
>>> > > > (and pre-commit hooks) + we can verify that they are updated in
>>> the CI.
>>> > > We
>>> > > > can then write custom setup.py that will use that requirements.txt
>>> and
>>> > > the
>>> > > > existing "extras" and generate "pinned" extras automatically. That
>>> > sounds
>>> > > > like fully doable and with very limited maintenance effort.
>>> > > >
>>> > > > J.
>>> > > >
>>> > > > On Thu, Aug 1, [2019](tel:2019) at 10:45 PM Qingping Hou <
>>> > qph@scribd.com>
>>> > > wrote:
>>> > > >
>>> > > >> On Thu, Aug 1, [2019](tel:2019) at 1:33 PM Chen Tong <
>>> > cixuuz@gmail.com>
>>> > > wrote:
>>> > > >> > It is sometimes hard to distinguish if it is a library or an
>>> > > application.
>>> > > >> > Take operator as an example, a non-tech people may think it as a
>>> > > >> well-built
>>> > > >> > application while an engineer may consider it as a library and
>>> > extends
>>> > > >> > functionalities on it.
>>> > > >> >
>>> > > >>
>>> > > >> Yeah, I agree. Personally, I would consider operator to be a
>>> library
>>> > > >> due to the expectation that other people will import them in
>>> their own
>>> > > >> projects/source tree.
>>> > > >>
>>> > > >> For things like REST endpoint handlers and perhaps scheduler, it
>>> seems
>>> > > >> safe to assume all changes and improvements will happen within
>>> Airflow
>>> > > >> source tree. In that case, it's safe to classify that part of
>>> code as
>>> > > >> application and freeze all its dependencies. The current plugin
>>> system
>>> > > >> might make this slightly complicated because people can still
>>> extend
>>> > > >> the core with custom code. But again, in an ideal world, plugins
>>> > > >> should be self-contained and communicating with core through a
>>> well
>>> > > >> defined interface ;)
>>> > > >>
>>> > > >
>>> > > > --
>>> > > >
>>> > > > Jarek Potiuk
>>> > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
>>> > > >
>>> > > > M: [+48 660 796 129](tel:+48660796129)
>>> > <[+48660796129](tel:+48660796129)>
>>> > > > [image: Polidea] <https://www.polidea.com/>
>>> >
>>> >
>>> >
>>> > --
>>> >
>>> > Jarek Potiuk
>>> > Polidea <https://www.polidea.com/> | Principal Software Engineer
>>> >
>>> > M: +48 660 796 129 <+48660796129>
>>> > [image: Polidea] <https://www.polidea.com/>
>>> >
>>>
>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Back to (some) dependency pinning
Posted by Jarek Potiuk <Ja...@polidea.com>.
I think I have finally found a good and simple solution that I will also be
able to use in the production image.
It boils down to adding "-apache-airflow-pinned" package additionally to
"apache-airflow". This package is the same as apache-airflow but it has
pinned dependencies (so it will never fail to install because of transitive
dependencies). The original 'apache-airflow' package is untouched - it
works as previously and sometimes it will fail when some transitive
dependencies will change.
With this change we also have self-managed standard python requirements.txt
file (every time you update setup.py with dependencies, it will be
regenerated via pre-commit when you rebuild the breeze image). So this
will make it nice for IDE integration (a lot of ides will automatically
manage new requirements from requirements.txt file).
It is designed to be self-managing.
The PR is here: https://github.com/apache/airflow/pull/7730 - looking
forward to reviews and feedback.
I also described it in CONTRIBUTING.rst so I just copy the description here:
Pinned Airflow version and requirements.txt file
<https://github.com/PolideaInternal/airflow/blob/AIRFLOW-7067-pinned-version-of-airflow/CONTRIBUTING.rst#id18>
Airflow is not a standard python project. Most of the python projects fall
into one of two types - application or library. As described in [Should I
pin my python dependencies versions](
https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions)
decision whether to pin (freeze) requirements for a python project depdends
on the type. For applications, dependencies should be pinned, but for
libraries, they should be open.
For application, pinning the dependencies makes it more stable to install
in the future - because new (even transitive) dependencies might cause
installation to fail. For libraries - the dependencies should be open to
allow several different libraries with the same requirements to be
installed at the same time.
The problem is that Apache Airflow is a bit of both - application to
install and library to be used when you are developing your own operators
and DAGs.
This - seemingly unsolvable - puzzle is solved as follows:
- by default when you install apache-airflow package - the dependencies
are as open as possible while still allowing the apache-airflow to install.
This means that 'apache-airflow' package might fail to install in case a
direct or transitive dependency is released that breaks the installation.
In such case when installing apache-airflow, you might need to provide
additional constraints (for example pip install apache-airflow==1.10.2
Werkzeug<1.0.0)
- we have requirements.txt file generated automatically based on the set
of all latest working and tested requirement versions. You can also use
that file as a constraints file when installing apache airflow - either
from the sources pip install -e . --constraint requirements.txt or from
the pypi package pip install apache-airflow --constraint requirements.txt.
Note that this will also work with extras for example pip install .[gcp]
--constraint requirements.txt or pip install apache-airflow[gcp]
--constraint requirements.txt
- we also release apache-airflow-pinned package which has all
dependencies frozen. This is guaranteed to work and be installable no
matter if some of the direct or transitive dependencies are released. If
you want to only install airflow and start using it - install
apache-airflow-pinned package.
The requirements.txt file is maintained automatically as we update
dependencies and verified at CI time via pre-commit check/generation. It
reflects the current set of dependencies installed in the CI image of
Apache Airflow. The same set of requirements will be used to produce the
production image.
J.
On Sat, Nov 30, 2019 at 1:33 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Just to update the plan and have some summary for those who did not read
> my TL;DR mail (and include probot integration). Here is a shorter summary
> of the proposal I have:
>
> 1. Keep the current process of *pip install -e[<extras>]* unchanged -
> taking into account the limits from setup.py
> 2. Add *requirements.txt* file with a fixed set of requirements
> (always *== -* including transitive deps). Those will be *managed by
> probot*, similarly to dependabot.
> 3. PRs in CI will use the requirements.txt file as a constraint file
> (known to work) when running tests to avoid transitive dependency problems
> 4. Daily CRON job installs all deps from scratch without *requirements.txt
> - * to detect early problems with current set of dependencies. An
> example of that from last night [1]: the CRON job failed and I
> could prepare the fix [2] while PRs of contributors continued to work.
> 5. Airflow installation for development will be guaranteed to work
> with *pip install -e . [<extras>] --constraints requirements.txt *from
> master - no matter the state of transitive dependencies
> 6. We will modify setup.py to also have a way to take the constraints,
> to guarantee installation works. Something like: *pip install
> apache-airflow==1.10.6[<extras>,constrained] *
>
> I'd love to hear more comments and ask you about opinion on that proposal.
>
> [1] Failed CRON job from last night - triggered likely by "from scratch"
> dependencies: https://travis-ci.org/apache/airflow/builds/618832805
> [2] Fix to the last night CRON job failure:
> https://github.com/apache/airflow/pull/6691
> [4] AIP-8 - Split Hooks/Operators to separate packages:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827303
> [3] AIP-21 - Changes in import paths
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
>
> *@Dan Davydow* -> Fully agree we should have something like that. I think
> this should be part of AIP-8 [3] when we get there. I hope we can learn a
> lot about those when we get AIP-21 [4] fully implemented and backporting
> packages generated and used for people as preparation for Airflow 2.0
> migration (I hope it will happen soon - we are very close to have
> automation in place to move all the packages from contrib - worked on by
> Kamil Breguła). Having such (probot managed) set of requirements + a way
> to test it automatically is - I think - an important prerequisite to start
> designing it (and work out a reasonable approach for cross-dependency
> problems).
>
> J.
>
> On Mon, Nov 18, 2019 at 6:18 PM Dan Davydov <dd...@twitter.com.invalid>
> wrote:
>
>> My 2 cents on the long-term plan:
>> Once Airflow has Dag Isolation (i.e. DAG python dependencies are
>> completely
>> decoupled from Airflow), we should pin the core Airflow deps, and package
>> operators separately with version ranges (for the reasons Ash mentioned
>> about libraries vs applications).
>>
>> On Sat, Nov 16, 2019 at 6:22 AM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>> > After some discussions and some tests by myself and Astronomer docker, I
>> > think I have finally a complete proposal for good, consistent solution
>> for
>> > our dependency management.
>> >
>> > It uses well-known standards, does not introduce any new tools (like
>> > poetry) and I think it serves all the use cases we have very well.
>> >
>> > I wanted to summarise it here before I create it as an official AIP. I
>> will
>> > need to finalise the approach to complete the work on the official
>> > production image.
>> >
>> > Here are some basic assumptions:
>> >
>> > 1) We keep the current process of setup.py which is used to install
>> Airflow
>> > for development *'pip install -e .[extras]'*. It's nice and modular. It
>> has
>> > open-dependencies (some of them constrained due to compatibilities).
>> This
>> > will break from time to time for fresh install when some transitive
>> > dependencies are changed (or like with yesterday's pymssql drama). No
>> > changes whatsoever for people who continue installing it this way. The
>> same
>> > setup.py is used to install airflow in production (but see point 7 below
>> > for constrained builds).
>> >
>> > 2) On top of setup.py, we will have a standard *requirements.txt* file
>> as
>> > well. This will be a snapshot of current "working" dependencies
>> (including
>> > all transitive dependencies!). This file might be generated
>> automatically
>> > by pip freeze (with some scripting) and possibly put as part of
>> pre-commit
>> > checks to make sure it is self-maintainable when new dependencies are
>> > added. With the current Breeze + Pre-commit setup we can do it fully
>> > automatically, so that it will happen behind the scenes (and it will be
>> > verified in CI). This requirements.txt file will be a set of "KNOWN TO
>> BE
>> > GOOD" requirements. This requirements.txt file will have ONLY
>> > 'dependency==VERSION' - specific versions of each requirement . Nice
>> > side-effect of the requirements.txt file is that if you use IDEs like
>> > Pycharm, VSCode, it will automatically suggest installing
>> missing/outdated
>> > requirements in your virtualenv so you do not have to remember about it.
>> >
>> > 3) By having *requirements.txt *we can add a dependabot
>> > <https://dependabot.com/> that will upgrade dependencies in this file
>> > automatically (it works in a way that it will create a PR every time
>> one of
>> > the dependencies is released). It works nicely with some of our other
>> > projects - see for example Oozie-2-Airflow here
>> > <https://github.com/GoogleCloudPlatform/oozie-to-airflow/commits/master>
>> .
>> > It has the benefit that it prepares super-nice, isolated commits/PRs.
>> Those
>> > PRs (example here
>> > <https://github.com/GoogleCloudPlatform/oozie-to-airflow/pull/438>)
>> have
>> > all the information about changes and commits. Those are nice and
>> > actionable in terms of checking if we have breaking changes before
>> merging
>> > it (example below). The nice thing about dependabot is that it will
>> create
>> > a PR and our CI will test it automatically. So committers will have to
>> just
>> > double-check, take a look at the release notes and will be able to
>> merge it
>> > straight away. I will have to check if dependabot can be used for Apache
>> > repos though (dependabot is owned by Github). It's an optional step and
>> we
>> > can do it manually though if we cannot use dependabot (but it requires
>> some
>> > more complex scripting).
>> >
>> > 4) We will use the requirements.txt file in CI in PRs so that we know
>> that
>> > CI always uses a "known to be good" set of requirements. Every PR will
>> have
>> > to have good set of requirements to be merged. No more transitive
>> > dependency changes.
>> >
>> > 5) We have a special CRON job run in CI daily and this job will use
>> '*pip
>> > install -e . [all]*' (not using the requirements.txt) - this way we
>> will
>> > have a mechanism to detect early problems with transitive dependencies -
>> > without breaking PR builds.
>> >
>> > 6) We can use the requirements.txt file as constraints file as well (!).
>> > For example you should be able to run '*pip install -e . [gcp]
>> > --constraints requirements.txt*' and it will install the "known to be
>> good"
>> > versions but only for those extras we want.
>> >
>> > 7) I believe I can customise setup.py to add extra "*constrained*" extra
>> > which will also use the information from the requirements.txt. This way
>> the
>> > users will be able to do `*pip install
>> > apache-airflow==1.10.6[gcp,aws,constrained]*` and have "known to be
>> good"
>> > installation of Airflow - even if some transitive dependencies break the
>> > "clean" install. I am not 100% sure if it is possible exactly this way,
>> but
>> > I think in the worst case it will be `*pip install
>> > apache-airflow=1.10.6[constrained,gcp-constrained,aws-constrained]*` -
>> we
>> > can generate a "-constrained" version of each of the extras
>> automatically
>> > based on the requirements.txt. I already had a POC for that. And then it
>> > should be possible to upgrade each of the dependencies on its own.
>> >
>> > 8) Last but not least - we will use the requirements.txt file to build
>> the
>> > production image. One of the requirements for a good official image
>> > <
>> >
>> https://github.com/docker-library/official-images/blob/master/README.md#repeatability
>> > >
>> > is repeatability - which boils down to dependency-pinning. Currently we
>> > have no mechanism to enforce that. With requirements.txt this will be
>> > possible.
>> >
>> > Let me know what you think!
>> >
>> > J.
>> >
>> >
>> > Appendix: example dependabot-generated commit:
>> >
>> > Bump pylint from 2.4.3 to 2.4.4
>> >
>> > Bumps [pylint](https://github.com/PyCQA/pylint) from 2.4.3 to 2.4.4.
>> > - [Release notes](https://github.com/PyCQA/pylint/releases)
>> > - [Changelog](https://github.com/PyCQA/pylint/blob/master/ChangeLog)
>> > - [Commits](PyCQA/pylint@pylint-2.4.3...pylint-2.4.4
>> > <https://github.com/PyCQA/pylint/compare/pylint-2.4.3...pylint-2.4.4>)
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Fri, Aug 2, 2019 at 8:34 AM Felix Uellendall <feluelle@pm.me.invalid
>> >
>> > wrote:
>> >
>> > > I understand what Ash and Jarek are saying. I actually use airflow
>> with
>> > > custom plugins to have a end-product that fully satisfy our needs and
>> > when
>> > > writing new hooks and operators I don't want to see "airflow uses
>> > > requirement foo=1 but you have foo=2" - but actually that sometimes
>> also
>> > > just works. So why not let the developer decide if he want to risk
>> > breaking
>> > > it?
>> > > I like the idea of firstly defining groups for core and non-core
>> > > dependencies. I think only need to manage the core ones to always work
>> > will
>> > > be a lot easier.
>> > > So I would suggest for core ones we have ranges including minor
>> releases
>> > > and non-core ones getting pinned by default installation (and ci).
>> > >
>> > > I think the more dependencies we add (which comes automatically as
>> > airflow
>> > > is growing) the more we are willing to set to more specific version
>> > ranges
>> > > and maybe even pin them at the end.
>> > >
>> > > Felix
>> > >
>> > > Sent from ProtonMail mobile
>> > >
>> > > -------- Original Message --------
>> > > On Aug 2, 2019, 06:10, Jarek Potiuk wrote:
>> > >
>> > > > Ash is totally right - that's exactly the difficulty we face.
>> Airflow
>> > is
>> > > > both a library and end product and this makes the usual advice (pin
>> if
>> > > you
>> > > > are end-product, don't pin if you are library) not really useful.
>> From
>> > > the
>> > > > very beginning of my adventures with Airflow I was for pinning of
>> > > > everything (and using dependabot or similar - I use it for other
>> > > projects),
>> > > > but over time I realised that this is very short-sighted approach.
>> it
>> > > does
>> > > > not take into account the "library" point of view.. Depending which
>> > user
>> > > of
>> > > > airflow you are, you have contradicting requirements. If you are
>> user
>> > of
>> > > > airflow who just want to use it as "end product" you want pinning.
>> If
>> > you
>> > > > want to develop your own operators or extend existing ones - you use
>> > > > airflow as "library" and you do not want pinning.
>> > > >
>> > > > I also proposed at the beginning of that thread that we split core
>> > > > requirements (and pin it) and non-core ones (and don't pin it). But
>> it
>> > > > ain't easy to separate those two sets in a clear way unfortunately.
>> > > >
>> > > > That's why the idea of choosing at the installation time (and not at
>> > > build
>> > > > time) whether you want to install "loose" or "frozen" dependencies
>> is
>> > so
>> > > > appealing.
>> > > > Possibly the best solution could be that you 'pip install airflow'
>> and
>> > > you
>> > > > get the pinned versions and then some other way to get the loose
>> one.
>> > > But I
>> > > > think we are a bit on the mercy of pip - this does not seem to be
>> > > possible.
>> > > >
>> > > > Then - it looks like using extras to add "pinning" mechanism is the
>> > next
>> > > > best idea.
>> > > >
>> > > > I am not afraid about complexity. We can fully automate generating
>> > those
>> > > > pinned requirements. I already have some ideas how we can make sure
>> > that
>> > > we
>> > > > keep those requirements in sync while developing and how they can
>> end
>> > up
>> > > > frozen in release. I would like to run a POC on that but in short
>> it is
>> > > > another "by-product" of the CI image we have now. Our CI image is
>> the
>> > > > perfect source of frozen requirements - we know those requirements
>> in
>> > CI
>> > > > image are ok and we can use them to generate the standard
>> > > > "requirements.txt" file and keep it updated via some local update
>> > script
>> > > > (and pre-commit hooks) + we can verify that they are updated in the
>> CI.
>> > > We
>> > > > can then write custom setup.py that will use that requirements.txt
>> and
>> > > the
>> > > > existing "extras" and generate "pinned" extras automatically. That
>> > sounds
>> > > > like fully doable and with very limited maintenance effort.
>> > > >
>> > > > J.
>> > > >
>> > > > On Thu, Aug 1, [2019](tel:2019) at 10:45 PM Qingping Hou <
>> > qph@scribd.com>
>> > > wrote:
>> > > >
>> > > >> On Thu, Aug 1, [2019](tel:2019) at 1:33 PM Chen Tong <
>> > cixuuz@gmail.com>
>> > > wrote:
>> > > >> > It is sometimes hard to distinguish if it is a library or an
>> > > application.
>> > > >> > Take operator as an example, a non-tech people may think it as a
>> > > >> well-built
>> > > >> > application while an engineer may consider it as a library and
>> > extends
>> > > >> > functionalities on it.
>> > > >> >
>> > > >>
>> > > >> Yeah, I agree. Personally, I would consider operator to be a
>> library
>> > > >> due to the expectation that other people will import them in their
>> own
>> > > >> projects/source tree.
>> > > >>
>> > > >> For things like REST endpoint handlers and perhaps scheduler, it
>> seems
>> > > >> safe to assume all changes and improvements will happen within
>> Airflow
>> > > >> source tree. In that case, it's safe to classify that part of code
>> as
>> > > >> application and freeze all its dependencies. The current plugin
>> system
>> > > >> might make this slightly complicated because people can still
>> extend
>> > > >> the core with custom code. But again, in an ideal world, plugins
>> > > >> should be self-contained and communicating with core through a well
>> > > >> defined interface ;)
>> > > >>
>> > > >
>> > > > --
>> > > >
>> > > > Jarek Potiuk
>> > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
>> > > >
>> > > > M: [+48 660 796 129](tel:+48660796129)
>> > <[+48660796129](tel:+48660796129)>
>> > > > [image: Polidea] <https://www.polidea.com/>
>> >
>> >
>> >
>> > --
>> >
>> > Jarek Potiuk
>> > Polidea <https://www.polidea.com/> | Principal Software Engineer
>> >
>> > M: +48 660 796 129 <+48660796129>
>> > [image: Polidea] <https://www.polidea.com/>
>> >
>>
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Back to (some) dependency pinning
Posted by Jarek Potiuk <Ja...@polidea.com>.
Just to update the plan and have some summary for those who did not read my
TL;DR mail (and include probot integration). Here is a shorter summary of
the proposal I have:
1. Keep the current process of *pip install -e[<extras>]* unchanged -
taking into account the limits from setup.py
2. Add *requirements.txt* file with a fixed set of requirements (always *==
-* including transitive deps). Those will be *managed by probot*,
similarly to dependabot.
3. PRs in CI will use the requirements.txt file as a constraint file
(known to work) when running tests to avoid transitive dependency problems
4. Daily CRON job installs all deps from scratch without *requirements.txt
- * to detect early problems with current set of dependencies. An
example of that from last night [1]: the CRON job failed and I
could prepare the fix [2] while PRs of contributors continued to work.
5. Airflow installation for development will be guaranteed to work with *pip
install -e . [<extras>] --constraints requirements.txt *from master - no
matter the state of transitive dependencies
6. We will modify setup.py to also have a way to take the constraints,
to guarantee installation works. Something like: *pip install
apache-airflow==1.10.6[<extras>,constrained] *
I'd love to hear more comments and ask you about opinion on that proposal.
[1] Failed CRON job from last night - triggered likely by "from scratch"
dependencies: https://travis-ci.org/apache/airflow/builds/618832805
[2] Fix to the last night CRON job failure:
https://github.com/apache/airflow/pull/6691
[4] AIP-8 - Split Hooks/Operators to separate packages:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827303
[3] AIP-21 - Changes in import paths
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
*@Dan Davydow* -> Fully agree we should have something like that. I think
this should be part of AIP-8 [3] when we get there. I hope we can learn a
lot about those when we get AIP-21 [4] fully implemented and backporting
packages generated and used for people as preparation for Airflow 2.0
migration (I hope it will happen soon - we are very close to have
automation in place to move all the packages from contrib - worked on by
Kamil Breguła). Having such (probot managed) set of requirements + a way
to test it automatically is - I think - an important prerequisite to start
designing it (and work out a reasonable approach for cross-dependency
problems).
J.
On Mon, Nov 18, 2019 at 6:18 PM Dan Davydov <dd...@twitter.com.invalid>
wrote:
> My 2 cents on the long-term plan:
> Once Airflow has Dag Isolation (i.e. DAG python dependencies are completely
> decoupled from Airflow), we should pin the core Airflow deps, and package
> operators separately with version ranges (for the reasons Ash mentioned
> about libraries vs applications).
>
> On Sat, Nov 16, 2019 at 6:22 AM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
> > After some discussions and some tests by myself and Astronomer docker, I
> > think I have finally a complete proposal for good, consistent solution
> for
> > our dependency management.
> >
> > It uses well-known standards, does not introduce any new tools (like
> > poetry) and I think it serves all the use cases we have very well.
> >
> > I wanted to summarise it here before I create it as an official AIP. I
> will
> > need to finalise the approach to complete the work on the official
> > production image.
> >
> > Here are some basic assumptions:
> >
> > 1) We keep the current process of setup.py which is used to install
> Airflow
> > for development *'pip install -e .[extras]'*. It's nice and modular. It
> has
> > open-dependencies (some of them constrained due to compatibilities). This
> > will break from time to time for fresh install when some transitive
> > dependencies are changed (or like with yesterday's pymssql drama). No
> > changes whatsoever for people who continue installing it this way. The
> same
> > setup.py is used to install airflow in production (but see point 7 below
> > for constrained builds).
> >
> > 2) On top of setup.py, we will have a standard *requirements.txt* file as
> > well. This will be a snapshot of current "working" dependencies
> (including
> > all transitive dependencies!). This file might be generated automatically
> > by pip freeze (with some scripting) and possibly put as part of
> pre-commit
> > checks to make sure it is self-maintainable when new dependencies are
> > added. With the current Breeze + Pre-commit setup we can do it fully
> > automatically, so that it will happen behind the scenes (and it will be
> > verified in CI). This requirements.txt file will be a set of "KNOWN TO
> BE
> > GOOD" requirements. This requirements.txt file will have ONLY
> > 'dependency==VERSION' - specific versions of each requirement . Nice
> > side-effect of the requirements.txt file is that if you use IDEs like
> > Pycharm, VSCode, it will automatically suggest installing
> missing/outdated
> > requirements in your virtualenv so you do not have to remember about it.
> >
> > 3) By having *requirements.txt *we can add a dependabot
> > <https://dependabot.com/> that will upgrade dependencies in this file
> > automatically (it works in a way that it will create a PR every time one
> of
> > the dependencies is released). It works nicely with some of our other
> > projects - see for example Oozie-2-Airflow here
> > <https://github.com/GoogleCloudPlatform/oozie-to-airflow/commits/master>
> .
> > It has the benefit that it prepares super-nice, isolated commits/PRs.
> Those
> > PRs (example here
> > <https://github.com/GoogleCloudPlatform/oozie-to-airflow/pull/438>) have
> > all the information about changes and commits. Those are nice and
> > actionable in terms of checking if we have breaking changes before
> merging
> > it (example below). The nice thing about dependabot is that it will
> create
> > a PR and our CI will test it automatically. So committers will have to
> just
> > double-check, take a look at the release notes and will be able to merge
> it
> > straight away. I will have to check if dependabot can be used for Apache
> > repos though (dependabot is owned by Github). It's an optional step and
> we
> > can do it manually though if we cannot use dependabot (but it requires
> some
> > more complex scripting).
> >
> > 4) We will use the requirements.txt file in CI in PRs so that we know
> that
> > CI always uses a "known to be good" set of requirements. Every PR will
> have
> > to have good set of requirements to be merged. No more transitive
> > dependency changes.
> >
> > 5) We have a special CRON job run in CI daily and this job will use '*pip
> > install -e . [all]*' (not using the requirements.txt) - this way we will
> > have a mechanism to detect early problems with transitive dependencies -
> > without breaking PR builds.
> >
> > 6) We can use the requirements.txt file as constraints file as well (!).
> > For example you should be able to run '*pip install -e . [gcp]
> > --constraints requirements.txt*' and it will install the "known to be
> good"
> > versions but only for those extras we want.
> >
> > 7) I believe I can customise setup.py to add extra "*constrained*" extra
> > which will also use the information from the requirements.txt. This way
> the
> > users will be able to do `*pip install
> > apache-airflow==1.10.6[gcp,aws,constrained]*` and have "known to be good"
> > installation of Airflow - even if some transitive dependencies break the
> > "clean" install. I am not 100% sure if it is possible exactly this way,
> but
> > I think in the worst case it will be `*pip install
> > apache-airflow=1.10.6[constrained,gcp-constrained,aws-constrained]*` - we
> > can generate a "-constrained" version of each of the extras automatically
> > based on the requirements.txt. I already had a POC for that. And then it
> > should be possible to upgrade each of the dependencies on its own.
> >
> > 8) Last but not least - we will use the requirements.txt file to build
> the
> > production image. One of the requirements for a good official image
> > <
> >
> https://github.com/docker-library/official-images/blob/master/README.md#repeatability
> > >
> > is repeatability - which boils down to dependency-pinning. Currently we
> > have no mechanism to enforce that. With requirements.txt this will be
> > possible.
> >
> > Let me know what you think!
> >
> > J.
> >
> >
> > Appendix: example dependabot-generated commit:
> >
> > Bump pylint from 2.4.3 to 2.4.4
> >
> > Bumps [pylint](https://github.com/PyCQA/pylint) from 2.4.3 to 2.4.4.
> > - [Release notes](https://github.com/PyCQA/pylint/releases)
> > - [Changelog](https://github.com/PyCQA/pylint/blob/master/ChangeLog)
> > - [Commits](PyCQA/pylint@pylint-2.4.3...pylint-2.4.4
> > <https://github.com/PyCQA/pylint/compare/pylint-2.4.3...pylint-2.4.4>)
> >
> >
> >
> >
> >
> >
> > On Fri, Aug 2, 2019 at 8:34 AM Felix Uellendall <fe...@pm.me.invalid>
> > wrote:
> >
> > > I understand what Ash and Jarek are saying. I actually use airflow with
> > > custom plugins to have a end-product that fully satisfy our needs and
> > when
> > > writing new hooks and operators I don't want to see "airflow uses
> > > requirement foo=1 but you have foo=2" - but actually that sometimes
> also
> > > just works. So why not let the developer decide if he want to risk
> > breaking
> > > it?
> > > I like the idea of firstly defining groups for core and non-core
> > > dependencies. I think only need to manage the core ones to always work
> > will
> > > be a lot easier.
> > > So I would suggest for core ones we have ranges including minor
> releases
> > > and non-core ones getting pinned by default installation (and ci).
> > >
> > > I think the more dependencies we add (which comes automatically as
> > airflow
> > > is growing) the more we are willing to set to more specific version
> > ranges
> > > and maybe even pin them at the end.
> > >
> > > Felix
> > >
> > > Sent from ProtonMail mobile
> > >
> > > -------- Original Message --------
> > > On Aug 2, 2019, 06:10, Jarek Potiuk wrote:
> > >
> > > > Ash is totally right - that's exactly the difficulty we face. Airflow
> > is
> > > > both a library and end product and this makes the usual advice (pin
> if
> > > you
> > > > are end-product, don't pin if you are library) not really useful.
> From
> > > the
> > > > very beginning of my adventures with Airflow I was for pinning of
> > > > everything (and using dependabot or similar - I use it for other
> > > projects),
> > > > but over time I realised that this is very short-sighted approach. it
> > > does
> > > > not take into account the "library" point of view.. Depending which
> > user
> > > of
> > > > airflow you are, you have contradicting requirements. If you are user
> > of
> > > > airflow who just want to use it as "end product" you want pinning. If
> > you
> > > > want to develop your own operators or extend existing ones - you use
> > > > airflow as "library" and you do not want pinning.
> > > >
> > > > I also proposed at the beginning of that thread that we split core
> > > > requirements (and pin it) and non-core ones (and don't pin it). But
> it
> > > > ain't easy to separate those two sets in a clear way unfortunately.
> > > >
> > > > That's why the idea of choosing at the installation time (and not at
> > > build
> > > > time) whether you want to install "loose" or "frozen" dependencies is
> > so
> > > > appealing.
> > > > Possibly the best solution could be that you 'pip install airflow'
> and
> > > you
> > > > get the pinned versions and then some other way to get the loose one.
> > > But I
> > > > think we are a bit on the mercy of pip - this does not seem to be
> > > possible.
> > > >
> > > > Then - it looks like using extras to add "pinning" mechanism is the
> > next
> > > > best idea.
> > > >
> > > > I am not afraid about complexity. We can fully automate generating
> > those
> > > > pinned requirements. I already have some ideas how we can make sure
> > that
> > > we
> > > > keep those requirements in sync while developing and how they can end
> > up
> > > > frozen in release. I would like to run a POC on that but in short it
> is
> > > > another "by-product" of the CI image we have now. Our CI image is the
> > > > perfect source of frozen requirements - we know those requirements in
> > CI
> > > > image are ok and we can use them to generate the standard
> > > > "requirements.txt" file and keep it updated via some local update
> > script
> > > > (and pre-commit hooks) + we can verify that they are updated in the
> CI.
> > > We
> > > > can then write custom setup.py that will use that requirements.txt
> and
> > > the
> > > > existing "extras" and generate "pinned" extras automatically. That
> > sounds
> > > > like fully doable and with very limited maintenance effort.
> > > >
> > > > J.
> > > >
> > > > On Thu, Aug 1, [2019](tel:2019) at 10:45 PM Qingping Hou <
> > qph@scribd.com>
> > > wrote:
> > > >
> > > >> On Thu, Aug 1, [2019](tel:2019) at 1:33 PM Chen Tong <
> > cixuuz@gmail.com>
> > > wrote:
> > > >> > It is sometimes hard to distinguish if it is a library or an
> > > application.
> > > >> > Take operator as an example, a non-tech people may think it as a
> > > >> well-built
> > > >> > application while an engineer may consider it as a library and
> > extends
> > > >> > functionalities on it.
> > > >> >
> > > >>
> > > >> Yeah, I agree. Personally, I would consider operator to be a library
> > > >> due to the expectation that other people will import them in their
> own
> > > >> projects/source tree.
> > > >>
> > > >> For things like REST endpoint handlers and perhaps scheduler, it
> seems
> > > >> safe to assume all changes and improvements will happen within
> Airflow
> > > >> source tree. In that case, it's safe to classify that part of code
> as
> > > >> application and freeze all its dependencies. The current plugin
> system
> > > >> might make this slightly complicated because people can still extend
> > > >> the core with custom code. But again, in an ideal world, plugins
> > > >> should be self-contained and communicating with core through a well
> > > >> defined interface ;)
> > > >>
> > > >
> > > > --
> > > >
> > > > Jarek Potiuk
> > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > >
> > > > M: [+48 660 796 129](tel:+48660796129)
> > <[+48660796129](tel:+48660796129)>
> > > > [image: Polidea] <https://www.polidea.com/>
> >
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
> >
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Back to (some) dependency pinning
Posted by Dan Davydov <dd...@twitter.com.INVALID>.
My 2 cents on the long-term plan:
Once Airflow has Dag Isolation (i.e. DAG python dependencies are completely
decoupled from Airflow), we should pin the core Airflow deps, and package
operators separately with version ranges (for the reasons Ash mentioned
about libraries vs applications).
On Sat, Nov 16, 2019 at 6:22 AM Jarek Potiuk <Ja...@polidea.com>
wrote:
> After some discussions and some tests by myself and Astronomer docker, I
> think I have finally a complete proposal for good, consistent solution for
> our dependency management.
>
> It uses well-known standards, does not introduce any new tools (like
> poetry) and I think it serves all the use cases we have very well.
>
> I wanted to summarise it here before I create it as an official AIP. I will
> need to finalise the approach to complete the work on the official
> production image.
>
> Here are some basic assumptions:
>
> 1) We keep the current process of setup.py which is used to install Airflow
> for development *'pip install -e .[extras]'*. It's nice and modular. It has
> open-dependencies (some of them constrained due to compatibilities). This
> will break from time to time for fresh install when some transitive
> dependencies are changed (or like with yesterday's pymssql drama). No
> changes whatsoever for people who continue installing it this way. The same
> setup.py is used to install airflow in production (but see point 7 below
> for constrained builds).
>
> 2) On top of setup.py, we will have a standard *requirements.txt* file as
> well. This will be a snapshot of current "working" dependencies (including
> all transitive dependencies!). This file might be generated automatically
> by pip freeze (with some scripting) and possibly put as part of pre-commit
> checks to make sure it is self-maintainable when new dependencies are
> added. With the current Breeze + Pre-commit setup we can do it fully
> automatically, so that it will happen behind the scenes (and it will be
> verified in CI). This requirements.txt file will be a set of "KNOWN TO BE
> GOOD" requirements. This requirements.txt file will have ONLY
> 'dependency==VERSION' - specific versions of each requirement . Nice
> side-effect of the requirements.txt file is that if you use IDEs like
> Pycharm, VSCode, it will automatically suggest installing missing/outdated
> requirements in your virtualenv so you do not have to remember about it.
>
> 3) By having *requirements.txt *we can add a dependabot
> <https://dependabot.com/> that will upgrade dependencies in this file
> automatically (it works in a way that it will create a PR every time one of
> the dependencies is released). It works nicely with some of our other
> projects - see for example Oozie-2-Airflow here
> <https://github.com/GoogleCloudPlatform/oozie-to-airflow/commits/master> .
> It has the benefit that it prepares super-nice, isolated commits/PRs. Those
> PRs (example here
> <https://github.com/GoogleCloudPlatform/oozie-to-airflow/pull/438>) have
> all the information about changes and commits. Those are nice and
> actionable in terms of checking if we have breaking changes before merging
> it (example below). The nice thing about dependabot is that it will create
> a PR and our CI will test it automatically. So committers will have to just
> double-check, take a look at the release notes and will be able to merge it
> straight away. I will have to check if dependabot can be used for Apache
> repos though (dependabot is owned by Github). It's an optional step and we
> can do it manually though if we cannot use dependabot (but it requires some
> more complex scripting).
>
> 4) We will use the requirements.txt file in CI in PRs so that we know that
> CI always uses a "known to be good" set of requirements. Every PR will have
> to have good set of requirements to be merged. No more transitive
> dependency changes.
>
> 5) We have a special CRON job run in CI daily and this job will use '*pip
> install -e . [all]*' (not using the requirements.txt) - this way we will
> have a mechanism to detect early problems with transitive dependencies -
> without breaking PR builds.
>
> 6) We can use the requirements.txt file as constraints file as well (!).
> For example you should be able to run '*pip install -e . [gcp]
> --constraints requirements.txt*' and it will install the "known to be good"
> versions but only for those extras we want.
>
> 7) I believe I can customise setup.py to add extra "*constrained*" extra
> which will also use the information from the requirements.txt. This way the
> users will be able to do `*pip install
> apache-airflow==1.10.6[gcp,aws,constrained]*` and have "known to be good"
> installation of Airflow - even if some transitive dependencies break the
> "clean" install. I am not 100% sure if it is possible exactly this way, but
> I think in the worst case it will be `*pip install
> apache-airflow=1.10.6[constrained,gcp-constrained,aws-constrained]*` - we
> can generate a "-constrained" version of each of the extras automatically
> based on the requirements.txt. I already had a POC for that. And then it
> should be possible to upgrade each of the dependencies on its own.
>
> 8) Last but not least - we will use the requirements.txt file to build the
> production image. One of the requirements for a good official image
> <
> https://github.com/docker-library/official-images/blob/master/README.md#repeatability
> >
> is repeatability - which boils down to dependency-pinning. Currently we
> have no mechanism to enforce that. With requirements.txt this will be
> possible.
>
> Let me know what you think!
>
> J.
>
>
> Appendix: example dependabot-generated commit:
>
> Bump pylint from 2.4.3 to 2.4.4
>
> Bumps [pylint](https://github.com/PyCQA/pylint) from 2.4.3 to 2.4.4.
> - [Release notes](https://github.com/PyCQA/pylint/releases)
> - [Changelog](https://github.com/PyCQA/pylint/blob/master/ChangeLog)
> - [Commits](PyCQA/pylint@pylint-2.4.3...pylint-2.4.4
> <https://github.com/PyCQA/pylint/compare/pylint-2.4.3...pylint-2.4.4>)
>
>
>
>
>
>
> On Fri, Aug 2, 2019 at 8:34 AM Felix Uellendall <fe...@pm.me.invalid>
> wrote:
>
> > I understand what Ash and Jarek are saying. I actually use airflow with
> > custom plugins to have a end-product that fully satisfy our needs and
> when
> > writing new hooks and operators I don't want to see "airflow uses
> > requirement foo=1 but you have foo=2" - but actually that sometimes also
> > just works. So why not let the developer decide if he want to risk
> breaking
> > it?
> > I like the idea of firstly defining groups for core and non-core
> > dependencies. I think only need to manage the core ones to always work
> will
> > be a lot easier.
> > So I would suggest for core ones we have ranges including minor releases
> > and non-core ones getting pinned by default installation (and ci).
> >
> > I think the more dependencies we add (which comes automatically as
> airflow
> > is growing) the more we are willing to set to more specific version
> ranges
> > and maybe even pin them at the end.
> >
> > Felix
> >
> > Sent from ProtonMail mobile
> >
> > -------- Original Message --------
> > On Aug 2, 2019, 06:10, Jarek Potiuk wrote:
> >
> > > Ash is totally right - that's exactly the difficulty we face. Airflow
> is
> > > both a library and end product and this makes the usual advice (pin if
> > you
> > > are end-product, don't pin if you are library) not really useful. From
> > the
> > > very beginning of my adventures with Airflow I was for pinning of
> > > everything (and using dependabot or similar - I use it for other
> > projects),
> > > but over time I realised that this is very short-sighted approach. it
> > does
> > > not take into account the "library" point of view.. Depending which
> user
> > of
> > > airflow you are, you have contradicting requirements. If you are user
> of
> > > airflow who just want to use it as "end product" you want pinning. If
> you
> > > want to develop your own operators or extend existing ones - you use
> > > airflow as "library" and you do not want pinning.
> > >
> > > I also proposed at the beginning of that thread that we split core
> > > requirements (and pin it) and non-core ones (and don't pin it). But it
> > > ain't easy to separate those two sets in a clear way unfortunately.
> > >
> > > That's why the idea of choosing at the installation time (and not at
> > build
> > > time) whether you want to install "loose" or "frozen" dependencies is
> so
> > > appealing.
> > > Possibly the best solution could be that you 'pip install airflow' and
> > you
> > > get the pinned versions and then some other way to get the loose one.
> > But I
> > > think we are a bit on the mercy of pip - this does not seem to be
> > possible.
> > >
> > > Then - it looks like using extras to add "pinning" mechanism is the
> next
> > > best idea.
> > >
> > > I am not afraid about complexity. We can fully automate generating
> those
> > > pinned requirements. I already have some ideas how we can make sure
> that
> > we
> > > keep those requirements in sync while developing and how they can end
> up
> > > frozen in release. I would like to run a POC on that but in short it is
> > > another "by-product" of the CI image we have now. Our CI image is the
> > > perfect source of frozen requirements - we know those requirements in
> CI
> > > image are ok and we can use them to generate the standard
> > > "requirements.txt" file and keep it updated via some local update
> script
> > > (and pre-commit hooks) + we can verify that they are updated in the CI.
> > We
> > > can then write custom setup.py that will use that requirements.txt and
> > the
> > > existing "extras" and generate "pinned" extras automatically. That
> sounds
> > > like fully doable and with very limited maintenance effort.
> > >
> > > J.
> > >
> > > On Thu, Aug 1, [2019](tel:2019) at 10:45 PM Qingping Hou <
> qph@scribd.com>
> > wrote:
> > >
> > >> On Thu, Aug 1, [2019](tel:2019) at 1:33 PM Chen Tong <
> cixuuz@gmail.com>
> > wrote:
> > >> > It is sometimes hard to distinguish if it is a library or an
> > application.
> > >> > Take operator as an example, a non-tech people may think it as a
> > >> well-built
> > >> > application while an engineer may consider it as a library and
> extends
> > >> > functionalities on it.
> > >> >
> > >>
> > >> Yeah, I agree. Personally, I would consider operator to be a library
> > >> due to the expectation that other people will import them in their own
> > >> projects/source tree.
> > >>
> > >> For things like REST endpoint handlers and perhaps scheduler, it seems
> > >> safe to assume all changes and improvements will happen within Airflow
> > >> source tree. In that case, it's safe to classify that part of code as
> > >> application and freeze all its dependencies. The current plugin system
> > >> might make this slightly complicated because people can still extend
> > >> the core with custom code. But again, in an ideal world, plugins
> > >> should be self-contained and communicating with core through a well
> > >> defined interface ;)
> > >>
> > >
> > > --
> > >
> > > Jarek Potiuk
> > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >
> > > M: [+48 660 796 129](tel:+48660796129)
> <[+48660796129](tel:+48660796129)>
> > > [image: Polidea] <https://www.polidea.com/>
>
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>