Re: Backwards compability - what do we mean? when? how long?

[Chris Riccomini <cr...@apache.org>]

@ash, I think your proposal is actually pretty reasonable. Do you want to
document it on the wiki, and we can discuss/take a vote?

On Tue, Dec 19, 2017 at 10:45 AM, Ash Berlin-Taylor <
ash_airflowlist@firemirror.com> wrote:

> Hi,
>
> A question came up on a github issue about what exactly we meant about
> backwards compatibility, and I figured we as a project should work out what
> we mean when we say we want to maintain compat. And most importantly
> document it (don't worry, I'm volunteering to do bit, so long as we reach
> consensus).
>
> So the issue that spawned this was https://github.com/apache/
> incubator-airflow/pull/2806 <https://github.com/apache/
> incubator-airflow/pull/2806> which changes some config setting names.
>
> In the example of this particular PR it's not a big deal, and supporting
> both names is not a difficult change, but I felt this was a discussion
> worth having.
>
> My view is somewhat influenced by Django which has a view that if you have
> no deprecation warnings now (say on 1.8) then if you upgrade to 1.9 things
> will still work, but now you might get deprecation warnings that will need
> to be fixed before upgrading to 1.10. (Version numbers just for example.)
>
> Bolke's view is: "backwards compatible is more along the lines of FreeBSD.
> API/ABI compatible, but config changes can happen and are in UPDATING"
>
> Now, Airflow is still a relatively young project so perhaps we want to be
> a little more relaxed about this for the time being? Do we want to just say
> backwards compatibility is required for operators and hooks -- i.e. things
> that most users are going to interact with, but other bits of internals are
> "fair game to change", but perhaps with a best-effort goal.
>
> So some questions:
>
> 1) Do we want to follow something like SemVer (semver.org <
> http://semver.org/>) where backwards-incompatible changes need a major
> version bump, which would be to Airflow 2.0 in this case. I think this is
> what we do, but I don't believe is written down anywhere.
>
> 2) Do we want to limit or exclude the back-compart _guarantees_ to any
> areas? ie. just include Hooks and Operators? What about data model/DB
> tables? Log formats? Log file paths?
>
> Does anyone have any other strong opinions about areas I haven't mentioned?
>
> My answers are:
> 1) Yes to SemVer
> 2) Strong Yes to Hooks and Operators and anything used directly in DAGs,
> with a weak yes to including config and other python classes too.
>
> My aim here is to start some discussion. If we get to any consensus then
> after the holidays I'll open a PR to update the docs.
>
> Cheers,
> -ash
>
>
>

Re: Backwards compability - what do we mean? when? how long?

[George Leslie-Waksman <ge...@cloverhealth.com.INVALID>]

I'm glad to see us start having this discussion. Thank you, Ash.

1) +1 to using SemVer.

2) I'm a strong Yes to Hooks and Operators, moderate on log output, and
otherwise pretty weak in terms of the guarantees we should provide.

That said, I think some of this is made more problematic for us by Airflow
being a single monolith; it makes it hard for us to change some things but
not others.

If the core engine/models, webserver, and Hooks/Operators were three
different projects, we would have a lot more flexibility in how we manage
versioning. We could establish a guarantee that everything would remain
compatible for one subsequent engine version. For example, engine 2.1 would
run DAGs written using Hooks/Operators 2.0, 2.1. People could upgrade their
engine and take advantage of new features while having a reasonable
migration path to upgrade their DAGs to the new Hooks/Operators.

--George

On Mon, Jan 8, 2018 at 9:57 AM Chris Riccomini <cr...@apache.org>
wrote:

> @ash, I think your proposal is actually pretty reasonable. Do you want to
> document it on the wiki, and we can discuss/take a vote?
>
> On Tue, Dec 19, 2017 at 10:45 AM, Ash Berlin-Taylor <
> ash_airflowlist@firemirror.com> wrote:
>
> > Hi,
> >
> > A question came up on a github issue about what exactly we meant about
> > backwards compatibility, and I figured we as a project should work out
> what
> > we mean when we say we want to maintain compat. And most importantly
> > document it (don't worry, I'm volunteering to do bit, so long as we reach
> > consensus).
> >
> > So the issue that spawned this was https://github.com/apache/
> > incubator-airflow/pull/2806 <https://github.com/apache/
> > incubator-airflow/pull/2806> which changes some config setting names.
> >
> > In the example of this particular PR it's not a big deal, and supporting
> > both names is not a difficult change, but I felt this was a discussion
> > worth having.
> >
> > My view is somewhat influenced by Django which has a view that if you
> have
> > no deprecation warnings now (say on 1.8) then if you upgrade to 1.9
> things
> > will still work, but now you might get deprecation warnings that will
> need
> > to be fixed before upgrading to 1.10. (Version numbers just for example.)
> >
> > Bolke's view is: "backwards compatible is more along the lines of
> FreeBSD.
> > API/ABI compatible, but config changes can happen and are in UPDATING"
> >
> > Now, Airflow is still a relatively young project so perhaps we want to be
> > a little more relaxed about this for the time being? Do we want to just
> say
> > backwards compatibility is required for operators and hooks -- i.e.
> things
> > that most users are going to interact with, but other bits of internals
> are
> > "fair game to change", but perhaps with a best-effort goal.
> >
> > So some questions:
> >
> > 1) Do we want to follow something like SemVer (semver.org <
> > http://semver.org/>) where backwards-incompatible changes need a major
> > version bump, which would be to Airflow 2.0 in this case. I think this is
> > what we do, but I don't believe is written down anywhere.
> >
> > 2) Do we want to limit or exclude the back-compart _guarantees_ to any
> > areas? ie. just include Hooks and Operators? What about data model/DB
> > tables? Log formats? Log file paths?
> >
> > Does anyone have any other strong opinions about areas I haven't
> mentioned?
> >
> > My answers are:
> > 1) Yes to SemVer
> > 2) Strong Yes to Hooks and Operators and anything used directly in DAGs,
> > with a weak yes to including config and other python classes too.
> >
> > My aim here is to start some discussion. If we get to any consensus then
> > after the holidays I'll open a PR to update the docs.
> >
> > Cheers,
> > -ash
> >
> >
> >
>