Releasing Apache Airflow's Python Client

[Sumit Maheshwari <ms...@apache.org>]

Hello Airflow dev community

I'm writing this mail in regards to seek feedback on the first release of
Apache Airflow's Python client
<https://github.com/apache/airflow-client-python>. This is an icebreaker
email only and we will follow the Apache process of releasing the version
later on. As of now, I've raised a small PR
<https://github.com/apache/airflow-client-python/pull/6> to add the missing
files in the repo, which are required to do the release of the client lib.

The main question I've in mind about the versioning. I see that there are
mainly two theories related to maintain the release versions of clients:

   1. The first is about following the semantic release of the clients on
   its own. The release version of the client doesn't include the version of
   the software itself. Something like we are doing with Airflow's provider
   packages. However, in each release, we mention the release version of the
   main software (Airflow in this case). This is the simple most way of
   maintaining releases and followed majorly.
   2. The second way is to include the software's version as well in its
   client's versioning, something like it's been done in the K8s Python
   client
   <https://github.com/kubernetes-client/python/blob/master/CHANGELOG.md>.
   In this approach, the client version provides more implicit knowledge about
   the compatible software, however it creates an extra burden of releasing
   the client with every minor release of the software, otherwise the
   versioning would fall apart. Also, there is a good chance that most of
   these minor or even a major release has no changes in the APIs, but the
   client would be needed to re-released nevertheless, just to keep the
   versioning in check.

Personally, I'm in favour of the 1st way, as it's simple to manage and
doesn't tightly couple client with its software.

Another unrelated question I have is about the release of different client
libraries. As of now, Airflow has Python and Go
<https://github.com/apache/airflow-client-go> clients, so should we try to
release them together or they should follow their own paths? In general,
I'm against batching them together, cause the person who is releasing them
may not be able to test all the different clients in the future.

Let me know your thoughts on these 2 points.

Thanks,
Sumit Maheshwari
PMC Apache Airflow

Re: Releasing Apache Airflow's Python Client

[Sumit Maheshwari <su...@gmail.com>]

I think it's OK to conclude the release strategy for clients:

So the first two digits of the client's release number will be borrowed
from the first two digits of Airflow's release. The last and third digit
would be independent of the Airflow release and will denote any patches
done on the client side.

For example, the first release of the Python client would be *2.0.0* which
would be based on *Airflow 2.0.1*.

On Mon, Apr 5, 2021 at 5:46 PM Sumit Maheshwari <su...@gmail.com>
wrote:

> I think I'm fine with the approach you suggested Jarek. This approach
> mitigates the need to release clients with each patch release of Airflow,
> and that does make sense.
>
> I hope that others are fine as well with the approach, where we've to
> release clients with every major or minor release of Airflow.
>
> On Mon, Apr 5, 2021 at 5:31 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> Hmm. I think it's a great direction, but 4 digits is a bit too much :)
>> keeping three-digit SemVer is good for clarity/tools. And itroducing 4th
>> digit is not really needed i think.
>>
>> Your proposal Sumit made me think,  and I asked myself a few questions:
>>
>> 1) will we ever release API client new feature without releasing that
>> feature in airflow ? I think not. The API client should provide API calls
>> for all the endpoints available and any change in those endpoints or adding
>> new one  - by definition - should mean a bump in airflow minor version at
>> least.
>>
>> 2) is it likely we release a bugfix to a single API client only without
>> releasing Airflow new bugfix? I think so. We might upgrade generator for
>> one client only to fix a particular problem we find for that language.
>>
>> 3) is it likely we release airflow bugfix without the need of releasing
>> any of the clients? Yeah it's very likely the client will work without any
>> changes with the new Airflow bugfix version.
>>
>> ... thinking ....
>>
>> So, how about a hybrid solution where we also include the minor version
>> of Airflow in the client version
>>
>> Say airflow 2.1.x will work only with client 2.1.n but x and n are
>> independent ?
>>
>> Example:
>>
>> We release 2.1.0 of Airflow and 2.1.0 of the Python and Java clients.
>>
>> Then we release 2.1.1, 2.1.2, 2.1.3 of Airflow. No release of clients.
>>
>> Then we find a bug in the Python generator, upgrade it and fix it in
>> 2.1.1 version of the Python client. But the Java client might still be at
>> 2.1.0 as the generator for Java worked fine.
>>
>>  At this point everyone who uses 2.1 airflow SHOULD upgrade to 2.1.3 of
>> Airflow and 2.1.1 of the Python client and 2.1.0 of the Java client. If
>> anyone uses any other combination, they should first upgrade to those
>> latest 2.1 ones before 'complaining' that there are some problems.
>>
>> Next - whenever we release 2.2.0 of Airflow we also release 2.2.0 of all
>> clients. And our client  check if Airflow is the same major/minor version
>> and fails if it is not. So you cannot use 2.1.1 Python client with 2.0
>> Airflow - you must switch to 2.2.0.
>>
>> I think that might be the nicest approach for both users and maintainers.
>> In this case we do not even have to maintain the 'compatibility' matrix i
>> mentioned before. And we keep 100% Semver semantics. And we can add
>> automated tests for all the 'last X.Y client + airflow' combination as
>> there is always only one 'blessed' combination airflow + clients that is
>> 'supposed' to work.
>>
>> WDYT?
>>
>> J.
>>
>>
>>
>>
>> pon., 5 kwi 2021, 08:09 użytkownik Sumit Maheshwari <
>> sumeet.manit@gmail.com> napisał:
>>
>>> Thanks, everyone for the inputs.
>>>
>>> I think it's clear that we want to go with the hybrid way of versioning.
>>> Given that we can automate the client releases easily, how if we release a
>>> client automatically with each Airflow release, and go for a four numbered
>>> release, where the first three would be the Airflow's release number and
>>> the fourth one would be specific to the client, something like this
>>>
>>> *Airflow 2.0.1 <=> Client 2.0.1.0* -- Initial release
>>> *Airflow 2.0.1 <=> Client 2.0.1.1* -- Some fix done in client (treating
>>> major or minor same for client, ideally there won't be any major patches in
>>> client)
>>>
>>> The upside of this process is that we can automate it & the client
>>> version will be in sync with the Airflow version always. The downside that
>>> we would be releasing more packages than required.
>>>
>>> About Vikram's question of not supporting any APIs in the client, AFAIK
>>> there are none.
>>>
>>> On Mon, Apr 5, 2021 at 2:51 AM Vikram Koka <vi...@astronomer.io.invalid>
>>> wrote:
>>>
>>>> Sumit,
>>>> Thank you for bringing up this discussion.
>>>>
>>>> Reading through the thread, I am definitely far more comfortable with
>>>> the hybrid approach proposed by Kaxil, than the first approach, primarily
>>>> because of the same concerns articulated by Jarek regarding end user
>>>> experience. Prior to that, I was leaning towards the second approach.
>>>>
>>>> Having said that, I think we need to be sure to follow certain
>>>> guidelines with Core Airflow to make sure this approach works consistently,
>>>> with a minimum of documentation needed and documentation to be read by
>>>> Airflow users. Since there are generated clients, I am specifically
>>>> referring to changes within Airflow regarding the API. Most importantly, I
>>>> think we need to have any API additions, whether a new endpoint or an
>>>> enhancement to an existing endpoint, only be available as part of a
>>>> "feature release", rather than a "patch release". And that, this should
>>>> trigger an update to the generated clients, triggering a new version of the
>>>> same. This may be obvious, because we agreed on following semantic
>>>> versioning strictly, but I thought it was worth reiterating.
>>>>
>>>> Having said that, I think it is worth asking the question if that is
>>>> actually true, or if there are any APIs which are not  (or should not be)
>>>> supported in the clients?
>>>> I didn't think so, but wanted to make sure.
>>>>
>>>> Best regards,
>>>> Vikram
>>>>
>>>>
>>>> On Sat, Apr 3, 2021 at 4:11 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>
>>>>> I also like the hybrid approach proposed by Kaxil. It is not as
>>>>> simplenfor dependency management as 1-1 version, but IMHO it is
>>>>> simple-enough. As long as we strictly follow SemVer, a number of questions
>>>>> i listed above from the user, are answered:
>>>>>
>>>>>
>>>>> 1) Will my client be compatible with the version of Airflow ?
>>>>>
>>>>> Easy: 2.x.y will be compatible with 2.n.m of Airflow and so on.. Since
>>>>> we will not have breaking changes, the 'broad compatibility' level should
>>>>> be fine (with some feature -level differences but no general compatibility
>>>>> problems)
>>>>>
>>>>> 2) Will feature (API endpoint) x available in this version of client
>>>>> be working with the version of Airflow I have?
>>>>>
>>>>> We have to make sure that features are documented in the APIs at the
>>>>> level of '@since' if we keep it updated the form 'This feature is available
>>>>> in  'Airflow 2.3.*' and 'API 2.1.*' - we are covered. Since there will be
>>>>> no breaking changes in neither API nor Airflow - this will be enough.
>>>>>
>>>>>
>>>>> 3) Which version of Airflow should I install in order to use feature X
>>>>> that I have in this new version of client ?
>>>>>
>>>>> I think this might be possible to generate such compatibility matrix
>>>>> for all new API features added since Airflow 2.0 - it might be
>>>>> automatically generated from the API specification in this case. If we
>>>>> parse the API definition and have the right 'since' definition for both
>>>>> Airflow and API, it should be rather easy to generate such table:
>>>>>
>>>>> Feature       | Min API version | Min Airflow version
>>>>> MoonShot feature X  | 2.3 | 2.2
>>>>>
>>>>> Having such compatibility matric would, I think, be a great help for
>>>>> the users i think when they would plan their Airflow/API releases
>>>>>
>>>>>
>>>>> J.
>>>>>
>>>>> sob., 3 kwi 2021, 12:06 użytkownik .... <dz...@gmail.com> napisał:
>>>>>
>>>>>> +1 for Kaxil proposal
>>>>>>
>>>>>> I think, attempting to assign a specific client version to the library
>>>>>> application makes no sense in creating an API that is based on a
>>>>>> strict specification.
>>>>>>
>>>>>> One of the goals of this API was to make it easier to manage multiple
>>>>>> instances, and it's not often that you have an identically homogeneous
>>>>>> environment. It's natural to have different versions of Airflow in one
>>>>>> organization. If we want users to use a specific version of the
>>>>>> library on the client-side, it will probably not be possible in many
>>>>>> cases.
>>>>>>
>>>>>> pt., 2 kwi 2021 o 23:35 Kaxil Naik <ka...@gmail.com> napisał(a):
>>>>>> >
>>>>>> > Thanks, Sumit, for bringing this up.
>>>>>> >
>>>>>> > re: versioning for Python client:
>>>>>> >
>>>>>> > I favour a bit of a hybrid of 1 and 2. Initially, I favored just
>>>>>> (1) when we talked but after seeing how Elastic-Search does it I liked
>>>>>> their versioning strategy.
>>>>>> > How their python-client is compatible with ES itself is documented
>>>>>> at https://elasticsearch-py.readthedocs.io/en/v7.12.0/#compatibility.
>>>>>> >
>>>>>> > Since Airflow from 2.0 strictly follows SemVer we can continue with
>>>>>> the assumption that we (Airflow) won't break backwards-compatibility (to
>>>>>> the best of our ability).
>>>>>> > So we can have M.x.y as our versioning for our client where M is
>>>>>> the Major Version of the client that matches the Major version of Airflow;
>>>>>> x is the minor version for the client & y is the patch release. Here x & y
>>>>>> are independent of Airflow version. Our goal and message to user would be
>>>>>> that "You have to use a version of client library matching the major
>>>>>> version of Airflow"
>>>>>> >
>>>>>> > re: the release of different client libraries:
>>>>>> >
>>>>>> > Client for each language should be versioned & released separately.
>>>>>> >
>>>>>> > Regards,
>>>>>> > Kaxil
>>>>>> >
>>>>>> > On Wed, Mar 31, 2021 at 7:01 PM Jarek Potiuk <ja...@potiuk.com>
>>>>>> wrote:
>>>>>> >>
>>>>>> >> I think there are more questions we need to answer when it comes
>>>>>> to different versions. If we are going to different versioning for agent
>>>>>> and Airflow, we will have to accept the fact that over time people will
>>>>>> have a "matrix of versions" - for both Airflow and the client. And this
>>>>>> introduces "dependency mess" if we do not handle it well.
>>>>>> >>
>>>>>> >> I think in this we should give the user much clearer indication on
>>>>>> which version of Airflow the client should work on, and (more important)
>>>>>> which version it does NOT work on and which features will NOT be available.
>>>>>> Both cases are possible.
>>>>>> >>
>>>>>> >> There should be some way of knowing for the user, (in case there
>>>>>> is a backwards incompatibility or in case some features are not available)
>>>>>> that a given version of the client does not work with the version of
>>>>>> Airflow that the user has.
>>>>>> >>
>>>>>> >> I think users looking at the documentation should be able to be
>>>>>> able to figure out:
>>>>>> >>
>>>>>> >> 1) Will my client be compatible with the version of Airflow ?
>>>>>> >> 2) Will feature (API endpoint) x available in this version of
>>>>>> client be working with the version of Airflow I have?
>>>>>> >> 3) Which version of Airflow should I install in order to use
>>>>>> feature X that I have in this new version of client ?
>>>>>> >>
>>>>>> >> Similarly proper error messages should be displayed if there is
>>>>>> incompatibility or when a feature X is not available when you try to use a
>>>>>> version of client that supports it with Airflow version of Agent that does
>>>>>> not have it.
>>>>>> >>
>>>>>> >> So I'd say - separate versioning is fine as long as we handle the
>>>>>> above.
>>>>>> >>
>>>>>> >>
>>>>>> >> Saying all that - there is a beautiful "maintenance" simplicity if
>>>>>> we actually synchronize the version and we say "Always use SAME version of
>>>>>> client as Airflow" (and we fail the client if the versions is different).
>>>>>> This is way more simple in communicating to the users, and with most
>>>>>> deployments, I think upgrading airflow and clients at the same time should
>>>>>> be no problem.
>>>>>> >>
>>>>>> >> I think over time as the matrix of potential Airflow/Client
>>>>>> combinations will grow this might mean bigger number of issues that we have
>>>>>> to handle ("Why ma client does not do this and this (and of course it will
>>>>>> not have neither client nor airflow version specified and we will have to
>>>>>> ask for clarification).
>>>>>> >>
>>>>>> >> So as I see it - maybe release-wise it's a little redundant and
>>>>>> you'd have to release a bugfix version of Airflow if you find a critical
>>>>>> problem in Client (which should be rare as agent is mostly automatically
>>>>>> generated), but synchronizing versions is very tempting from the
>>>>>> "supportability" point of view.
>>>>>> >>
>>>>>> >> Those are my 2 cents, I was not involved too much in the client
>>>>>> part, so I do not feel like I have a strong voice in there, but if I would
>>>>>> vote between the two - I'd choose synchronizing the releases.
>>>>>> >>
>>>>>> >> BTW. In providers we have a very different story - we keep
>>>>>> dependency to airflow via PIP (which we can set it, if we find a provider
>>>>>> stops being compatible with a version of Airflow). Also we automatically
>>>>>> test in CI if the provider installs and imports on Airflow 2.0.0 to verify
>>>>>> that it is backwards compatible (at least to the level of importability).
>>>>>> This has already saved us from  "accidental" incompatibilities (wit yaml
>>>>>> package import change recently). So there, separate versioning makes much
>>>>>> more sense - also because the version of the provider relates exclusively
>>>>>> to the feature that the provider version has, not Airflow itself). I think
>>>>>> this is quite a different story from the client, which is very closely
>>>>>> coupled with Airflow.
>>>>>> >>
>>>>>> >> J.
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> On Wed, Mar 31, 2021 at 11:08 AM Deng Xiaodong <
>>>>>> xd.deng.r@gmail.com> wrote:
>>>>>> >>>
>>>>>> >>> Yep, if we can, I would prefer to explicitly mention something
>>>>>> like 2.0.X :)
>>>>>> >>>
>>>>>> >>> On Wed, Mar 31, 2021 at 10:54 AM Sumit Maheshwari <
>>>>>> sumeet.manit@gmail.com> wrote:
>>>>>> >>>>
>>>>>> >>>> Yeah totally agrees with you XD about the range vs the fixed
>>>>>> version of the main software. And with Airflow 2 dot & beyond, we would be
>>>>>> following the semantic versioning more judiciously, so with that can we
>>>>>> assume that users would be cognizant of the related Airflow version
>>>>>> implicitly, or do you think we should mention something like 2.0.X in the
>>>>>> changelog explicitly?
>>>>>> >>>>
>>>>>> >>>> On Wed, Mar 31, 2021 at 1:38 PM Deng Xiaodong <
>>>>>> xd.deng.r@gmail.com> wrote:
>>>>>> >>>>>
>>>>>> >>>>> Thanks, Sumit, for bringing this up.
>>>>>> >>>>>
>>>>>> >>>>> - Regarding versioning for Python client: I favour the 1st way
>>>>>> as well. The only minor comment I have is: instead of "mention the release
>>>>>> version of the main software", possibly what users would expect is a "range
>>>>>> of compatible versions of main software". If we follow semver for Airflow
>>>>>> itself relatively strictly, this should not be hard.
>>>>>> >>>>>
>>>>>> >>>>> - Regarding the release of different client libraries: I agree
>>>>>> with you. They should be versioned & released separately.
>>>>>> >>>>>
>>>>>> >>>>>
>>>>>> >>>>> Regards,
>>>>>> >>>>> XD
>>>>>> >>>>>
>>>>>> >>>>> On Wed, Mar 31, 2021 at 9:21 AM Sumit Maheshwari <
>>>>>> sumeet.manit@gmail.com> wrote:
>>>>>> >>>>>>
>>>>>> >>>>>> Thanks QP for your feedback. I think the latest release of K8s
>>>>>> client (v17.x.y) is moving in another direction.
>>>>>> >>>>>>
>>>>>> >>>>>> On Wed, Mar 31, 2021 at 12:13 AM QP Hou <qp...@scribd.com.invalid>
>>>>>> wrote:
>>>>>> >>>>>>>
>>>>>> >>>>>>> Thanks Sumit for kicking off this discussion!
>>>>>> >>>>>>>
>>>>>> >>>>>>> I am in favor of approach #1 as well. Airflow software
>>>>>> version can
>>>>>> >>>>>>> change due to none API related changes, but the client should
>>>>>> only
>>>>>> >>>>>>> care about the public API contract. I believe this is also
>>>>>> the same
>>>>>> >>>>>>> approach that the k8s python client took as well? For
>>>>>> example, the k8s
>>>>>> >>>>>>> python client version 12.0.1 targets k8s api version 1.16.15.
>>>>>> >>>>>>>
>>>>>> >>>>>>> I am also with you that clients for different languages
>>>>>> should have
>>>>>> >>>>>>> their own release cycles. Other than testing burdens, every
>>>>>> client
>>>>>> >>>>>>> could introduce their own bug fixes and breaking changes
>>>>>> without
>>>>>> >>>>>>> having to trigger releases for all other clients. It would be
>>>>>> a waste
>>>>>> >>>>>>> of work for us to bump the python client version just for a
>>>>>> bug fix in
>>>>>> >>>>>>> the go client.
>>>>>> >>>>>>>
>>>>>> >>>>>>> Thanks,
>>>>>> >>>>>>> QP
>>>>>> >>>>>>>
>>>>>> >>>>>>> On Tue, Mar 30, 2021 at 1:56 AM Sumit Maheshwari <
>>>>>> msumit@apache.org> wrote:
>>>>>> >>>>>>> >
>>>>>> >>>>>>> > Hello Airflow dev community
>>>>>> >>>>>>> >
>>>>>> >>>>>>> > I'm writing this mail in regards to seek feedback on the
>>>>>> first release of Apache Airflow's Python client. This is an icebreaker
>>>>>> email only and we will follow the Apache process of releasing the version
>>>>>> later on. As of now, I've raised a small PR to add the missing files in the
>>>>>> repo, which are required to do the release of the client lib.
>>>>>> >>>>>>> >
>>>>>> >>>>>>> > The main question I've in mind about the versioning. I see
>>>>>> that there are mainly two theories related to maintain the release versions
>>>>>> of clients:
>>>>>> >>>>>>> >
>>>>>> >>>>>>> > The first is about following the semantic release of the
>>>>>> clients on its own. The release version of the client doesn't include the
>>>>>> version of the software itself. Something like we are doing with Airflow's
>>>>>> provider packages. However, in each release, we mention the release version
>>>>>> of the main software (Airflow in this case). This is the simple most way of
>>>>>> maintaining releases and followed majorly.
>>>>>> >>>>>>> > The second way is to include the software's version as well
>>>>>> in its client's versioning, something like it's been done in the K8s Python
>>>>>> client. In this approach, the client version provides more implicit
>>>>>> knowledge about the compatible software, however it creates an extra burden
>>>>>> of releasing the client with every minor release of the software, otherwise
>>>>>> the versioning would fall apart. Also, there is a good chance that most of
>>>>>> these minor or even a major release has no changes in the APIs, but the
>>>>>> client would be needed to re-released nevertheless, just to keep the
>>>>>> versioning in check.
>>>>>> >>>>>>> >
>>>>>> >>>>>>> > Personally, I'm in favour of the 1st way, as it's simple to
>>>>>> manage and doesn't tightly couple client with its software.
>>>>>> >>>>>>> >
>>>>>> >>>>>>> > Another unrelated question I have is about the release of
>>>>>> different client libraries. As of now, Airflow has Python and Go clients,
>>>>>> so should we try to release them together or they should follow their own
>>>>>> paths? In general, I'm against batching them together, cause the person who
>>>>>> is releasing them may not be able to test all the different clients in the
>>>>>> future.
>>>>>> >>>>>>> >
>>>>>> >>>>>>> > Let me know your thoughts on these 2 points.
>>>>>> >>>>>>> >
>>>>>> >>>>>>> > Thanks,
>>>>>> >>>>>>> > Sumit Maheshwari
>>>>>> >>>>>>> > PMC Apache Airflow
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> --
>>>>>> >> +48 660 796 129
>>>>>>
>>>>>>

Re: Releasing Apache Airflow's Python Client

[Sumit Maheshwari <su...@gmail.com>]

I think I'm fine with the approach you suggested Jarek. This approach
mitigates the need to release clients with each patch release of Airflow,
and that does make sense.

I hope that others are fine as well with the approach, where we've to
release clients with every major or minor release of Airflow.

On Mon, Apr 5, 2021 at 5:31 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> Hmm. I think it's a great direction, but 4 digits is a bit too much :)
> keeping three-digit SemVer is good for clarity/tools. And itroducing 4th
> digit is not really needed i think.
>
> Your proposal Sumit made me think,  and I asked myself a few questions:
>
> 1) will we ever release API client new feature without releasing that
> feature in airflow ? I think not. The API client should provide API calls
> for all the endpoints available and any change in those endpoints or adding
> new one  - by definition - should mean a bump in airflow minor version at
> least.
>
> 2) is it likely we release a bugfix to a single API client only without
> releasing Airflow new bugfix? I think so. We might upgrade generator for
> one client only to fix a particular problem we find for that language.
>
> 3) is it likely we release airflow bugfix without the need of releasing
> any of the clients? Yeah it's very likely the client will work without any
> changes with the new Airflow bugfix version.
>
> ... thinking ....
>
> So, how about a hybrid solution where we also include the minor version of
> Airflow in the client version
>
> Say airflow 2.1.x will work only with client 2.1.n but x and n are
> independent ?
>
> Example:
>
> We release 2.1.0 of Airflow and 2.1.0 of the Python and Java clients.
>
> Then we release 2.1.1, 2.1.2, 2.1.3 of Airflow. No release of clients.
>
> Then we find a bug in the Python generator, upgrade it and fix it in 2.1.1
> version of the Python client. But the Java client might still be at 2.1.0
> as the generator for Java worked fine.
>
>  At this point everyone who uses 2.1 airflow SHOULD upgrade to 2.1.3 of
> Airflow and 2.1.1 of the Python client and 2.1.0 of the Java client. If
> anyone uses any other combination, they should first upgrade to those
> latest 2.1 ones before 'complaining' that there are some problems.
>
> Next - whenever we release 2.2.0 of Airflow we also release 2.2.0 of all
> clients. And our client  check if Airflow is the same major/minor version
> and fails if it is not. So you cannot use 2.1.1 Python client with 2.0
> Airflow - you must switch to 2.2.0.
>
> I think that might be the nicest approach for both users and maintainers.
> In this case we do not even have to maintain the 'compatibility' matrix i
> mentioned before. And we keep 100% Semver semantics. And we can add
> automated tests for all the 'last X.Y client + airflow' combination as
> there is always only one 'blessed' combination airflow + clients that is
> 'supposed' to work.
>
> WDYT?
>
> J.
>
>
>
>
> pon., 5 kwi 2021, 08:09 użytkownik Sumit Maheshwari <
> sumeet.manit@gmail.com> napisał:
>
>> Thanks, everyone for the inputs.
>>
>> I think it's clear that we want to go with the hybrid way of versioning.
>> Given that we can automate the client releases easily, how if we release a
>> client automatically with each Airflow release, and go for a four numbered
>> release, where the first three would be the Airflow's release number and
>> the fourth one would be specific to the client, something like this
>>
>> *Airflow 2.0.1 <=> Client 2.0.1.0* -- Initial release
>> *Airflow 2.0.1 <=> Client 2.0.1.1* -- Some fix done in client (treating
>> major or minor same for client, ideally there won't be any major patches in
>> client)
>>
>> The upside of this process is that we can automate it & the client
>> version will be in sync with the Airflow version always. The downside that
>> we would be releasing more packages than required.
>>
>> About Vikram's question of not supporting any APIs in the client, AFAIK
>> there are none.
>>
>> On Mon, Apr 5, 2021 at 2:51 AM Vikram Koka <vi...@astronomer.io.invalid>
>> wrote:
>>
>>> Sumit,
>>> Thank you for bringing up this discussion.
>>>
>>> Reading through the thread, I am definitely far more comfortable with
>>> the hybrid approach proposed by Kaxil, than the first approach, primarily
>>> because of the same concerns articulated by Jarek regarding end user
>>> experience. Prior to that, I was leaning towards the second approach.
>>>
>>> Having said that, I think we need to be sure to follow certain
>>> guidelines with Core Airflow to make sure this approach works consistently,
>>> with a minimum of documentation needed and documentation to be read by
>>> Airflow users. Since there are generated clients, I am specifically
>>> referring to changes within Airflow regarding the API. Most importantly, I
>>> think we need to have any API additions, whether a new endpoint or an
>>> enhancement to an existing endpoint, only be available as part of a
>>> "feature release", rather than a "patch release". And that, this should
>>> trigger an update to the generated clients, triggering a new version of the
>>> same. This may be obvious, because we agreed on following semantic
>>> versioning strictly, but I thought it was worth reiterating.
>>>
>>> Having said that, I think it is worth asking the question if that is
>>> actually true, or if there are any APIs which are not  (or should not be)
>>> supported in the clients?
>>> I didn't think so, but wanted to make sure.
>>>
>>> Best regards,
>>> Vikram
>>>
>>>
>>> On Sat, Apr 3, 2021 at 4:11 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>>> I also like the hybrid approach proposed by Kaxil. It is not as
>>>> simplenfor dependency management as 1-1 version, but IMHO it is
>>>> simple-enough. As long as we strictly follow SemVer, a number of questions
>>>> i listed above from the user, are answered:
>>>>
>>>>
>>>> 1) Will my client be compatible with the version of Airflow ?
>>>>
>>>> Easy: 2.x.y will be compatible with 2.n.m of Airflow and so on.. Since
>>>> we will not have breaking changes, the 'broad compatibility' level should
>>>> be fine (with some feature -level differences but no general compatibility
>>>> problems)
>>>>
>>>> 2) Will feature (API endpoint) x available in this version of client be
>>>> working with the version of Airflow I have?
>>>>
>>>> We have to make sure that features are documented in the APIs at the
>>>> level of '@since' if we keep it updated the form 'This feature is available
>>>> in  'Airflow 2.3.*' and 'API 2.1.*' - we are covered. Since there will be
>>>> no breaking changes in neither API nor Airflow - this will be enough.
>>>>
>>>>
>>>> 3) Which version of Airflow should I install in order to use feature X
>>>> that I have in this new version of client ?
>>>>
>>>> I think this might be possible to generate such compatibility matrix
>>>> for all new API features added since Airflow 2.0 - it might be
>>>> automatically generated from the API specification in this case. If we
>>>> parse the API definition and have the right 'since' definition for both
>>>> Airflow and API, it should be rather easy to generate such table:
>>>>
>>>> Feature       | Min API version | Min Airflow version
>>>> MoonShot feature X  | 2.3 | 2.2
>>>>
>>>> Having such compatibility matric would, I think, be a great help for
>>>> the users i think when they would plan their Airflow/API releases
>>>>
>>>>
>>>> J.
>>>>
>>>> sob., 3 kwi 2021, 12:06 użytkownik .... <dz...@gmail.com> napisał:
>>>>
>>>>> +1 for Kaxil proposal
>>>>>
>>>>> I think, attempting to assign a specific client version to the library
>>>>> application makes no sense in creating an API that is based on a
>>>>> strict specification.
>>>>>
>>>>> One of the goals of this API was to make it easier to manage multiple
>>>>> instances, and it's not often that you have an identically homogeneous
>>>>> environment. It's natural to have different versions of Airflow in one
>>>>> organization. If we want users to use a specific version of the
>>>>> library on the client-side, it will probably not be possible in many
>>>>> cases.
>>>>>
>>>>> pt., 2 kwi 2021 o 23:35 Kaxil Naik <ka...@gmail.com> napisał(a):
>>>>> >
>>>>> > Thanks, Sumit, for bringing this up.
>>>>> >
>>>>> > re: versioning for Python client:
>>>>> >
>>>>> > I favour a bit of a hybrid of 1 and 2. Initially, I favored just (1)
>>>>> when we talked but after seeing how Elastic-Search does it I liked their
>>>>> versioning strategy.
>>>>> > How their python-client is compatible with ES itself is documented
>>>>> at https://elasticsearch-py.readthedocs.io/en/v7.12.0/#compatibility.
>>>>> >
>>>>> > Since Airflow from 2.0 strictly follows SemVer we can continue with
>>>>> the assumption that we (Airflow) won't break backwards-compatibility (to
>>>>> the best of our ability).
>>>>> > So we can have M.x.y as our versioning for our client where M is the
>>>>> Major Version of the client that matches the Major version of Airflow; x is
>>>>> the minor version for the client & y is the patch release. Here x & y are
>>>>> independent of Airflow version. Our goal and message to user would be that
>>>>> "You have to use a version of client library matching the major version of
>>>>> Airflow"
>>>>> >
>>>>> > re: the release of different client libraries:
>>>>> >
>>>>> > Client for each language should be versioned & released separately.
>>>>> >
>>>>> > Regards,
>>>>> > Kaxil
>>>>> >
>>>>> > On Wed, Mar 31, 2021 at 7:01 PM Jarek Potiuk <ja...@potiuk.com>
>>>>> wrote:
>>>>> >>
>>>>> >> I think there are more questions we need to answer when it comes to
>>>>> different versions. If we are going to different versioning for agent and
>>>>> Airflow, we will have to accept the fact that over time people will have a
>>>>> "matrix of versions" - for both Airflow and the client. And this introduces
>>>>> "dependency mess" if we do not handle it well.
>>>>> >>
>>>>> >> I think in this we should give the user much clearer indication on
>>>>> which version of Airflow the client should work on, and (more important)
>>>>> which version it does NOT work on and which features will NOT be available.
>>>>> Both cases are possible.
>>>>> >>
>>>>> >> There should be some way of knowing for the user, (in case there is
>>>>> a backwards incompatibility or in case some features are not available)
>>>>> that a given version of the client does not work with the version of
>>>>> Airflow that the user has.
>>>>> >>
>>>>> >> I think users looking at the documentation should be able to be
>>>>> able to figure out:
>>>>> >>
>>>>> >> 1) Will my client be compatible with the version of Airflow ?
>>>>> >> 2) Will feature (API endpoint) x available in this version of
>>>>> client be working with the version of Airflow I have?
>>>>> >> 3) Which version of Airflow should I install in order to use
>>>>> feature X that I have in this new version of client ?
>>>>> >>
>>>>> >> Similarly proper error messages should be displayed if there is
>>>>> incompatibility or when a feature X is not available when you try to use a
>>>>> version of client that supports it with Airflow version of Agent that does
>>>>> not have it.
>>>>> >>
>>>>> >> So I'd say - separate versioning is fine as long as we handle the
>>>>> above.
>>>>> >>
>>>>> >>
>>>>> >> Saying all that - there is a beautiful "maintenance" simplicity if
>>>>> we actually synchronize the version and we say "Always use SAME version of
>>>>> client as Airflow" (and we fail the client if the versions is different).
>>>>> This is way more simple in communicating to the users, and with most
>>>>> deployments, I think upgrading airflow and clients at the same time should
>>>>> be no problem.
>>>>> >>
>>>>> >> I think over time as the matrix of potential Airflow/Client
>>>>> combinations will grow this might mean bigger number of issues that we have
>>>>> to handle ("Why ma client does not do this and this (and of course it will
>>>>> not have neither client nor airflow version specified and we will have to
>>>>> ask for clarification).
>>>>> >>
>>>>> >> So as I see it - maybe release-wise it's a little redundant and
>>>>> you'd have to release a bugfix version of Airflow if you find a critical
>>>>> problem in Client (which should be rare as agent is mostly automatically
>>>>> generated), but synchronizing versions is very tempting from the
>>>>> "supportability" point of view.
>>>>> >>
>>>>> >> Those are my 2 cents, I was not involved too much in the client
>>>>> part, so I do not feel like I have a strong voice in there, but if I would
>>>>> vote between the two - I'd choose synchronizing the releases.
>>>>> >>
>>>>> >> BTW. In providers we have a very different story - we keep
>>>>> dependency to airflow via PIP (which we can set it, if we find a provider
>>>>> stops being compatible with a version of Airflow). Also we automatically
>>>>> test in CI if the provider installs and imports on Airflow 2.0.0 to verify
>>>>> that it is backwards compatible (at least to the level of importability).
>>>>> This has already saved us from  "accidental" incompatibilities (wit yaml
>>>>> package import change recently). So there, separate versioning makes much
>>>>> more sense - also because the version of the provider relates exclusively
>>>>> to the feature that the provider version has, not Airflow itself). I think
>>>>> this is quite a different story from the client, which is very closely
>>>>> coupled with Airflow.
>>>>> >>
>>>>> >> J.
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> On Wed, Mar 31, 2021 at 11:08 AM Deng Xiaodong <xd...@gmail.com>
>>>>> wrote:
>>>>> >>>
>>>>> >>> Yep, if we can, I would prefer to explicitly mention something
>>>>> like 2.0.X :)
>>>>> >>>
>>>>> >>> On Wed, Mar 31, 2021 at 10:54 AM Sumit Maheshwari <
>>>>> sumeet.manit@gmail.com> wrote:
>>>>> >>>>
>>>>> >>>> Yeah totally agrees with you XD about the range vs the fixed
>>>>> version of the main software. And with Airflow 2 dot & beyond, we would be
>>>>> following the semantic versioning more judiciously, so with that can we
>>>>> assume that users would be cognizant of the related Airflow version
>>>>> implicitly, or do you think we should mention something like 2.0.X in the
>>>>> changelog explicitly?
>>>>> >>>>
>>>>> >>>> On Wed, Mar 31, 2021 at 1:38 PM Deng Xiaodong <
>>>>> xd.deng.r@gmail.com> wrote:
>>>>> >>>>>
>>>>> >>>>> Thanks, Sumit, for bringing this up.
>>>>> >>>>>
>>>>> >>>>> - Regarding versioning for Python client: I favour the 1st way
>>>>> as well. The only minor comment I have is: instead of "mention the release
>>>>> version of the main software", possibly what users would expect is a "range
>>>>> of compatible versions of main software". If we follow semver for Airflow
>>>>> itself relatively strictly, this should not be hard.
>>>>> >>>>>
>>>>> >>>>> - Regarding the release of different client libraries: I agree
>>>>> with you. They should be versioned & released separately.
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>> Regards,
>>>>> >>>>> XD
>>>>> >>>>>
>>>>> >>>>> On Wed, Mar 31, 2021 at 9:21 AM Sumit Maheshwari <
>>>>> sumeet.manit@gmail.com> wrote:
>>>>> >>>>>>
>>>>> >>>>>> Thanks QP for your feedback. I think the latest release of K8s
>>>>> client (v17.x.y) is moving in another direction.
>>>>> >>>>>>
>>>>> >>>>>> On Wed, Mar 31, 2021 at 12:13 AM QP Hou <qp...@scribd.com.invalid>
>>>>> wrote:
>>>>> >>>>>>>
>>>>> >>>>>>> Thanks Sumit for kicking off this discussion!
>>>>> >>>>>>>
>>>>> >>>>>>> I am in favor of approach #1 as well. Airflow software version
>>>>> can
>>>>> >>>>>>> change due to none API related changes, but the client should
>>>>> only
>>>>> >>>>>>> care about the public API contract. I believe this is also the
>>>>> same
>>>>> >>>>>>> approach that the k8s python client took as well? For example,
>>>>> the k8s
>>>>> >>>>>>> python client version 12.0.1 targets k8s api version 1.16.15.
>>>>> >>>>>>>
>>>>> >>>>>>> I am also with you that clients for different languages should
>>>>> have
>>>>> >>>>>>> their own release cycles. Other than testing burdens, every
>>>>> client
>>>>> >>>>>>> could introduce their own bug fixes and breaking changes
>>>>> without
>>>>> >>>>>>> having to trigger releases for all other clients. It would be
>>>>> a waste
>>>>> >>>>>>> of work for us to bump the python client version just for a
>>>>> bug fix in
>>>>> >>>>>>> the go client.
>>>>> >>>>>>>
>>>>> >>>>>>> Thanks,
>>>>> >>>>>>> QP
>>>>> >>>>>>>
>>>>> >>>>>>> On Tue, Mar 30, 2021 at 1:56 AM Sumit Maheshwari <
>>>>> msumit@apache.org> wrote:
>>>>> >>>>>>> >
>>>>> >>>>>>> > Hello Airflow dev community
>>>>> >>>>>>> >
>>>>> >>>>>>> > I'm writing this mail in regards to seek feedback on the
>>>>> first release of Apache Airflow's Python client. This is an icebreaker
>>>>> email only and we will follow the Apache process of releasing the version
>>>>> later on. As of now, I've raised a small PR to add the missing files in the
>>>>> repo, which are required to do the release of the client lib.
>>>>> >>>>>>> >
>>>>> >>>>>>> > The main question I've in mind about the versioning. I see
>>>>> that there are mainly two theories related to maintain the release versions
>>>>> of clients:
>>>>> >>>>>>> >
>>>>> >>>>>>> > The first is about following the semantic release of the
>>>>> clients on its own. The release version of the client doesn't include the
>>>>> version of the software itself. Something like we are doing with Airflow's
>>>>> provider packages. However, in each release, we mention the release version
>>>>> of the main software (Airflow in this case). This is the simple most way of
>>>>> maintaining releases and followed majorly.
>>>>> >>>>>>> > The second way is to include the software's version as well
>>>>> in its client's versioning, something like it's been done in the K8s Python
>>>>> client. In this approach, the client version provides more implicit
>>>>> knowledge about the compatible software, however it creates an extra burden
>>>>> of releasing the client with every minor release of the software, otherwise
>>>>> the versioning would fall apart. Also, there is a good chance that most of
>>>>> these minor or even a major release has no changes in the APIs, but the
>>>>> client would be needed to re-released nevertheless, just to keep the
>>>>> versioning in check.
>>>>> >>>>>>> >
>>>>> >>>>>>> > Personally, I'm in favour of the 1st way, as it's simple to
>>>>> manage and doesn't tightly couple client with its software.
>>>>> >>>>>>> >
>>>>> >>>>>>> > Another unrelated question I have is about the release of
>>>>> different client libraries. As of now, Airflow has Python and Go clients,
>>>>> so should we try to release them together or they should follow their own
>>>>> paths? In general, I'm against batching them together, cause the person who
>>>>> is releasing them may not be able to test all the different clients in the
>>>>> future.
>>>>> >>>>>>> >
>>>>> >>>>>>> > Let me know your thoughts on these 2 points.
>>>>> >>>>>>> >
>>>>> >>>>>>> > Thanks,
>>>>> >>>>>>> > Sumit Maheshwari
>>>>> >>>>>>> > PMC Apache Airflow
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> +48 660 796 129
>>>>>
>>>>>

Re: Releasing Apache Airflow's Python Client

[Jarek Potiuk <ja...@potiuk.com>]

Hmm. I think it's a great direction, but 4 digits is a bit too much :)
keeping three-digit SemVer is good for clarity/tools. And itroducing 4th
digit is not really needed i think.

Your proposal Sumit made me think,  and I asked myself a few questions:

1) will we ever release API client new feature without releasing that
feature in airflow ? I think not. The API client should provide API calls
for all the endpoints available and any change in those endpoints or adding
new one  - by definition - should mean a bump in airflow minor version at
least.

2) is it likely we release a bugfix to a single API client only without
releasing Airflow new bugfix? I think so. We might upgrade generator for
one client only to fix a particular problem we find for that language.

3) is it likely we release airflow bugfix without the need of releasing any
of the clients? Yeah it's very likely the client will work without any
changes with the new Airflow bugfix version.

... thinking ....

So, how about a hybrid solution where we also include the minor version of
Airflow in the client version

Say airflow 2.1.x will work only with client 2.1.n but x and n are
independent ?

Example:

We release 2.1.0 of Airflow and 2.1.0 of the Python and Java clients.

Then we release 2.1.1, 2.1.2, 2.1.3 of Airflow. No release of clients.

Then we find a bug in the Python generator, upgrade it and fix it in 2.1.1
version of the Python client. But the Java client might still be at 2.1.0
as the generator for Java worked fine.

 At this point everyone who uses 2.1 airflow SHOULD upgrade to 2.1.3 of
Airflow and 2.1.1 of the Python client and 2.1.0 of the Java client. If
anyone uses any other combination, they should first upgrade to those
latest 2.1 ones before 'complaining' that there are some problems.

Next - whenever we release 2.2.0 of Airflow we also release 2.2.0 of all
clients. And our client  check if Airflow is the same major/minor version
and fails if it is not. So you cannot use 2.1.1 Python client with 2.0
Airflow - you must switch to 2.2.0.

I think that might be the nicest approach for both users and maintainers.
In this case we do not even have to maintain the 'compatibility' matrix i
mentioned before. And we keep 100% Semver semantics. And we can add
automated tests for all the 'last X.Y client + airflow' combination as
there is always only one 'blessed' combination airflow + clients that is
'supposed' to work.

WDYT?

J.




pon., 5 kwi 2021, 08:09 użytkownik Sumit Maheshwari <su...@gmail.com>
napisał:

> Thanks, everyone for the inputs.
>
> I think it's clear that we want to go with the hybrid way of versioning.
> Given that we can automate the client releases easily, how if we release a
> client automatically with each Airflow release, and go for a four numbered
> release, where the first three would be the Airflow's release number and
> the fourth one would be specific to the client, something like this
>
> *Airflow 2.0.1 <=> Client 2.0.1.0* -- Initial release
> *Airflow 2.0.1 <=> Client 2.0.1.1* -- Some fix done in client (treating
> major or minor same for client, ideally there won't be any major patches in
> client)
>
> The upside of this process is that we can automate it & the client version
> will be in sync with the Airflow version always. The downside that we would
> be releasing more packages than required.
>
> About Vikram's question of not supporting any APIs in the client, AFAIK
> there are none.
>
> On Mon, Apr 5, 2021 at 2:51 AM Vikram Koka <vi...@astronomer.io.invalid>
> wrote:
>
>> Sumit,
>> Thank you for bringing up this discussion.
>>
>> Reading through the thread, I am definitely far more comfortable with the
>> hybrid approach proposed by Kaxil, than the first approach, primarily
>> because of the same concerns articulated by Jarek regarding end user
>> experience. Prior to that, I was leaning towards the second approach.
>>
>> Having said that, I think we need to be sure to follow certain guidelines
>> with Core Airflow to make sure this approach works consistently, with a
>> minimum of documentation needed and documentation to be read by Airflow
>> users. Since there are generated clients, I am specifically referring to
>> changes within Airflow regarding the API. Most importantly, I think we need
>> to have any API additions, whether a new endpoint or an enhancement to an
>> existing endpoint, only be available as part of a "feature release", rather
>> than a "patch release". And that, this should trigger an update to the
>> generated clients, triggering a new version of the same. This may be
>> obvious, because we agreed on following semantic versioning strictly, but I
>> thought it was worth reiterating.
>>
>> Having said that, I think it is worth asking the question if that is
>> actually true, or if there are any APIs which are not  (or should not be)
>> supported in the clients?
>> I didn't think so, but wanted to make sure.
>>
>> Best regards,
>> Vikram
>>
>>
>> On Sat, Apr 3, 2021 at 4:11 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>>> I also like the hybrid approach proposed by Kaxil. It is not as
>>> simplenfor dependency management as 1-1 version, but IMHO it is
>>> simple-enough. As long as we strictly follow SemVer, a number of questions
>>> i listed above from the user, are answered:
>>>
>>>
>>> 1) Will my client be compatible with the version of Airflow ?
>>>
>>> Easy: 2.x.y will be compatible with 2.n.m of Airflow and so on.. Since
>>> we will not have breaking changes, the 'broad compatibility' level should
>>> be fine (with some feature -level differences but no general compatibility
>>> problems)
>>>
>>> 2) Will feature (API endpoint) x available in this version of client be
>>> working with the version of Airflow I have?
>>>
>>> We have to make sure that features are documented in the APIs at the
>>> level of '@since' if we keep it updated the form 'This feature is available
>>> in  'Airflow 2.3.*' and 'API 2.1.*' - we are covered. Since there will be
>>> no breaking changes in neither API nor Airflow - this will be enough.
>>>
>>>
>>> 3) Which version of Airflow should I install in order to use feature X
>>> that I have in this new version of client ?
>>>
>>> I think this might be possible to generate such compatibility matrix for
>>> all new API features added since Airflow 2.0 - it might be automatically
>>> generated from the API specification in this case. If we parse the API
>>> definition and have the right 'since' definition for both Airflow and API,
>>> it should be rather easy to generate such table:
>>>
>>> Feature       | Min API version | Min Airflow version
>>> MoonShot feature X  | 2.3 | 2.2
>>>
>>> Having such compatibility matric would, I think, be a great help for the
>>> users i think when they would plan their Airflow/API releases
>>>
>>>
>>> J.
>>>
>>> sob., 3 kwi 2021, 12:06 użytkownik .... <dz...@gmail.com> napisał:
>>>
>>>> +1 for Kaxil proposal
>>>>
>>>> I think, attempting to assign a specific client version to the library
>>>> application makes no sense in creating an API that is based on a
>>>> strict specification.
>>>>
>>>> One of the goals of this API was to make it easier to manage multiple
>>>> instances, and it's not often that you have an identically homogeneous
>>>> environment. It's natural to have different versions of Airflow in one
>>>> organization. If we want users to use a specific version of the
>>>> library on the client-side, it will probably not be possible in many
>>>> cases.
>>>>
>>>> pt., 2 kwi 2021 o 23:35 Kaxil Naik <ka...@gmail.com> napisał(a):
>>>> >
>>>> > Thanks, Sumit, for bringing this up.
>>>> >
>>>> > re: versioning for Python client:
>>>> >
>>>> > I favour a bit of a hybrid of 1 and 2. Initially, I favored just (1)
>>>> when we talked but after seeing how Elastic-Search does it I liked their
>>>> versioning strategy.
>>>> > How their python-client is compatible with ES itself is documented at
>>>> https://elasticsearch-py.readthedocs.io/en/v7.12.0/#compatibility.
>>>> >
>>>> > Since Airflow from 2.0 strictly follows SemVer we can continue with
>>>> the assumption that we (Airflow) won't break backwards-compatibility (to
>>>> the best of our ability).
>>>> > So we can have M.x.y as our versioning for our client where M is the
>>>> Major Version of the client that matches the Major version of Airflow; x is
>>>> the minor version for the client & y is the patch release. Here x & y are
>>>> independent of Airflow version. Our goal and message to user would be that
>>>> "You have to use a version of client library matching the major version of
>>>> Airflow"
>>>> >
>>>> > re: the release of different client libraries:
>>>> >
>>>> > Client for each language should be versioned & released separately.
>>>> >
>>>> > Regards,
>>>> > Kaxil
>>>> >
>>>> > On Wed, Mar 31, 2021 at 7:01 PM Jarek Potiuk <ja...@potiuk.com>
>>>> wrote:
>>>> >>
>>>> >> I think there are more questions we need to answer when it comes to
>>>> different versions. If we are going to different versioning for agent and
>>>> Airflow, we will have to accept the fact that over time people will have a
>>>> "matrix of versions" - for both Airflow and the client. And this introduces
>>>> "dependency mess" if we do not handle it well.
>>>> >>
>>>> >> I think in this we should give the user much clearer indication on
>>>> which version of Airflow the client should work on, and (more important)
>>>> which version it does NOT work on and which features will NOT be available.
>>>> Both cases are possible.
>>>> >>
>>>> >> There should be some way of knowing for the user, (in case there is
>>>> a backwards incompatibility or in case some features are not available)
>>>> that a given version of the client does not work with the version of
>>>> Airflow that the user has.
>>>> >>
>>>> >> I think users looking at the documentation should be able to be able
>>>> to figure out:
>>>> >>
>>>> >> 1) Will my client be compatible with the version of Airflow ?
>>>> >> 2) Will feature (API endpoint) x available in this version of client
>>>> be working with the version of Airflow I have?
>>>> >> 3) Which version of Airflow should I install in order to use feature
>>>> X that I have in this new version of client ?
>>>> >>
>>>> >> Similarly proper error messages should be displayed if there is
>>>> incompatibility or when a feature X is not available when you try to use a
>>>> version of client that supports it with Airflow version of Agent that does
>>>> not have it.
>>>> >>
>>>> >> So I'd say - separate versioning is fine as long as we handle the
>>>> above.
>>>> >>
>>>> >>
>>>> >> Saying all that - there is a beautiful "maintenance" simplicity if
>>>> we actually synchronize the version and we say "Always use SAME version of
>>>> client as Airflow" (and we fail the client if the versions is different).
>>>> This is way more simple in communicating to the users, and with most
>>>> deployments, I think upgrading airflow and clients at the same time should
>>>> be no problem.
>>>> >>
>>>> >> I think over time as the matrix of potential Airflow/Client
>>>> combinations will grow this might mean bigger number of issues that we have
>>>> to handle ("Why ma client does not do this and this (and of course it will
>>>> not have neither client nor airflow version specified and we will have to
>>>> ask for clarification).
>>>> >>
>>>> >> So as I see it - maybe release-wise it's a little redundant and
>>>> you'd have to release a bugfix version of Airflow if you find a critical
>>>> problem in Client (which should be rare as agent is mostly automatically
>>>> generated), but synchronizing versions is very tempting from the
>>>> "supportability" point of view.
>>>> >>
>>>> >> Those are my 2 cents, I was not involved too much in the client
>>>> part, so I do not feel like I have a strong voice in there, but if I would
>>>> vote between the two - I'd choose synchronizing the releases.
>>>> >>
>>>> >> BTW. In providers we have a very different story - we keep
>>>> dependency to airflow via PIP (which we can set it, if we find a provider
>>>> stops being compatible with a version of Airflow). Also we automatically
>>>> test in CI if the provider installs and imports on Airflow 2.0.0 to verify
>>>> that it is backwards compatible (at least to the level of importability).
>>>> This has already saved us from  "accidental" incompatibilities (wit yaml
>>>> package import change recently). So there, separate versioning makes much
>>>> more sense - also because the version of the provider relates exclusively
>>>> to the feature that the provider version has, not Airflow itself). I think
>>>> this is quite a different story from the client, which is very closely
>>>> coupled with Airflow.
>>>> >>
>>>> >> J.
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Wed, Mar 31, 2021 at 11:08 AM Deng Xiaodong <xd...@gmail.com>
>>>> wrote:
>>>> >>>
>>>> >>> Yep, if we can, I would prefer to explicitly mention something like
>>>> 2.0.X :)
>>>> >>>
>>>> >>> On Wed, Mar 31, 2021 at 10:54 AM Sumit Maheshwari <
>>>> sumeet.manit@gmail.com> wrote:
>>>> >>>>
>>>> >>>> Yeah totally agrees with you XD about the range vs the fixed
>>>> version of the main software. And with Airflow 2 dot & beyond, we would be
>>>> following the semantic versioning more judiciously, so with that can we
>>>> assume that users would be cognizant of the related Airflow version
>>>> implicitly, or do you think we should mention something like 2.0.X in the
>>>> changelog explicitly?
>>>> >>>>
>>>> >>>> On Wed, Mar 31, 2021 at 1:38 PM Deng Xiaodong <xd...@gmail.com>
>>>> wrote:
>>>> >>>>>
>>>> >>>>> Thanks, Sumit, for bringing this up.
>>>> >>>>>
>>>> >>>>> - Regarding versioning for Python client: I favour the 1st way as
>>>> well. The only minor comment I have is: instead of "mention the release
>>>> version of the main software", possibly what users would expect is a "range
>>>> of compatible versions of main software". If we follow semver for Airflow
>>>> itself relatively strictly, this should not be hard.
>>>> >>>>>
>>>> >>>>> - Regarding the release of different client libraries: I agree
>>>> with you. They should be versioned & released separately.
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> Regards,
>>>> >>>>> XD
>>>> >>>>>
>>>> >>>>> On Wed, Mar 31, 2021 at 9:21 AM Sumit Maheshwari <
>>>> sumeet.manit@gmail.com> wrote:
>>>> >>>>>>
>>>> >>>>>> Thanks QP for your feedback. I think the latest release of K8s
>>>> client (v17.x.y) is moving in another direction.
>>>> >>>>>>
>>>> >>>>>> On Wed, Mar 31, 2021 at 12:13 AM QP Hou <qp...@scribd.com.invalid>
>>>> wrote:
>>>> >>>>>>>
>>>> >>>>>>> Thanks Sumit for kicking off this discussion!
>>>> >>>>>>>
>>>> >>>>>>> I am in favor of approach #1 as well. Airflow software version
>>>> can
>>>> >>>>>>> change due to none API related changes, but the client should
>>>> only
>>>> >>>>>>> care about the public API contract. I believe this is also the
>>>> same
>>>> >>>>>>> approach that the k8s python client took as well? For example,
>>>> the k8s
>>>> >>>>>>> python client version 12.0.1 targets k8s api version 1.16.15.
>>>> >>>>>>>
>>>> >>>>>>> I am also with you that clients for different languages should
>>>> have
>>>> >>>>>>> their own release cycles. Other than testing burdens, every
>>>> client
>>>> >>>>>>> could introduce their own bug fixes and breaking changes without
>>>> >>>>>>> having to trigger releases for all other clients. It would be a
>>>> waste
>>>> >>>>>>> of work for us to bump the python client version just for a bug
>>>> fix in
>>>> >>>>>>> the go client.
>>>> >>>>>>>
>>>> >>>>>>> Thanks,
>>>> >>>>>>> QP
>>>> >>>>>>>
>>>> >>>>>>> On Tue, Mar 30, 2021 at 1:56 AM Sumit Maheshwari <
>>>> msumit@apache.org> wrote:
>>>> >>>>>>> >
>>>> >>>>>>> > Hello Airflow dev community
>>>> >>>>>>> >
>>>> >>>>>>> > I'm writing this mail in regards to seek feedback on the
>>>> first release of Apache Airflow's Python client. This is an icebreaker
>>>> email only and we will follow the Apache process of releasing the version
>>>> later on. As of now, I've raised a small PR to add the missing files in the
>>>> repo, which are required to do the release of the client lib.
>>>> >>>>>>> >
>>>> >>>>>>> > The main question I've in mind about the versioning. I see
>>>> that there are mainly two theories related to maintain the release versions
>>>> of clients:
>>>> >>>>>>> >
>>>> >>>>>>> > The first is about following the semantic release of the
>>>> clients on its own. The release version of the client doesn't include the
>>>> version of the software itself. Something like we are doing with Airflow's
>>>> provider packages. However, in each release, we mention the release version
>>>> of the main software (Airflow in this case). This is the simple most way of
>>>> maintaining releases and followed majorly.
>>>> >>>>>>> > The second way is to include the software's version as well
>>>> in its client's versioning, something like it's been done in the K8s Python
>>>> client. In this approach, the client version provides more implicit
>>>> knowledge about the compatible software, however it creates an extra burden
>>>> of releasing the client with every minor release of the software, otherwise
>>>> the versioning would fall apart. Also, there is a good chance that most of
>>>> these minor or even a major release has no changes in the APIs, but the
>>>> client would be needed to re-released nevertheless, just to keep the
>>>> versioning in check.
>>>> >>>>>>> >
>>>> >>>>>>> > Personally, I'm in favour of the 1st way, as it's simple to
>>>> manage and doesn't tightly couple client with its software.
>>>> >>>>>>> >
>>>> >>>>>>> > Another unrelated question I have is about the release of
>>>> different client libraries. As of now, Airflow has Python and Go clients,
>>>> so should we try to release them together or they should follow their own
>>>> paths? In general, I'm against batching them together, cause the person who
>>>> is releasing them may not be able to test all the different clients in the
>>>> future.
>>>> >>>>>>> >
>>>> >>>>>>> > Let me know your thoughts on these 2 points.
>>>> >>>>>>> >
>>>> >>>>>>> > Thanks,
>>>> >>>>>>> > Sumit Maheshwari
>>>> >>>>>>> > PMC Apache Airflow
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> +48 660 796 129
>>>>
>>>>

Re: Releasing Apache Airflow's Python Client

[Sumit Maheshwari <su...@gmail.com>]

Thanks, everyone for the inputs.

I think it's clear that we want to go with the hybrid way of versioning.
Given that we can automate the client releases easily, how if we release a
client automatically with each Airflow release, and go for a four numbered
release, where the first three would be the Airflow's release number and
the fourth one would be specific to the client, something like this

*Airflow 2.0.1 <=> Client 2.0.1.0* -- Initial release
*Airflow 2.0.1 <=> Client 2.0.1.1* -- Some fix done in client (treating
major or minor same for client, ideally there won't be any major patches in
client)

The upside of this process is that we can automate it & the client version
will be in sync with the Airflow version always. The downside that we would
be releasing more packages than required.

About Vikram's question of not supporting any APIs in the client, AFAIK
there are none.

On Mon, Apr 5, 2021 at 2:51 AM Vikram Koka <vi...@astronomer.io.invalid>
wrote:

> Sumit,
> Thank you for bringing up this discussion.
>
> Reading through the thread, I am definitely far more comfortable with the
> hybrid approach proposed by Kaxil, than the first approach, primarily
> because of the same concerns articulated by Jarek regarding end user
> experience. Prior to that, I was leaning towards the second approach.
>
> Having said that, I think we need to be sure to follow certain guidelines
> with Core Airflow to make sure this approach works consistently, with a
> minimum of documentation needed and documentation to be read by Airflow
> users. Since there are generated clients, I am specifically referring to
> changes within Airflow regarding the API. Most importantly, I think we need
> to have any API additions, whether a new endpoint or an enhancement to an
> existing endpoint, only be available as part of a "feature release", rather
> than a "patch release". And that, this should trigger an update to the
> generated clients, triggering a new version of the same. This may be
> obvious, because we agreed on following semantic versioning strictly, but I
> thought it was worth reiterating.
>
> Having said that, I think it is worth asking the question if that is
> actually true, or if there are any APIs which are not  (or should not be)
> supported in the clients?
> I didn't think so, but wanted to make sure.
>
> Best regards,
> Vikram
>
>
> On Sat, Apr 3, 2021 at 4:11 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> I also like the hybrid approach proposed by Kaxil. It is not as
>> simplenfor dependency management as 1-1 version, but IMHO it is
>> simple-enough. As long as we strictly follow SemVer, a number of questions
>> i listed above from the user, are answered:
>>
>>
>> 1) Will my client be compatible with the version of Airflow ?
>>
>> Easy: 2.x.y will be compatible with 2.n.m of Airflow and so on.. Since we
>> will not have breaking changes, the 'broad compatibility' level should be
>> fine (with some feature -level differences but no general compatibility
>> problems)
>>
>> 2) Will feature (API endpoint) x available in this version of client be
>> working with the version of Airflow I have?
>>
>> We have to make sure that features are documented in the APIs at the
>> level of '@since' if we keep it updated the form 'This feature is available
>> in  'Airflow 2.3.*' and 'API 2.1.*' - we are covered. Since there will be
>> no breaking changes in neither API nor Airflow - this will be enough.
>>
>>
>> 3) Which version of Airflow should I install in order to use feature X
>> that I have in this new version of client ?
>>
>> I think this might be possible to generate such compatibility matrix for
>> all new API features added since Airflow 2.0 - it might be automatically
>> generated from the API specification in this case. If we parse the API
>> definition and have the right 'since' definition for both Airflow and API,
>> it should be rather easy to generate such table:
>>
>> Feature       | Min API version | Min Airflow version
>> MoonShot feature X  | 2.3 | 2.2
>>
>> Having such compatibility matric would, I think, be a great help for the
>> users i think when they would plan their Airflow/API releases
>>
>>
>> J.
>>
>> sob., 3 kwi 2021, 12:06 użytkownik .... <dz...@gmail.com> napisał:
>>
>>> +1 for Kaxil proposal
>>>
>>> I think, attempting to assign a specific client version to the library
>>> application makes no sense in creating an API that is based on a
>>> strict specification.
>>>
>>> One of the goals of this API was to make it easier to manage multiple
>>> instances, and it's not often that you have an identically homogeneous
>>> environment. It's natural to have different versions of Airflow in one
>>> organization. If we want users to use a specific version of the
>>> library on the client-side, it will probably not be possible in many
>>> cases.
>>>
>>> pt., 2 kwi 2021 o 23:35 Kaxil Naik <ka...@gmail.com> napisał(a):
>>> >
>>> > Thanks, Sumit, for bringing this up.
>>> >
>>> > re: versioning for Python client:
>>> >
>>> > I favour a bit of a hybrid of 1 and 2. Initially, I favored just (1)
>>> when we talked but after seeing how Elastic-Search does it I liked their
>>> versioning strategy.
>>> > How their python-client is compatible with ES itself is documented at
>>> https://elasticsearch-py.readthedocs.io/en/v7.12.0/#compatibility.
>>> >
>>> > Since Airflow from 2.0 strictly follows SemVer we can continue with
>>> the assumption that we (Airflow) won't break backwards-compatibility (to
>>> the best of our ability).
>>> > So we can have M.x.y as our versioning for our client where M is the
>>> Major Version of the client that matches the Major version of Airflow; x is
>>> the minor version for the client & y is the patch release. Here x & y are
>>> independent of Airflow version. Our goal and message to user would be that
>>> "You have to use a version of client library matching the major version of
>>> Airflow"
>>> >
>>> > re: the release of different client libraries:
>>> >
>>> > Client for each language should be versioned & released separately.
>>> >
>>> > Regards,
>>> > Kaxil
>>> >
>>> > On Wed, Mar 31, 2021 at 7:01 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>> >>
>>> >> I think there are more questions we need to answer when it comes to
>>> different versions. If we are going to different versioning for agent and
>>> Airflow, we will have to accept the fact that over time people will have a
>>> "matrix of versions" - for both Airflow and the client. And this introduces
>>> "dependency mess" if we do not handle it well.
>>> >>
>>> >> I think in this we should give the user much clearer indication on
>>> which version of Airflow the client should work on, and (more important)
>>> which version it does NOT work on and which features will NOT be available.
>>> Both cases are possible.
>>> >>
>>> >> There should be some way of knowing for the user, (in case there is a
>>> backwards incompatibility or in case some features are not available) that
>>> a given version of the client does not work with the version of Airflow
>>> that the user has.
>>> >>
>>> >> I think users looking at the documentation should be able to be able
>>> to figure out:
>>> >>
>>> >> 1) Will my client be compatible with the version of Airflow ?
>>> >> 2) Will feature (API endpoint) x available in this version of client
>>> be working with the version of Airflow I have?
>>> >> 3) Which version of Airflow should I install in order to use feature
>>> X that I have in this new version of client ?
>>> >>
>>> >> Similarly proper error messages should be displayed if there is
>>> incompatibility or when a feature X is not available when you try to use a
>>> version of client that supports it with Airflow version of Agent that does
>>> not have it.
>>> >>
>>> >> So I'd say - separate versioning is fine as long as we handle the
>>> above.
>>> >>
>>> >>
>>> >> Saying all that - there is a beautiful "maintenance" simplicity if we
>>> actually synchronize the version and we say "Always use SAME version of
>>> client as Airflow" (and we fail the client if the versions is different).
>>> This is way more simple in communicating to the users, and with most
>>> deployments, I think upgrading airflow and clients at the same time should
>>> be no problem.
>>> >>
>>> >> I think over time as the matrix of potential Airflow/Client
>>> combinations will grow this might mean bigger number of issues that we have
>>> to handle ("Why ma client does not do this and this (and of course it will
>>> not have neither client nor airflow version specified and we will have to
>>> ask for clarification).
>>> >>
>>> >> So as I see it - maybe release-wise it's a little redundant and you'd
>>> have to release a bugfix version of Airflow if you find a critical problem
>>> in Client (which should be rare as agent is mostly automatically
>>> generated), but synchronizing versions is very tempting from the
>>> "supportability" point of view.
>>> >>
>>> >> Those are my 2 cents, I was not involved too much in the client part,
>>> so I do not feel like I have a strong voice in there, but if I would vote
>>> between the two - I'd choose synchronizing the releases.
>>> >>
>>> >> BTW. In providers we have a very different story - we keep dependency
>>> to airflow via PIP (which we can set it, if we find a provider stops being
>>> compatible with a version of Airflow). Also we automatically test in CI if
>>> the provider installs and imports on Airflow 2.0.0 to verify that it is
>>> backwards compatible (at least to the level of importability). This has
>>> already saved us from  "accidental" incompatibilities (wit yaml package
>>> import change recently). So there, separate versioning makes much more
>>> sense - also because the version of the provider relates exclusively to the
>>> feature that the provider version has, not Airflow itself). I think this is
>>> quite a different story from the client, which is very closely coupled with
>>> Airflow.
>>> >>
>>> >> J.
>>> >>
>>> >>
>>> >>
>>> >> On Wed, Mar 31, 2021 at 11:08 AM Deng Xiaodong <xd...@gmail.com>
>>> wrote:
>>> >>>
>>> >>> Yep, if we can, I would prefer to explicitly mention something like
>>> 2.0.X :)
>>> >>>
>>> >>> On Wed, Mar 31, 2021 at 10:54 AM Sumit Maheshwari <
>>> sumeet.manit@gmail.com> wrote:
>>> >>>>
>>> >>>> Yeah totally agrees with you XD about the range vs the fixed
>>> version of the main software. And with Airflow 2 dot & beyond, we would be
>>> following the semantic versioning more judiciously, so with that can we
>>> assume that users would be cognizant of the related Airflow version
>>> implicitly, or do you think we should mention something like 2.0.X in the
>>> changelog explicitly?
>>> >>>>
>>> >>>> On Wed, Mar 31, 2021 at 1:38 PM Deng Xiaodong <xd...@gmail.com>
>>> wrote:
>>> >>>>>
>>> >>>>> Thanks, Sumit, for bringing this up.
>>> >>>>>
>>> >>>>> - Regarding versioning for Python client: I favour the 1st way as
>>> well. The only minor comment I have is: instead of "mention the release
>>> version of the main software", possibly what users would expect is a "range
>>> of compatible versions of main software". If we follow semver for Airflow
>>> itself relatively strictly, this should not be hard.
>>> >>>>>
>>> >>>>> - Regarding the release of different client libraries: I agree
>>> with you. They should be versioned & released separately.
>>> >>>>>
>>> >>>>>
>>> >>>>> Regards,
>>> >>>>> XD
>>> >>>>>
>>> >>>>> On Wed, Mar 31, 2021 at 9:21 AM Sumit Maheshwari <
>>> sumeet.manit@gmail.com> wrote:
>>> >>>>>>
>>> >>>>>> Thanks QP for your feedback. I think the latest release of K8s
>>> client (v17.x.y) is moving in another direction.
>>> >>>>>>
>>> >>>>>> On Wed, Mar 31, 2021 at 12:13 AM QP Hou <qp...@scribd.com.invalid>
>>> wrote:
>>> >>>>>>>
>>> >>>>>>> Thanks Sumit for kicking off this discussion!
>>> >>>>>>>
>>> >>>>>>> I am in favor of approach #1 as well. Airflow software version
>>> can
>>> >>>>>>> change due to none API related changes, but the client should
>>> only
>>> >>>>>>> care about the public API contract. I believe this is also the
>>> same
>>> >>>>>>> approach that the k8s python client took as well? For example,
>>> the k8s
>>> >>>>>>> python client version 12.0.1 targets k8s api version 1.16.15.
>>> >>>>>>>
>>> >>>>>>> I am also with you that clients for different languages should
>>> have
>>> >>>>>>> their own release cycles. Other than testing burdens, every
>>> client
>>> >>>>>>> could introduce their own bug fixes and breaking changes without
>>> >>>>>>> having to trigger releases for all other clients. It would be a
>>> waste
>>> >>>>>>> of work for us to bump the python client version just for a bug
>>> fix in
>>> >>>>>>> the go client.
>>> >>>>>>>
>>> >>>>>>> Thanks,
>>> >>>>>>> QP
>>> >>>>>>>
>>> >>>>>>> On Tue, Mar 30, 2021 at 1:56 AM Sumit Maheshwari <
>>> msumit@apache.org> wrote:
>>> >>>>>>> >
>>> >>>>>>> > Hello Airflow dev community
>>> >>>>>>> >
>>> >>>>>>> > I'm writing this mail in regards to seek feedback on the first
>>> release of Apache Airflow's Python client. This is an icebreaker email only
>>> and we will follow the Apache process of releasing the version later on. As
>>> of now, I've raised a small PR to add the missing files in the repo, which
>>> are required to do the release of the client lib.
>>> >>>>>>> >
>>> >>>>>>> > The main question I've in mind about the versioning. I see
>>> that there are mainly two theories related to maintain the release versions
>>> of clients:
>>> >>>>>>> >
>>> >>>>>>> > The first is about following the semantic release of the
>>> clients on its own. The release version of the client doesn't include the
>>> version of the software itself. Something like we are doing with Airflow's
>>> provider packages. However, in each release, we mention the release version
>>> of the main software (Airflow in this case). This is the simple most way of
>>> maintaining releases and followed majorly.
>>> >>>>>>> > The second way is to include the software's version as well in
>>> its client's versioning, something like it's been done in the K8s Python
>>> client. In this approach, the client version provides more implicit
>>> knowledge about the compatible software, however it creates an extra burden
>>> of releasing the client with every minor release of the software, otherwise
>>> the versioning would fall apart. Also, there is a good chance that most of
>>> these minor or even a major release has no changes in the APIs, but the
>>> client would be needed to re-released nevertheless, just to keep the
>>> versioning in check.
>>> >>>>>>> >
>>> >>>>>>> > Personally, I'm in favour of the 1st way, as it's simple to
>>> manage and doesn't tightly couple client with its software.
>>> >>>>>>> >
>>> >>>>>>> > Another unrelated question I have is about the release of
>>> different client libraries. As of now, Airflow has Python and Go clients,
>>> so should we try to release them together or they should follow their own
>>> paths? In general, I'm against batching them together, cause the person who
>>> is releasing them may not be able to test all the different clients in the
>>> future.
>>> >>>>>>> >
>>> >>>>>>> > Let me know your thoughts on these 2 points.
>>> >>>>>>> >
>>> >>>>>>> > Thanks,
>>> >>>>>>> > Sumit Maheshwari
>>> >>>>>>> > PMC Apache Airflow
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> +48 660 796 129
>>>
>>

Re: Releasing Apache Airflow's Python Client

[Vikram Koka <vi...@astronomer.io.INVALID>]

Sumit,
Thank you for bringing up this discussion.

Reading through the thread, I am definitely far more comfortable with the
hybrid approach proposed by Kaxil, than the first approach, primarily
because of the same concerns articulated by Jarek regarding end user
experience. Prior to that, I was leaning towards the second approach.

Having said that, I think we need to be sure to follow certain guidelines
with Core Airflow to make sure this approach works consistently, with a
minimum of documentation needed and documentation to be read by Airflow
users. Since there are generated clients, I am specifically referring to
changes within Airflow regarding the API. Most importantly, I think we need
to have any API additions, whether a new endpoint or an enhancement to an
existing endpoint, only be available as part of a "feature release", rather
than a "patch release". And that, this should trigger an update to the
generated clients, triggering a new version of the same. This may be
obvious, because we agreed on following semantic versioning strictly, but I
thought it was worth reiterating.

Having said that, I think it is worth asking the question if that is
actually true, or if there are any APIs which are not  (or should not be)
supported in the clients?
I didn't think so, but wanted to make sure.

Best regards,
Vikram


On Sat, Apr 3, 2021 at 4:11 AM Jarek Potiuk <ja...@potiuk.com> wrote:

> I also like the hybrid approach proposed by Kaxil. It is not as simplenfor
> dependency management as 1-1 version, but IMHO it is simple-enough. As long
> as we strictly follow SemVer, a number of questions i listed above from the
> user, are answered:
>
>
> 1) Will my client be compatible with the version of Airflow ?
>
> Easy: 2.x.y will be compatible with 2.n.m of Airflow and so on.. Since we
> will not have breaking changes, the 'broad compatibility' level should be
> fine (with some feature -level differences but no general compatibility
> problems)
>
> 2) Will feature (API endpoint) x available in this version of client be
> working with the version of Airflow I have?
>
> We have to make sure that features are documented in the APIs at the level
> of '@since' if we keep it updated the form 'This feature is available in
> 'Airflow 2.3.*' and 'API 2.1.*' - we are covered. Since there will be no
> breaking changes in neither API nor Airflow - this will be enough.
>
>
> 3) Which version of Airflow should I install in order to use feature X
> that I have in this new version of client ?
>
> I think this might be possible to generate such compatibility matrix for
> all new API features added since Airflow 2.0 - it might be automatically
> generated from the API specification in this case. If we parse the API
> definition and have the right 'since' definition for both Airflow and API,
> it should be rather easy to generate such table:
>
> Feature       | Min API version | Min Airflow version
> MoonShot feature X  | 2.3 | 2.2
>
> Having such compatibility matric would, I think, be a great help for the
> users i think when they would plan their Airflow/API releases
>
>
> J.
>
> sob., 3 kwi 2021, 12:06 użytkownik .... <dz...@gmail.com> napisał:
>
>> +1 for Kaxil proposal
>>
>> I think, attempting to assign a specific client version to the library
>> application makes no sense in creating an API that is based on a
>> strict specification.
>>
>> One of the goals of this API was to make it easier to manage multiple
>> instances, and it's not often that you have an identically homogeneous
>> environment. It's natural to have different versions of Airflow in one
>> organization. If we want users to use a specific version of the
>> library on the client-side, it will probably not be possible in many
>> cases.
>>
>> pt., 2 kwi 2021 o 23:35 Kaxil Naik <ka...@gmail.com> napisał(a):
>> >
>> > Thanks, Sumit, for bringing this up.
>> >
>> > re: versioning for Python client:
>> >
>> > I favour a bit of a hybrid of 1 and 2. Initially, I favored just (1)
>> when we talked but after seeing how Elastic-Search does it I liked their
>> versioning strategy.
>> > How their python-client is compatible with ES itself is documented at
>> https://elasticsearch-py.readthedocs.io/en/v7.12.0/#compatibility.
>> >
>> > Since Airflow from 2.0 strictly follows SemVer we can continue with the
>> assumption that we (Airflow) won't break backwards-compatibility (to the
>> best of our ability).
>> > So we can have M.x.y as our versioning for our client where M is the
>> Major Version of the client that matches the Major version of Airflow; x is
>> the minor version for the client & y is the patch release. Here x & y are
>> independent of Airflow version. Our goal and message to user would be that
>> "You have to use a version of client library matching the major version of
>> Airflow"
>> >
>> > re: the release of different client libraries:
>> >
>> > Client for each language should be versioned & released separately.
>> >
>> > Regards,
>> > Kaxil
>> >
>> > On Wed, Mar 31, 2021 at 7:01 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>> >>
>> >> I think there are more questions we need to answer when it comes to
>> different versions. If we are going to different versioning for agent and
>> Airflow, we will have to accept the fact that over time people will have a
>> "matrix of versions" - for both Airflow and the client. And this introduces
>> "dependency mess" if we do not handle it well.
>> >>
>> >> I think in this we should give the user much clearer indication on
>> which version of Airflow the client should work on, and (more important)
>> which version it does NOT work on and which features will NOT be available.
>> Both cases are possible.
>> >>
>> >> There should be some way of knowing for the user, (in case there is a
>> backwards incompatibility or in case some features are not available) that
>> a given version of the client does not work with the version of Airflow
>> that the user has.
>> >>
>> >> I think users looking at the documentation should be able to be able
>> to figure out:
>> >>
>> >> 1) Will my client be compatible with the version of Airflow ?
>> >> 2) Will feature (API endpoint) x available in this version of client
>> be working with the version of Airflow I have?
>> >> 3) Which version of Airflow should I install in order to use feature X
>> that I have in this new version of client ?
>> >>
>> >> Similarly proper error messages should be displayed if there is
>> incompatibility or when a feature X is not available when you try to use a
>> version of client that supports it with Airflow version of Agent that does
>> not have it.
>> >>
>> >> So I'd say - separate versioning is fine as long as we handle the
>> above.
>> >>
>> >>
>> >> Saying all that - there is a beautiful "maintenance" simplicity if we
>> actually synchronize the version and we say "Always use SAME version of
>> client as Airflow" (and we fail the client if the versions is different).
>> This is way more simple in communicating to the users, and with most
>> deployments, I think upgrading airflow and clients at the same time should
>> be no problem.
>> >>
>> >> I think over time as the matrix of potential Airflow/Client
>> combinations will grow this might mean bigger number of issues that we have
>> to handle ("Why ma client does not do this and this (and of course it will
>> not have neither client nor airflow version specified and we will have to
>> ask for clarification).
>> >>
>> >> So as I see it - maybe release-wise it's a little redundant and you'd
>> have to release a bugfix version of Airflow if you find a critical problem
>> in Client (which should be rare as agent is mostly automatically
>> generated), but synchronizing versions is very tempting from the
>> "supportability" point of view.
>> >>
>> >> Those are my 2 cents, I was not involved too much in the client part,
>> so I do not feel like I have a strong voice in there, but if I would vote
>> between the two - I'd choose synchronizing the releases.
>> >>
>> >> BTW. In providers we have a very different story - we keep dependency
>> to airflow via PIP (which we can set it, if we find a provider stops being
>> compatible with a version of Airflow). Also we automatically test in CI if
>> the provider installs and imports on Airflow 2.0.0 to verify that it is
>> backwards compatible (at least to the level of importability). This has
>> already saved us from  "accidental" incompatibilities (wit yaml package
>> import change recently). So there, separate versioning makes much more
>> sense - also because the version of the provider relates exclusively to the
>> feature that the provider version has, not Airflow itself). I think this is
>> quite a different story from the client, which is very closely coupled with
>> Airflow.
>> >>
>> >> J.
>> >>
>> >>
>> >>
>> >> On Wed, Mar 31, 2021 at 11:08 AM Deng Xiaodong <xd...@gmail.com>
>> wrote:
>> >>>
>> >>> Yep, if we can, I would prefer to explicitly mention something like
>> 2.0.X :)
>> >>>
>> >>> On Wed, Mar 31, 2021 at 10:54 AM Sumit Maheshwari <
>> sumeet.manit@gmail.com> wrote:
>> >>>>
>> >>>> Yeah totally agrees with you XD about the range vs the fixed version
>> of the main software. And with Airflow 2 dot & beyond, we would be
>> following the semantic versioning more judiciously, so with that can we
>> assume that users would be cognizant of the related Airflow version
>> implicitly, or do you think we should mention something like 2.0.X in the
>> changelog explicitly?
>> >>>>
>> >>>> On Wed, Mar 31, 2021 at 1:38 PM Deng Xiaodong <xd...@gmail.com>
>> wrote:
>> >>>>>
>> >>>>> Thanks, Sumit, for bringing this up.
>> >>>>>
>> >>>>> - Regarding versioning for Python client: I favour the 1st way as
>> well. The only minor comment I have is: instead of "mention the release
>> version of the main software", possibly what users would expect is a "range
>> of compatible versions of main software". If we follow semver for Airflow
>> itself relatively strictly, this should not be hard.
>> >>>>>
>> >>>>> - Regarding the release of different client libraries: I agree with
>> you. They should be versioned & released separately.
>> >>>>>
>> >>>>>
>> >>>>> Regards,
>> >>>>> XD
>> >>>>>
>> >>>>> On Wed, Mar 31, 2021 at 9:21 AM Sumit Maheshwari <
>> sumeet.manit@gmail.com> wrote:
>> >>>>>>
>> >>>>>> Thanks QP for your feedback. I think the latest release of K8s
>> client (v17.x.y) is moving in another direction.
>> >>>>>>
>> >>>>>> On Wed, Mar 31, 2021 at 12:13 AM QP Hou <qp...@scribd.com.invalid>
>> wrote:
>> >>>>>>>
>> >>>>>>> Thanks Sumit for kicking off this discussion!
>> >>>>>>>
>> >>>>>>> I am in favor of approach #1 as well. Airflow software version can
>> >>>>>>> change due to none API related changes, but the client should only
>> >>>>>>> care about the public API contract. I believe this is also the
>> same
>> >>>>>>> approach that the k8s python client took as well? For example,
>> the k8s
>> >>>>>>> python client version 12.0.1 targets k8s api version 1.16.15.
>> >>>>>>>
>> >>>>>>> I am also with you that clients for different languages should
>> have
>> >>>>>>> their own release cycles. Other than testing burdens, every client
>> >>>>>>> could introduce their own bug fixes and breaking changes without
>> >>>>>>> having to trigger releases for all other clients. It would be a
>> waste
>> >>>>>>> of work for us to bump the python client version just for a bug
>> fix in
>> >>>>>>> the go client.
>> >>>>>>>
>> >>>>>>> Thanks,
>> >>>>>>> QP
>> >>>>>>>
>> >>>>>>> On Tue, Mar 30, 2021 at 1:56 AM Sumit Maheshwari <
>> msumit@apache.org> wrote:
>> >>>>>>> >
>> >>>>>>> > Hello Airflow dev community
>> >>>>>>> >
>> >>>>>>> > I'm writing this mail in regards to seek feedback on the first
>> release of Apache Airflow's Python client. This is an icebreaker email only
>> and we will follow the Apache process of releasing the version later on. As
>> of now, I've raised a small PR to add the missing files in the repo, which
>> are required to do the release of the client lib.
>> >>>>>>> >
>> >>>>>>> > The main question I've in mind about the versioning. I see that
>> there are mainly two theories related to maintain the release versions of
>> clients:
>> >>>>>>> >
>> >>>>>>> > The first is about following the semantic release of the
>> clients on its own. The release version of the client doesn't include the
>> version of the software itself. Something like we are doing with Airflow's
>> provider packages. However, in each release, we mention the release version
>> of the main software (Airflow in this case). This is the simple most way of
>> maintaining releases and followed majorly.
>> >>>>>>> > The second way is to include the software's version as well in
>> its client's versioning, something like it's been done in the K8s Python
>> client. In this approach, the client version provides more implicit
>> knowledge about the compatible software, however it creates an extra burden
>> of releasing the client with every minor release of the software, otherwise
>> the versioning would fall apart. Also, there is a good chance that most of
>> these minor or even a major release has no changes in the APIs, but the
>> client would be needed to re-released nevertheless, just to keep the
>> versioning in check.
>> >>>>>>> >
>> >>>>>>> > Personally, I'm in favour of the 1st way, as it's simple to
>> manage and doesn't tightly couple client with its software.
>> >>>>>>> >
>> >>>>>>> > Another unrelated question I have is about the release of
>> different client libraries. As of now, Airflow has Python and Go clients,
>> so should we try to release them together or they should follow their own
>> paths? In general, I'm against batching them together, cause the person who
>> is releasing them may not be able to test all the different clients in the
>> future.
>> >>>>>>> >
>> >>>>>>> > Let me know your thoughts on these 2 points.
>> >>>>>>> >
>> >>>>>>> > Thanks,
>> >>>>>>> > Sumit Maheshwari
>> >>>>>>> > PMC Apache Airflow
>> >>
>> >>
>> >>
>> >> --
>> >> +48 660 796 129
>>
>

Re: Releasing Apache Airflow's Python Client

[Jarek Potiuk <ja...@potiuk.com>]

I also like the hybrid approach proposed by Kaxil. It is not as simplenfor
dependency management as 1-1 version, but IMHO it is simple-enough. As long
as we strictly follow SemVer, a number of questions i listed above from the
user, are answered:


1) Will my client be compatible with the version of Airflow ?

Easy: 2.x.y will be compatible with 2.n.m of Airflow and so on.. Since we
will not have breaking changes, the 'broad compatibility' level should be
fine (with some feature -level differences but no general compatibility
problems)

2) Will feature (API endpoint) x available in this version of client be
working with the version of Airflow I have?

We have to make sure that features are documented in the APIs at the level
of '@since' if we keep it updated the form 'This feature is available in
'Airflow 2.3.*' and 'API 2.1.*' - we are covered. Since there will be no
breaking changes in neither API nor Airflow - this will be enough.


3) Which version of Airflow should I install in order to use feature X that
I have in this new version of client ?

I think this might be possible to generate such compatibility matrix for
all new API features added since Airflow 2.0 - it might be automatically
generated from the API specification in this case. If we parse the API
definition and have the right 'since' definition for both Airflow and API,
it should be rather easy to generate such table:

Feature       | Min API version | Min Airflow version
MoonShot feature X  | 2.3 | 2.2

Having such compatibility matric would, I think, be a great help for the
users i think when they would plan their Airflow/API releases


J.

sob., 3 kwi 2021, 12:06 użytkownik .... <dz...@gmail.com> napisał:

> +1 for Kaxil proposal
>
> I think, attempting to assign a specific client version to the library
> application makes no sense in creating an API that is based on a
> strict specification.
>
> One of the goals of this API was to make it easier to manage multiple
> instances, and it's not often that you have an identically homogeneous
> environment. It's natural to have different versions of Airflow in one
> organization. If we want users to use a specific version of the
> library on the client-side, it will probably not be possible in many
> cases.
>
> pt., 2 kwi 2021 o 23:35 Kaxil Naik <ka...@gmail.com> napisał(a):
> >
> > Thanks, Sumit, for bringing this up.
> >
> > re: versioning for Python client:
> >
> > I favour a bit of a hybrid of 1 and 2. Initially, I favored just (1)
> when we talked but after seeing how Elastic-Search does it I liked their
> versioning strategy.
> > How their python-client is compatible with ES itself is documented at
> https://elasticsearch-py.readthedocs.io/en/v7.12.0/#compatibility.
> >
> > Since Airflow from 2.0 strictly follows SemVer we can continue with the
> assumption that we (Airflow) won't break backwards-compatibility (to the
> best of our ability).
> > So we can have M.x.y as our versioning for our client where M is the
> Major Version of the client that matches the Major version of Airflow; x is
> the minor version for the client & y is the patch release. Here x & y are
> independent of Airflow version. Our goal and message to user would be that
> "You have to use a version of client library matching the major version of
> Airflow"
> >
> > re: the release of different client libraries:
> >
> > Client for each language should be versioned & released separately.
> >
> > Regards,
> > Kaxil
> >
> > On Wed, Mar 31, 2021 at 7:01 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >>
> >> I think there are more questions we need to answer when it comes to
> different versions. If we are going to different versioning for agent and
> Airflow, we will have to accept the fact that over time people will have a
> "matrix of versions" - for both Airflow and the client. And this introduces
> "dependency mess" if we do not handle it well.
> >>
> >> I think in this we should give the user much clearer indication on
> which version of Airflow the client should work on, and (more important)
> which version it does NOT work on and which features will NOT be available.
> Both cases are possible.
> >>
> >> There should be some way of knowing for the user, (in case there is a
> backwards incompatibility or in case some features are not available) that
> a given version of the client does not work with the version of Airflow
> that the user has.
> >>
> >> I think users looking at the documentation should be able to be able to
> figure out:
> >>
> >> 1) Will my client be compatible with the version of Airflow ?
> >> 2) Will feature (API endpoint) x available in this version of client be
> working with the version of Airflow I have?
> >> 3) Which version of Airflow should I install in order to use feature X
> that I have in this new version of client ?
> >>
> >> Similarly proper error messages should be displayed if there is
> incompatibility or when a feature X is not available when you try to use a
> version of client that supports it with Airflow version of Agent that does
> not have it.
> >>
> >> So I'd say - separate versioning is fine as long as we handle the above.
> >>
> >>
> >> Saying all that - there is a beautiful "maintenance" simplicity if we
> actually synchronize the version and we say "Always use SAME version of
> client as Airflow" (and we fail the client if the versions is different).
> This is way more simple in communicating to the users, and with most
> deployments, I think upgrading airflow and clients at the same time should
> be no problem.
> >>
> >> I think over time as the matrix of potential Airflow/Client
> combinations will grow this might mean bigger number of issues that we have
> to handle ("Why ma client does not do this and this (and of course it will
> not have neither client nor airflow version specified and we will have to
> ask for clarification).
> >>
> >> So as I see it - maybe release-wise it's a little redundant and you'd
> have to release a bugfix version of Airflow if you find a critical problem
> in Client (which should be rare as agent is mostly automatically
> generated), but synchronizing versions is very tempting from the
> "supportability" point of view.
> >>
> >> Those are my 2 cents, I was not involved too much in the client part,
> so I do not feel like I have a strong voice in there, but if I would vote
> between the two - I'd choose synchronizing the releases.
> >>
> >> BTW. In providers we have a very different story - we keep dependency
> to airflow via PIP (which we can set it, if we find a provider stops being
> compatible with a version of Airflow). Also we automatically test in CI if
> the provider installs and imports on Airflow 2.0.0 to verify that it is
> backwards compatible (at least to the level of importability). This has
> already saved us from  "accidental" incompatibilities (wit yaml package
> import change recently). So there, separate versioning makes much more
> sense - also because the version of the provider relates exclusively to the
> feature that the provider version has, not Airflow itself). I think this is
> quite a different story from the client, which is very closely coupled with
> Airflow.
> >>
> >> J.
> >>
> >>
> >>
> >> On Wed, Mar 31, 2021 at 11:08 AM Deng Xiaodong <xd...@gmail.com>
> wrote:
> >>>
> >>> Yep, if we can, I would prefer to explicitly mention something like
> 2.0.X :)
> >>>
> >>> On Wed, Mar 31, 2021 at 10:54 AM Sumit Maheshwari <
> sumeet.manit@gmail.com> wrote:
> >>>>
> >>>> Yeah totally agrees with you XD about the range vs the fixed version
> of the main software. And with Airflow 2 dot & beyond, we would be
> following the semantic versioning more judiciously, so with that can we
> assume that users would be cognizant of the related Airflow version
> implicitly, or do you think we should mention something like 2.0.X in the
> changelog explicitly?
> >>>>
> >>>> On Wed, Mar 31, 2021 at 1:38 PM Deng Xiaodong <xd...@gmail.com>
> wrote:
> >>>>>
> >>>>> Thanks, Sumit, for bringing this up.
> >>>>>
> >>>>> - Regarding versioning for Python client: I favour the 1st way as
> well. The only minor comment I have is: instead of "mention the release
> version of the main software", possibly what users would expect is a "range
> of compatible versions of main software". If we follow semver for Airflow
> itself relatively strictly, this should not be hard.
> >>>>>
> >>>>> - Regarding the release of different client libraries: I agree with
> you. They should be versioned & released separately.
> >>>>>
> >>>>>
> >>>>> Regards,
> >>>>> XD
> >>>>>
> >>>>> On Wed, Mar 31, 2021 at 9:21 AM Sumit Maheshwari <
> sumeet.manit@gmail.com> wrote:
> >>>>>>
> >>>>>> Thanks QP for your feedback. I think the latest release of K8s
> client (v17.x.y) is moving in another direction.
> >>>>>>
> >>>>>> On Wed, Mar 31, 2021 at 12:13 AM QP Hou <qp...@scribd.com.invalid>
> wrote:
> >>>>>>>
> >>>>>>> Thanks Sumit for kicking off this discussion!
> >>>>>>>
> >>>>>>> I am in favor of approach #1 as well. Airflow software version can
> >>>>>>> change due to none API related changes, but the client should only
> >>>>>>> care about the public API contract. I believe this is also the same
> >>>>>>> approach that the k8s python client took as well? For example, the
> k8s
> >>>>>>> python client version 12.0.1 targets k8s api version 1.16.15.
> >>>>>>>
> >>>>>>> I am also with you that clients for different languages should have
> >>>>>>> their own release cycles. Other than testing burdens, every client
> >>>>>>> could introduce their own bug fixes and breaking changes without
> >>>>>>> having to trigger releases for all other clients. It would be a
> waste
> >>>>>>> of work for us to bump the python client version just for a bug
> fix in
> >>>>>>> the go client.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> QP
> >>>>>>>
> >>>>>>> On Tue, Mar 30, 2021 at 1:56 AM Sumit Maheshwari <
> msumit@apache.org> wrote:
> >>>>>>> >
> >>>>>>> > Hello Airflow dev community
> >>>>>>> >
> >>>>>>> > I'm writing this mail in regards to seek feedback on the first
> release of Apache Airflow's Python client. This is an icebreaker email only
> and we will follow the Apache process of releasing the version later on. As
> of now, I've raised a small PR to add the missing files in the repo, which
> are required to do the release of the client lib.
> >>>>>>> >
> >>>>>>> > The main question I've in mind about the versioning. I see that
> there are mainly two theories related to maintain the release versions of
> clients:
> >>>>>>> >
> >>>>>>> > The first is about following the semantic release of the clients
> on its own. The release version of the client doesn't include the version
> of the software itself. Something like we are doing with Airflow's provider
> packages. However, in each release, we mention the release version of the
> main software (Airflow in this case). This is the simple most way of
> maintaining releases and followed majorly.
> >>>>>>> > The second way is to include the software's version as well in
> its client's versioning, something like it's been done in the K8s Python
> client. In this approach, the client version provides more implicit
> knowledge about the compatible software, however it creates an extra burden
> of releasing the client with every minor release of the software, otherwise
> the versioning would fall apart. Also, there is a good chance that most of
> these minor or even a major release has no changes in the APIs, but the
> client would be needed to re-released nevertheless, just to keep the
> versioning in check.
> >>>>>>> >
> >>>>>>> > Personally, I'm in favour of the 1st way, as it's simple to
> manage and doesn't tightly couple client with its software.
> >>>>>>> >
> >>>>>>> > Another unrelated question I have is about the release of
> different client libraries. As of now, Airflow has Python and Go clients,
> so should we try to release them together or they should follow their own
> paths? In general, I'm against batching them together, cause the person who
> is releasing them may not be able to test all the different clients in the
> future.
> >>>>>>> >
> >>>>>>> > Let me know your thoughts on these 2 points.
> >>>>>>> >
> >>>>>>> > Thanks,
> >>>>>>> > Sumit Maheshwari
> >>>>>>> > PMC Apache Airflow
> >>
> >>
> >>
> >> --
> >> +48 660 796 129
>

Re: Releasing Apache Airflow's Python Client

["...." <dz...@gmail.com>]

+1 for Kaxil proposal

I think, attempting to assign a specific client version to the library
application makes no sense in creating an API that is based on a
strict specification.

One of the goals of this API was to make it easier to manage multiple
instances, and it's not often that you have an identically homogeneous
environment. It's natural to have different versions of Airflow in one
organization. If we want users to use a specific version of the
library on the client-side, it will probably not be possible in many
cases.

pt., 2 kwi 2021 o 23:35 Kaxil Naik <ka...@gmail.com> napisał(a):
>
> Thanks, Sumit, for bringing this up.
>
> re: versioning for Python client:
>
> I favour a bit of a hybrid of 1 and 2. Initially, I favored just (1) when we talked but after seeing how Elastic-Search does it I liked their versioning strategy.
> How their python-client is compatible with ES itself is documented at https://elasticsearch-py.readthedocs.io/en/v7.12.0/#compatibility.
>
> Since Airflow from 2.0 strictly follows SemVer we can continue with the assumption that we (Airflow) won't break backwards-compatibility (to the best of our ability).
> So we can have M.x.y as our versioning for our client where M is the Major Version of the client that matches the Major version of Airflow; x is the minor version for the client & y is the patch release. Here x & y are independent of Airflow version. Our goal and message to user would be that "You have to use a version of client library matching the major version of Airflow"
>
> re: the release of different client libraries:
>
> Client for each language should be versioned & released separately.
>
> Regards,
> Kaxil
>
> On Wed, Mar 31, 2021 at 7:01 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>> I think there are more questions we need to answer when it comes to different versions. If we are going to different versioning for agent and Airflow, we will have to accept the fact that over time people will have a "matrix of versions" - for both Airflow and the client. And this introduces "dependency mess" if we do not handle it well.
>>
>> I think in this we should give the user much clearer indication on which version of Airflow the client should work on, and (more important) which version it does NOT work on and which features will NOT be available. Both cases are possible.
>>
>> There should be some way of knowing for the user, (in case there is a backwards incompatibility or in case some features are not available) that a given version of the client does not work with the version of Airflow that the user has.
>>
>> I think users looking at the documentation should be able to be able to figure out:
>>
>> 1) Will my client be compatible with the version of Airflow ?
>> 2) Will feature (API endpoint) x available in this version of client be working with the version of Airflow I have?
>> 3) Which version of Airflow should I install in order to use feature X that I have in this new version of client ?
>>
>> Similarly proper error messages should be displayed if there is incompatibility or when a feature X is not available when you try to use a version of client that supports it with Airflow version of Agent that does not have it.
>>
>> So I'd say - separate versioning is fine as long as we handle the above.
>>
>>
>> Saying all that - there is a beautiful "maintenance" simplicity if we actually synchronize the version and we say "Always use SAME version of client as Airflow" (and we fail the client if the versions is different).  This is way more simple in communicating to the users, and with most deployments, I think upgrading airflow and clients at the same time should be no problem.
>>
>> I think over time as the matrix of potential Airflow/Client combinations will grow this might mean bigger number of issues that we have to handle ("Why ma client does not do this and this (and of course it will not have neither client nor airflow version specified and we will have to ask for clarification).
>>
>> So as I see it - maybe release-wise it's a little redundant and you'd have to release a bugfix version of Airflow if you find a critical problem in Client (which should be rare as agent is mostly automatically generated), but synchronizing versions is very tempting from the "supportability" point of view.
>>
>> Those are my 2 cents, I was not involved too much in the client part, so I do not feel like I have a strong voice in there, but if I would vote between the two - I'd choose synchronizing the releases.
>>
>> BTW. In providers we have a very different story - we keep dependency to airflow via PIP (which we can set it, if we find a provider stops being compatible with a version of Airflow). Also we automatically test in CI if the provider installs and imports on Airflow 2.0.0 to verify that it is backwards compatible (at least to the level of importability). This has already saved us from  "accidental" incompatibilities (wit yaml package import change recently). So there, separate versioning makes much more sense - also because the version of the provider relates exclusively to the feature that the provider version has, not Airflow itself). I think this is quite a different story from the client, which is very closely coupled with Airflow.
>>
>> J.
>>
>>
>>
>> On Wed, Mar 31, 2021 at 11:08 AM Deng Xiaodong <xd...@gmail.com> wrote:
>>>
>>> Yep, if we can, I would prefer to explicitly mention something like 2.0.X :)
>>>
>>> On Wed, Mar 31, 2021 at 10:54 AM Sumit Maheshwari <su...@gmail.com> wrote:
>>>>
>>>> Yeah totally agrees with you XD about the range vs the fixed version of the main software. And with Airflow 2 dot & beyond, we would be following the semantic versioning more judiciously, so with that can we assume that users would be cognizant of the related Airflow version implicitly, or do you think we should mention something like 2.0.X in the changelog explicitly?
>>>>
>>>> On Wed, Mar 31, 2021 at 1:38 PM Deng Xiaodong <xd...@gmail.com> wrote:
>>>>>
>>>>> Thanks, Sumit, for bringing this up.
>>>>>
>>>>> - Regarding versioning for Python client: I favour the 1st way as well. The only minor comment I have is: instead of "mention the release version of the main software", possibly what users would expect is a "range of compatible versions of main software". If we follow semver for Airflow itself relatively strictly, this should not be hard.
>>>>>
>>>>> - Regarding the release of different client libraries: I agree with you. They should be versioned & released separately.
>>>>>
>>>>>
>>>>> Regards,
>>>>> XD
>>>>>
>>>>> On Wed, Mar 31, 2021 at 9:21 AM Sumit Maheshwari <su...@gmail.com> wrote:
>>>>>>
>>>>>> Thanks QP for your feedback. I think the latest release of K8s client (v17.x.y) is moving in another direction.
>>>>>>
>>>>>> On Wed, Mar 31, 2021 at 12:13 AM QP Hou <qp...@scribd.com.invalid> wrote:
>>>>>>>
>>>>>>> Thanks Sumit for kicking off this discussion!
>>>>>>>
>>>>>>> I am in favor of approach #1 as well. Airflow software version can
>>>>>>> change due to none API related changes, but the client should only
>>>>>>> care about the public API contract. I believe this is also the same
>>>>>>> approach that the k8s python client took as well? For example, the k8s
>>>>>>> python client version 12.0.1 targets k8s api version 1.16.15.
>>>>>>>
>>>>>>> I am also with you that clients for different languages should have
>>>>>>> their own release cycles. Other than testing burdens, every client
>>>>>>> could introduce their own bug fixes and breaking changes without
>>>>>>> having to trigger releases for all other clients. It would be a waste
>>>>>>> of work for us to bump the python client version just for a bug fix in
>>>>>>> the go client.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> QP
>>>>>>>
>>>>>>> On Tue, Mar 30, 2021 at 1:56 AM Sumit Maheshwari <ms...@apache.org> wrote:
>>>>>>> >
>>>>>>> > Hello Airflow dev community
>>>>>>> >
>>>>>>> > I'm writing this mail in regards to seek feedback on the first release of Apache Airflow's Python client. This is an icebreaker email only and we will follow the Apache process of releasing the version later on. As of now, I've raised a small PR to add the missing files in the repo, which are required to do the release of the client lib.
>>>>>>> >
>>>>>>> > The main question I've in mind about the versioning. I see that there are mainly two theories related to maintain the release versions of clients:
>>>>>>> >
>>>>>>> > The first is about following the semantic release of the clients on its own. The release version of the client doesn't include the version of the software itself. Something like we are doing with Airflow's provider packages. However, in each release, we mention the release version of the main software (Airflow in this case). This is the simple most way of maintaining releases and followed majorly.
>>>>>>> > The second way is to include the software's version as well in its client's versioning, something like it's been done in the K8s Python client. In this approach, the client version provides more implicit knowledge about the compatible software, however it creates an extra burden of releasing the client with every minor release of the software, otherwise the versioning would fall apart. Also, there is a good chance that most of these minor or even a major release has no changes in the APIs, but the client would be needed to re-released nevertheless, just to keep the versioning in check.
>>>>>>> >
>>>>>>> > Personally, I'm in favour of the 1st way, as it's simple to manage and doesn't tightly couple client with its software.
>>>>>>> >
>>>>>>> > Another unrelated question I have is about the release of different client libraries. As of now, Airflow has Python and Go clients, so should we try to release them together or they should follow their own paths? In general, I'm against batching them together, cause the person who is releasing them may not be able to test all the different clients in the future.
>>>>>>> >
>>>>>>> > Let me know your thoughts on these 2 points.
>>>>>>> >
>>>>>>> > Thanks,
>>>>>>> > Sumit Maheshwari
>>>>>>> > PMC Apache Airflow
>>
>>
>>
>> --
>> +48 660 796 129

Re: Releasing Apache Airflow's Python Client

[Kaxil Naik <ka...@gmail.com>]

Thanks, Sumit, for bringing this up.

re: *versioning for Python client*:

I favour a bit of a hybrid of 1 and 2. Initially, I favored just (1) when
we talked but after seeing how Elastic-Search does it I liked
their versioning strategy.
How their python-client is compatible with ES itself is documented at
https://elasticsearch-py.readthedocs.io/en/v7.12.0/#compatibility.

Since Airflow from 2.0 strictly follows SemVer we can continue with the
assumption that we (Airflow) won't break backwards-compatibility (to the
best of our ability).
So we can have *M.x.y *as our versioning for our client where *M *is the
Major Version of the client *that matches the Major version of Airflow*; *x* is
the minor version for the client & *y* is the patch release. Here *x* & *y*
are independent of Airflow version. Our goal and message to user would be
that "You have to use a version of client library matching the major
version of Airflow"

re: *the release of different client libraries*:

Client for each language should be versioned & released separately.

Regards,
Kaxil

On Wed, Mar 31, 2021 at 7:01 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> I think there are more questions we need to answer when it comes to
> different versions. If we are going to different versioning for agent and
> Airflow, we will have to accept the fact that over time people will have a
> "matrix of versions" - for both Airflow and the client. And this introduces
> "dependency mess" if we do not handle it well.
>
> I think in this we should give the user much clearer indication on which
> version of Airflow the client should work on, and (more important) which
> version it does NOT work on and which features will NOT be available. Both
> cases are possible.
>
> There should be some way of knowing for the user, (in case there is a
> backwards incompatibility or in case some features are not available) that
> a given version of the client does not work with the version of Airflow
> that the user has.
>
> I think users looking at the documentation should be able to be able to
> figure out:
>
> 1) Will my client be compatible with the version of Airflow ?
> 2) Will feature (API endpoint) x available in this version of client be
> working with the version of Airflow I have?
> 3) Which version of Airflow should I install in order to use feature X
> that I have in this new version of client ?
>
> Similarly proper error messages should be displayed if there is
> incompatibility or when a feature X is not available when you try to use a
> version of client that supports it with Airflow version of Agent that does
> not have it.
>
> So I'd say - separate versioning is fine as long as we handle the above.
>
>
> Saying all that - there is a beautiful "maintenance" simplicity if we
> actually synchronize the version and we say "Always use SAME version of
> client as Airflow" (and we fail the client if the versions is different).
> This is way more simple in communicating to the users, and with most
> deployments, I think upgrading airflow and clients at the same time should
> be no problem.
>
> I think over time as the matrix of potential Airflow/Client combinations
> will grow this might mean bigger number of issues that we have to handle
> ("Why ma client does not do this and this (and of course it will not have
> neither client nor airflow version specified and we will have to ask for
> clarification).
>
> So as I see it - maybe release-wise it's a little redundant and you'd have
> to release a bugfix version of Airflow if you find a critical problem in
> Client (which should be rare as agent is mostly automatically generated),
> but synchronizing versions is very tempting from the "supportability" point
> of view.
>
> Those are my 2 cents, I was not involved too much in the client part, so I
> do not feel like I have a strong voice in there, but if I would vote
> between the two - I'd choose synchronizing the releases.
>
> BTW. In providers we have a very different story - we keep dependency to
> airflow via PIP (which we can set it, if we find a provider stops being
> compatible with a version of Airflow). Also we automatically test in CI if
> the provider installs and imports on Airflow 2.0.0 to verify that it is
> backwards compatible (at least to the level of importability). This has
> already saved us from  "accidental" incompatibilities (wit yaml package
> import change recently). So there, separate versioning makes much more
> sense - also because the version of the provider relates exclusively to the
> feature that the provider version has, not Airflow itself). I think this is
> quite a different story from the client, which is very closely coupled with
> Airflow.
>
> J.
>
>
>
> On Wed, Mar 31, 2021 at 11:08 AM Deng Xiaodong <xd...@gmail.com>
> wrote:
>
>> Yep, if we can, I would prefer to explicitly mention something like 2.0.X
>> :)
>>
>> On Wed, Mar 31, 2021 at 10:54 AM Sumit Maheshwari <su...@gmail.com>
>> wrote:
>>
>>> Yeah totally agrees with you XD about the range vs the fixed version of
>>> the main software. And with Airflow 2 dot & beyond, we would be following
>>> the semantic versioning more judiciously, so with that can we assume that
>>> users would be cognizant of the related Airflow version implicitly, or do
>>> you think we should mention something like *2.0.X* in the changelog
>>> explicitly?
>>>
>>> On Wed, Mar 31, 2021 at 1:38 PM Deng Xiaodong <xd...@gmail.com>
>>> wrote:
>>>
>>>> Thanks, Sumit, for bringing this up.
>>>>
>>>> - Regarding *versioning for Python client*: I favour the 1st way as
>>>> well. The only minor comment I have is: instead of "*mention the
>>>> release version of the main software*", possibly what users would
>>>> expect is a "*range of compatible versions of main software*". If we
>>>> follow semver for Airflow itself relatively strictly, this should not be
>>>> hard.
>>>>
>>>> - Regarding *the release of different client libraries*: I agree with
>>>> you. They should be versioned & released separately.
>>>>
>>>>
>>>> Regards,
>>>> XD
>>>>
>>>> On Wed, Mar 31, 2021 at 9:21 AM Sumit Maheshwari <
>>>> sumeet.manit@gmail.com> wrote:
>>>>
>>>>> Thanks QP for your feedback. I think the latest release of K8s client
>>>>> (v17.x.y) is moving in another direction.
>>>>>
>>>>> On Wed, Mar 31, 2021 at 12:13 AM QP Hou <qp...@scribd.com.invalid>
>>>>> wrote:
>>>>>
>>>>>> Thanks Sumit for kicking off this discussion!
>>>>>>
>>>>>> I am in favor of approach #1 as well. Airflow software version can
>>>>>> change due to none API related changes, but the client should only
>>>>>> care about the public API contract. I believe this is also the same
>>>>>> approach that the k8s python client took as well? For example, the k8s
>>>>>> python client version 12.0.1 targets k8s api version 1.16.15.
>>>>>>
>>>>>> I am also with you that clients for different languages should have
>>>>>> their own release cycles. Other than testing burdens, every client
>>>>>> could introduce their own bug fixes and breaking changes without
>>>>>> having to trigger releases for all other clients. It would be a waste
>>>>>> of work for us to bump the python client version just for a bug fix in
>>>>>> the go client.
>>>>>>
>>>>>> Thanks,
>>>>>> QP
>>>>>>
>>>>>> On Tue, Mar 30, 2021 at 1:56 AM Sumit Maheshwari <ms...@apache.org>
>>>>>> wrote:
>>>>>> >
>>>>>> > Hello Airflow dev community
>>>>>> >
>>>>>> > I'm writing this mail in regards to seek feedback on the first
>>>>>> release of Apache Airflow's Python client. This is an icebreaker email only
>>>>>> and we will follow the Apache process of releasing the version later on. As
>>>>>> of now, I've raised a small PR to add the missing files in the repo, which
>>>>>> are required to do the release of the client lib.
>>>>>> >
>>>>>> > The main question I've in mind about the versioning. I see that
>>>>>> there are mainly two theories related to maintain the release versions of
>>>>>> clients:
>>>>>> >
>>>>>> > The first is about following the semantic release of the clients on
>>>>>> its own. The release version of the client doesn't include the version of
>>>>>> the software itself. Something like we are doing with Airflow's provider
>>>>>> packages. However, in each release, we mention the release version of the
>>>>>> main software (Airflow in this case). This is the simple most way of
>>>>>> maintaining releases and followed majorly.
>>>>>> > The second way is to include the software's version as well in its
>>>>>> client's versioning, something like it's been done in the K8s Python
>>>>>> client. In this approach, the client version provides more implicit
>>>>>> knowledge about the compatible software, however it creates an extra burden
>>>>>> of releasing the client with every minor release of the software, otherwise
>>>>>> the versioning would fall apart. Also, there is a good chance that most of
>>>>>> these minor or even a major release has no changes in the APIs, but the
>>>>>> client would be needed to re-released nevertheless, just to keep the
>>>>>> versioning in check.
>>>>>> >
>>>>>> > Personally, I'm in favour of the 1st way, as it's simple to manage
>>>>>> and doesn't tightly couple client with its software.
>>>>>> >
>>>>>> > Another unrelated question I have is about the release of different
>>>>>> client libraries. As of now, Airflow has Python and Go clients, so should
>>>>>> we try to release them together or they should follow their own paths? In
>>>>>> general, I'm against batching them together, cause the person who is
>>>>>> releasing them may not be able to test all the different clients in the
>>>>>> future.
>>>>>> >
>>>>>> > Let me know your thoughts on these 2 points.
>>>>>> >
>>>>>> > Thanks,
>>>>>> > Sumit Maheshwari
>>>>>> > PMC Apache Airflow
>>>>>>
>>>>>
>
> --
> +48 660 796 129
>

Re: Releasing Apache Airflow's Python Client

[Jarek Potiuk <ja...@potiuk.com>]

I think there are more questions we need to answer when it comes to
different versions. If we are going to different versioning for agent and
Airflow, we will have to accept the fact that over time people will have a
"matrix of versions" - for both Airflow and the client. And this introduces
"dependency mess" if we do not handle it well.

I think in this we should give the user much clearer indication on which
version of Airflow the client should work on, and (more important) which
version it does NOT work on and which features will NOT be available. Both
cases are possible.

There should be some way of knowing for the user, (in case there is a
backwards incompatibility or in case some features are not available) that
a given version of the client does not work with the version of Airflow
that the user has.

I think users looking at the documentation should be able to be able to
figure out:

1) Will my client be compatible with the version of Airflow ?
2) Will feature (API endpoint) x available in this version of client be
working with the version of Airflow I have?
3) Which version of Airflow should I install in order to use feature X that
I have in this new version of client ?

Similarly proper error messages should be displayed if there is
incompatibility or when a feature X is not available when you try to use a
version of client that supports it with Airflow version of Agent that does
not have it.

So I'd say - separate versioning is fine as long as we handle the above.


Saying all that - there is a beautiful "maintenance" simplicity if we
actually synchronize the version and we say "Always use SAME version of
client as Airflow" (and we fail the client if the versions is different).
This is way more simple in communicating to the users, and with most
deployments, I think upgrading airflow and clients at the same time should
be no problem.

I think over time as the matrix of potential Airflow/Client combinations
will grow this might mean bigger number of issues that we have to handle
("Why ma client does not do this and this (and of course it will not have
neither client nor airflow version specified and we will have to ask for
clarification).

So as I see it - maybe release-wise it's a little redundant and you'd have
to release a bugfix version of Airflow if you find a critical problem in
Client (which should be rare as agent is mostly automatically generated),
but synchronizing versions is very tempting from the "supportability" point
of view.

Those are my 2 cents, I was not involved too much in the client part, so I
do not feel like I have a strong voice in there, but if I would vote
between the two - I'd choose synchronizing the releases.

BTW. In providers we have a very different story - we keep dependency to
airflow via PIP (which we can set it, if we find a provider stops being
compatible with a version of Airflow). Also we automatically test in CI if
the provider installs and imports on Airflow 2.0.0 to verify that it is
backwards compatible (at least to the level of importability). This has
already saved us from  "accidental" incompatibilities (wit yaml package
import change recently). So there, separate versioning makes much more
sense - also because the version of the provider relates exclusively to the
feature that the provider version has, not Airflow itself). I think this is
quite a different story from the client, which is very closely coupled with
Airflow.

J.



On Wed, Mar 31, 2021 at 11:08 AM Deng Xiaodong <xd...@gmail.com> wrote:

> Yep, if we can, I would prefer to explicitly mention something like 2.0.X
> :)
>
> On Wed, Mar 31, 2021 at 10:54 AM Sumit Maheshwari <su...@gmail.com>
> wrote:
>
>> Yeah totally agrees with you XD about the range vs the fixed version of
>> the main software. And with Airflow 2 dot & beyond, we would be following
>> the semantic versioning more judiciously, so with that can we assume that
>> users would be cognizant of the related Airflow version implicitly, or do
>> you think we should mention something like *2.0.X* in the changelog
>> explicitly?
>>
>> On Wed, Mar 31, 2021 at 1:38 PM Deng Xiaodong <xd...@gmail.com>
>> wrote:
>>
>>> Thanks, Sumit, for bringing this up.
>>>
>>> - Regarding *versioning for Python client*: I favour the 1st way as
>>> well. The only minor comment I have is: instead of "*mention the
>>> release version of the main software*", possibly what users would
>>> expect is a "*range of compatible versions of main software*". If we
>>> follow semver for Airflow itself relatively strictly, this should not be
>>> hard.
>>>
>>> - Regarding *the release of different client libraries*: I agree with
>>> you. They should be versioned & released separately.
>>>
>>>
>>> Regards,
>>> XD
>>>
>>> On Wed, Mar 31, 2021 at 9:21 AM Sumit Maheshwari <su...@gmail.com>
>>> wrote:
>>>
>>>> Thanks QP for your feedback. I think the latest release of K8s client
>>>> (v17.x.y) is moving in another direction.
>>>>
>>>> On Wed, Mar 31, 2021 at 12:13 AM QP Hou <qp...@scribd.com.invalid> wrote:
>>>>
>>>>> Thanks Sumit for kicking off this discussion!
>>>>>
>>>>> I am in favor of approach #1 as well. Airflow software version can
>>>>> change due to none API related changes, but the client should only
>>>>> care about the public API contract. I believe this is also the same
>>>>> approach that the k8s python client took as well? For example, the k8s
>>>>> python client version 12.0.1 targets k8s api version 1.16.15.
>>>>>
>>>>> I am also with you that clients for different languages should have
>>>>> their own release cycles. Other than testing burdens, every client
>>>>> could introduce their own bug fixes and breaking changes without
>>>>> having to trigger releases for all other clients. It would be a waste
>>>>> of work for us to bump the python client version just for a bug fix in
>>>>> the go client.
>>>>>
>>>>> Thanks,
>>>>> QP
>>>>>
>>>>> On Tue, Mar 30, 2021 at 1:56 AM Sumit Maheshwari <ms...@apache.org>
>>>>> wrote:
>>>>> >
>>>>> > Hello Airflow dev community
>>>>> >
>>>>> > I'm writing this mail in regards to seek feedback on the first
>>>>> release of Apache Airflow's Python client. This is an icebreaker email only
>>>>> and we will follow the Apache process of releasing the version later on. As
>>>>> of now, I've raised a small PR to add the missing files in the repo, which
>>>>> are required to do the release of the client lib.
>>>>> >
>>>>> > The main question I've in mind about the versioning. I see that
>>>>> there are mainly two theories related to maintain the release versions of
>>>>> clients:
>>>>> >
>>>>> > The first is about following the semantic release of the clients on
>>>>> its own. The release version of the client doesn't include the version of
>>>>> the software itself. Something like we are doing with Airflow's provider
>>>>> packages. However, in each release, we mention the release version of the
>>>>> main software (Airflow in this case). This is the simple most way of
>>>>> maintaining releases and followed majorly.
>>>>> > The second way is to include the software's version as well in its
>>>>> client's versioning, something like it's been done in the K8s Python
>>>>> client. In this approach, the client version provides more implicit
>>>>> knowledge about the compatible software, however it creates an extra burden
>>>>> of releasing the client with every minor release of the software, otherwise
>>>>> the versioning would fall apart. Also, there is a good chance that most of
>>>>> these minor or even a major release has no changes in the APIs, but the
>>>>> client would be needed to re-released nevertheless, just to keep the
>>>>> versioning in check.
>>>>> >
>>>>> > Personally, I'm in favour of the 1st way, as it's simple to manage
>>>>> and doesn't tightly couple client with its software.
>>>>> >
>>>>> > Another unrelated question I have is about the release of different
>>>>> client libraries. As of now, Airflow has Python and Go clients, so should
>>>>> we try to release them together or they should follow their own paths? In
>>>>> general, I'm against batching them together, cause the person who is
>>>>> releasing them may not be able to test all the different clients in the
>>>>> future.
>>>>> >
>>>>> > Let me know your thoughts on these 2 points.
>>>>> >
>>>>> > Thanks,
>>>>> > Sumit Maheshwari
>>>>> > PMC Apache Airflow
>>>>>
>>>>

-- 
+48 660 796 129

Re: Releasing Apache Airflow's Python Client

[Deng Xiaodong <xd...@gmail.com>]

Yep, if we can, I would prefer to explicitly mention something like 2.0.X :)

On Wed, Mar 31, 2021 at 10:54 AM Sumit Maheshwari <su...@gmail.com>
wrote:

> Yeah totally agrees with you XD about the range vs the fixed version of
> the main software. And with Airflow 2 dot & beyond, we would be following
> the semantic versioning more judiciously, so with that can we assume that
> users would be cognizant of the related Airflow version implicitly, or do
> you think we should mention something like *2.0.X* in the changelog
> explicitly?
>
> On Wed, Mar 31, 2021 at 1:38 PM Deng Xiaodong <xd...@gmail.com> wrote:
>
>> Thanks, Sumit, for bringing this up.
>>
>> - Regarding *versioning for Python client*: I favour the 1st way as
>> well. The only minor comment I have is: instead of "*mention the release
>> version of the main software*", possibly what users would expect is a "*range
>> of compatible versions of main software*". If we follow semver for
>> Airflow itself relatively strictly, this should not be hard.
>>
>> - Regarding *the release of different client libraries*: I agree with
>> you. They should be versioned & released separately.
>>
>>
>> Regards,
>> XD
>>
>> On Wed, Mar 31, 2021 at 9:21 AM Sumit Maheshwari <su...@gmail.com>
>> wrote:
>>
>>> Thanks QP for your feedback. I think the latest release of K8s client
>>> (v17.x.y) is moving in another direction.
>>>
>>> On Wed, Mar 31, 2021 at 12:13 AM QP Hou <qp...@scribd.com.invalid> wrote:
>>>
>>>> Thanks Sumit for kicking off this discussion!
>>>>
>>>> I am in favor of approach #1 as well. Airflow software version can
>>>> change due to none API related changes, but the client should only
>>>> care about the public API contract. I believe this is also the same
>>>> approach that the k8s python client took as well? For example, the k8s
>>>> python client version 12.0.1 targets k8s api version 1.16.15.
>>>>
>>>> I am also with you that clients for different languages should have
>>>> their own release cycles. Other than testing burdens, every client
>>>> could introduce their own bug fixes and breaking changes without
>>>> having to trigger releases for all other clients. It would be a waste
>>>> of work for us to bump the python client version just for a bug fix in
>>>> the go client.
>>>>
>>>> Thanks,
>>>> QP
>>>>
>>>> On Tue, Mar 30, 2021 at 1:56 AM Sumit Maheshwari <ms...@apache.org>
>>>> wrote:
>>>> >
>>>> > Hello Airflow dev community
>>>> >
>>>> > I'm writing this mail in regards to seek feedback on the first
>>>> release of Apache Airflow's Python client. This is an icebreaker email only
>>>> and we will follow the Apache process of releasing the version later on. As
>>>> of now, I've raised a small PR to add the missing files in the repo, which
>>>> are required to do the release of the client lib.
>>>> >
>>>> > The main question I've in mind about the versioning. I see that there
>>>> are mainly two theories related to maintain the release versions of clients:
>>>> >
>>>> > The first is about following the semantic release of the clients on
>>>> its own. The release version of the client doesn't include the version of
>>>> the software itself. Something like we are doing with Airflow's provider
>>>> packages. However, in each release, we mention the release version of the
>>>> main software (Airflow in this case). This is the simple most way of
>>>> maintaining releases and followed majorly.
>>>> > The second way is to include the software's version as well in its
>>>> client's versioning, something like it's been done in the K8s Python
>>>> client. In this approach, the client version provides more implicit
>>>> knowledge about the compatible software, however it creates an extra burden
>>>> of releasing the client with every minor release of the software, otherwise
>>>> the versioning would fall apart. Also, there is a good chance that most of
>>>> these minor or even a major release has no changes in the APIs, but the
>>>> client would be needed to re-released nevertheless, just to keep the
>>>> versioning in check.
>>>> >
>>>> > Personally, I'm in favour of the 1st way, as it's simple to manage
>>>> and doesn't tightly couple client with its software.
>>>> >
>>>> > Another unrelated question I have is about the release of different
>>>> client libraries. As of now, Airflow has Python and Go clients, so should
>>>> we try to release them together or they should follow their own paths? In
>>>> general, I'm against batching them together, cause the person who is
>>>> releasing them may not be able to test all the different clients in the
>>>> future.
>>>> >
>>>> > Let me know your thoughts on these 2 points.
>>>> >
>>>> > Thanks,
>>>> > Sumit Maheshwari
>>>> > PMC Apache Airflow
>>>>
>>>

Re: Releasing Apache Airflow's Python Client

[Sumit Maheshwari <su...@gmail.com>]

Yeah totally agrees with you XD about the range vs the fixed version of the
main software. And with Airflow 2 dot & beyond, we would be following the
semantic versioning more judiciously, so with that can we assume that users
would be cognizant of the related Airflow version implicitly, or do you
think we should mention something like *2.0.X* in the changelog explicitly?

On Wed, Mar 31, 2021 at 1:38 PM Deng Xiaodong <xd...@gmail.com> wrote:

> Thanks, Sumit, for bringing this up.
>
> - Regarding *versioning for Python client*: I favour the 1st way as well.
> The only minor comment I have is: instead of "*mention the release
> version of the main software*", possibly what users would expect is a "*range
> of compatible versions of main software*". If we follow semver for
> Airflow itself relatively strictly, this should not be hard.
>
> - Regarding *the release of different client libraries*: I agree with
> you. They should be versioned & released separately.
>
>
> Regards,
> XD
>
> On Wed, Mar 31, 2021 at 9:21 AM Sumit Maheshwari <su...@gmail.com>
> wrote:
>
>> Thanks QP for your feedback. I think the latest release of K8s client
>> (v17.x.y) is moving in another direction.
>>
>> On Wed, Mar 31, 2021 at 12:13 AM QP Hou <qp...@scribd.com.invalid> wrote:
>>
>>> Thanks Sumit for kicking off this discussion!
>>>
>>> I am in favor of approach #1 as well. Airflow software version can
>>> change due to none API related changes, but the client should only
>>> care about the public API contract. I believe this is also the same
>>> approach that the k8s python client took as well? For example, the k8s
>>> python client version 12.0.1 targets k8s api version 1.16.15.
>>>
>>> I am also with you that clients for different languages should have
>>> their own release cycles. Other than testing burdens, every client
>>> could introduce their own bug fixes and breaking changes without
>>> having to trigger releases for all other clients. It would be a waste
>>> of work for us to bump the python client version just for a bug fix in
>>> the go client.
>>>
>>> Thanks,
>>> QP
>>>
>>> On Tue, Mar 30, 2021 at 1:56 AM Sumit Maheshwari <ms...@apache.org>
>>> wrote:
>>> >
>>> > Hello Airflow dev community
>>> >
>>> > I'm writing this mail in regards to seek feedback on the first release
>>> of Apache Airflow's Python client. This is an icebreaker email only and we
>>> will follow the Apache process of releasing the version later on. As of
>>> now, I've raised a small PR to add the missing files in the repo, which are
>>> required to do the release of the client lib.
>>> >
>>> > The main question I've in mind about the versioning. I see that there
>>> are mainly two theories related to maintain the release versions of clients:
>>> >
>>> > The first is about following the semantic release of the clients on
>>> its own. The release version of the client doesn't include the version of
>>> the software itself. Something like we are doing with Airflow's provider
>>> packages. However, in each release, we mention the release version of the
>>> main software (Airflow in this case). This is the simple most way of
>>> maintaining releases and followed majorly.
>>> > The second way is to include the software's version as well in its
>>> client's versioning, something like it's been done in the K8s Python
>>> client. In this approach, the client version provides more implicit
>>> knowledge about the compatible software, however it creates an extra burden
>>> of releasing the client with every minor release of the software, otherwise
>>> the versioning would fall apart. Also, there is a good chance that most of
>>> these minor or even a major release has no changes in the APIs, but the
>>> client would be needed to re-released nevertheless, just to keep the
>>> versioning in check.
>>> >
>>> > Personally, I'm in favour of the 1st way, as it's simple to manage and
>>> doesn't tightly couple client with its software.
>>> >
>>> > Another unrelated question I have is about the release of different
>>> client libraries. As of now, Airflow has Python and Go clients, so should
>>> we try to release them together or they should follow their own paths? In
>>> general, I'm against batching them together, cause the person who is
>>> releasing them may not be able to test all the different clients in the
>>> future.
>>> >
>>> > Let me know your thoughts on these 2 points.
>>> >
>>> > Thanks,
>>> > Sumit Maheshwari
>>> > PMC Apache Airflow
>>>
>>

Re: Releasing Apache Airflow's Python Client

[Deng Xiaodong <xd...@gmail.com>]

Thanks, Sumit, for bringing this up.

- Regarding *versioning for Python client*: I favour the 1st way as well.
The only minor comment I have is: instead of "*mention the release version
of the main software*", possibly what users would expect is a "*range of
compatible versions of main software*". If we follow semver for Airflow
itself relatively strictly, this should not be hard.

- Regarding *the release of different client libraries*: I agree with you.
They should be versioned & released separately.


Regards,
XD

On Wed, Mar 31, 2021 at 9:21 AM Sumit Maheshwari <su...@gmail.com>
wrote:

> Thanks QP for your feedback. I think the latest release of K8s client
> (v17.x.y) is moving in another direction.
>
> On Wed, Mar 31, 2021 at 12:13 AM QP Hou <qp...@scribd.com.invalid> wrote:
>
>> Thanks Sumit for kicking off this discussion!
>>
>> I am in favor of approach #1 as well. Airflow software version can
>> change due to none API related changes, but the client should only
>> care about the public API contract. I believe this is also the same
>> approach that the k8s python client took as well? For example, the k8s
>> python client version 12.0.1 targets k8s api version 1.16.15.
>>
>> I am also with you that clients for different languages should have
>> their own release cycles. Other than testing burdens, every client
>> could introduce their own bug fixes and breaking changes without
>> having to trigger releases for all other clients. It would be a waste
>> of work for us to bump the python client version just for a bug fix in
>> the go client.
>>
>> Thanks,
>> QP
>>
>> On Tue, Mar 30, 2021 at 1:56 AM Sumit Maheshwari <ms...@apache.org>
>> wrote:
>> >
>> > Hello Airflow dev community
>> >
>> > I'm writing this mail in regards to seek feedback on the first release
>> of Apache Airflow's Python client. This is an icebreaker email only and we
>> will follow the Apache process of releasing the version later on. As of
>> now, I've raised a small PR to add the missing files in the repo, which are
>> required to do the release of the client lib.
>> >
>> > The main question I've in mind about the versioning. I see that there
>> are mainly two theories related to maintain the release versions of clients:
>> >
>> > The first is about following the semantic release of the clients on its
>> own. The release version of the client doesn't include the version of the
>> software itself. Something like we are doing with Airflow's provider
>> packages. However, in each release, we mention the release version of the
>> main software (Airflow in this case). This is the simple most way of
>> maintaining releases and followed majorly.
>> > The second way is to include the software's version as well in its
>> client's versioning, something like it's been done in the K8s Python
>> client. In this approach, the client version provides more implicit
>> knowledge about the compatible software, however it creates an extra burden
>> of releasing the client with every minor release of the software, otherwise
>> the versioning would fall apart. Also, there is a good chance that most of
>> these minor or even a major release has no changes in the APIs, but the
>> client would be needed to re-released nevertheless, just to keep the
>> versioning in check.
>> >
>> > Personally, I'm in favour of the 1st way, as it's simple to manage and
>> doesn't tightly couple client with its software.
>> >
>> > Another unrelated question I have is about the release of different
>> client libraries. As of now, Airflow has Python and Go clients, so should
>> we try to release them together or they should follow their own paths? In
>> general, I'm against batching them together, cause the person who is
>> releasing them may not be able to test all the different clients in the
>> future.
>> >
>> > Let me know your thoughts on these 2 points.
>> >
>> > Thanks,
>> > Sumit Maheshwari
>> > PMC Apache Airflow
>>
>

Re: Releasing Apache Airflow's Python Client

[Sumit Maheshwari <su...@gmail.com>]

Thanks QP for your feedback. I think the latest release of K8s client
(v17.x.y) is moving in another direction.

On Wed, Mar 31, 2021 at 12:13 AM QP Hou <qp...@scribd.com.invalid> wrote:

> Thanks Sumit for kicking off this discussion!
>
> I am in favor of approach #1 as well. Airflow software version can
> change due to none API related changes, but the client should only
> care about the public API contract. I believe this is also the same
> approach that the k8s python client took as well? For example, the k8s
> python client version 12.0.1 targets k8s api version 1.16.15.
>
> I am also with you that clients for different languages should have
> their own release cycles. Other than testing burdens, every client
> could introduce their own bug fixes and breaking changes without
> having to trigger releases for all other clients. It would be a waste
> of work for us to bump the python client version just for a bug fix in
> the go client.
>
> Thanks,
> QP
>
> On Tue, Mar 30, 2021 at 1:56 AM Sumit Maheshwari <ms...@apache.org>
> wrote:
> >
> > Hello Airflow dev community
> >
> > I'm writing this mail in regards to seek feedback on the first release
> of Apache Airflow's Python client. This is an icebreaker email only and we
> will follow the Apache process of releasing the version later on. As of
> now, I've raised a small PR to add the missing files in the repo, which are
> required to do the release of the client lib.
> >
> > The main question I've in mind about the versioning. I see that there
> are mainly two theories related to maintain the release versions of clients:
> >
> > The first is about following the semantic release of the clients on its
> own. The release version of the client doesn't include the version of the
> software itself. Something like we are doing with Airflow's provider
> packages. However, in each release, we mention the release version of the
> main software (Airflow in this case). This is the simple most way of
> maintaining releases and followed majorly.
> > The second way is to include the software's version as well in its
> client's versioning, something like it's been done in the K8s Python
> client. In this approach, the client version provides more implicit
> knowledge about the compatible software, however it creates an extra burden
> of releasing the client with every minor release of the software, otherwise
> the versioning would fall apart. Also, there is a good chance that most of
> these minor or even a major release has no changes in the APIs, but the
> client would be needed to re-released nevertheless, just to keep the
> versioning in check.
> >
> > Personally, I'm in favour of the 1st way, as it's simple to manage and
> doesn't tightly couple client with its software.
> >
> > Another unrelated question I have is about the release of different
> client libraries. As of now, Airflow has Python and Go clients, so should
> we try to release them together or they should follow their own paths? In
> general, I'm against batching them together, cause the person who is
> releasing them may not be able to test all the different clients in the
> future.
> >
> > Let me know your thoughts on these 2 points.
> >
> > Thanks,
> > Sumit Maheshwari
> > PMC Apache Airflow
>

Re: Releasing Apache Airflow's Python Client

[QP Hou <qp...@scribd.com.INVALID>]

Thanks Sumit for kicking off this discussion!

I am in favor of approach #1 as well. Airflow software version can
change due to none API related changes, but the client should only
care about the public API contract. I believe this is also the same
approach that the k8s python client took as well? For example, the k8s
python client version 12.0.1 targets k8s api version 1.16.15.

I am also with you that clients for different languages should have
their own release cycles. Other than testing burdens, every client
could introduce their own bug fixes and breaking changes without
having to trigger releases for all other clients. It would be a waste
of work for us to bump the python client version just for a bug fix in
the go client.

Thanks,
QP

On Tue, Mar 30, 2021 at 1:56 AM Sumit Maheshwari <ms...@apache.org> wrote:
>
> Hello Airflow dev community
>
> I'm writing this mail in regards to seek feedback on the first release of Apache Airflow's Python client. This is an icebreaker email only and we will follow the Apache process of releasing the version later on. As of now, I've raised a small PR to add the missing files in the repo, which are required to do the release of the client lib.
>
> The main question I've in mind about the versioning. I see that there are mainly two theories related to maintain the release versions of clients:
>
> The first is about following the semantic release of the clients on its own. The release version of the client doesn't include the version of the software itself. Something like we are doing with Airflow's provider packages. However, in each release, we mention the release version of the main software (Airflow in this case). This is the simple most way of maintaining releases and followed majorly.
> The second way is to include the software's version as well in its client's versioning, something like it's been done in the K8s Python client. In this approach, the client version provides more implicit knowledge about the compatible software, however it creates an extra burden of releasing the client with every minor release of the software, otherwise the versioning would fall apart. Also, there is a good chance that most of these minor or even a major release has no changes in the APIs, but the client would be needed to re-released nevertheless, just to keep the versioning in check.
>
> Personally, I'm in favour of the 1st way, as it's simple to manage and doesn't tightly couple client with its software.
>
> Another unrelated question I have is about the release of different client libraries. As of now, Airflow has Python and Go clients, so should we try to release them together or they should follow their own paths? In general, I'm against batching them together, cause the person who is releasing them may not be able to test all the different clients in the future.
>
> Let me know your thoughts on these 2 points.
>
> Thanks,
> Sumit Maheshwari
> PMC Apache Airflow