You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <ja...@potiuk.com> on 2021/09/29 22:14:55 UTC
Docker Provider min version ?
Hello Everyone,
I am just about to release the September Providers, but Kamil raised a
valid concern and he convinced me that it might be a problem if we release
a new Docker Provider with minimum airflow version 2.1.
The new provider has the new "Docker Decorator" feature which will be
available only in 2.2. But other than that - it is backwards compatible -
so you could install it on Airflow 2.1 and have the "classic" operator
work. But if you try to use the decorator, you will be able to write a Dag
with this decorator, but it simply won't work (and you will have no message
about it).
The Docker Provider has no other changes nor fixes.
I am tempted (and quite convinced) to increase the min airflow version of
the Docker Provider to 2.2 so that it can be only installed there (and we
make it ready for the 2.2 release).
Does anyone have a strong opinion on that ?
J.
Re: Docker Provider min version ?
Posted by Collin McNulty <co...@astronomer.io.INVALID>.
I agree with Kamil. I do not think we should knowingly release something
that can fail silently in a “compatible” configuration.
Collin McNulty
On Wed, Sep 29, 2021 at 5:23 PM Xiaodong Deng <xd...@apache.org> wrote:
> Hi Jarek,
>
> I have no strong opinion on this, but would like to have clarification on
> one question: I understand you highlighted there is no changes nor fixes in
> the provider, but what if in the future we need to fix something for the
> “classic” operator which may still be used in Airflow 2.1 or even lower
> version installations? How will releases etc. be managed in such cases?
>
> I may have misunderstood or missed something, and be asking a dumb
> question. Please correct me in that case.
>
> Thanks!
>
>
> XD
>
> On Wed, Sep 29, 2021 at 22:15 Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> Hello Everyone,
>>
>> I am just about to release the September Providers, but Kamil raised a
>> valid concern and he convinced me that it might be a problem if we release
>> a new Docker Provider with minimum airflow version 2.1.
>>
>> The new provider has the new "Docker Decorator" feature which will be
>> available only in 2.2. But other than that - it is backwards compatible -
>> so you could install it on Airflow 2.1 and have the "classic" operator
>> work. But if you try to use the decorator, you will be able to write a Dag
>> with this decorator, but it simply won't work (and you will have no message
>> about it).
>>
>> The Docker Provider has no other changes nor fixes.
>>
>> I am tempted (and quite convinced) to increase the min airflow version of
>> the Docker Provider to 2.2 so that it can be only installed there (and we
>> make it ready for the 2.2 release).
>>
>> Does anyone have a strong opinion on that ?
>>
>> J.
>>
>>
>>
>>
Re: Docker Provider min version ?
Posted by Jarek Potiuk <ja...@potiuk.com>.
Thanks for the input everyone.
I have updated the docs and changelog, and put the exact error message in
the docs (both the new Taskflow API docs and the Changelog of the provider)
. Hopefully Google indexes it and if users have the error message when
trying to use the task.decorator - I think that should be "good enough".
The docs updated here: https://github.com/apache/airflow/pull/18613
J.
On Thu, Sep 30, 2021 at 12:32 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>
> On Thu, Sep 30, 2021 at 9:56 AM Kamil Breguła <dz...@gmail.com> wrote:
>
>> If we release this package, the user will not be able to test it in
>> any released version as such a version does not exist yet. This could
>> also potentially mean that it will never work, as unreleased versions
>> of Airflow may have breaking API changes. We only maintain API
>> compatibility between released versions of Airflow. Airflow developer
>> versions may have breaking changes, and that's fine as long as this
>> issue is fixed before a new version is released.
>>
>>
> We will not bump the version now (as task.decorator is not silently
> falling - I just double checked it).
>
> But even if we would - we should release it before with >= 2.2. It's quite
> the opposite from what you say. Users who are currently testing beta and
> RCS of 2.2 will be able to test both RC AND released versions of Docker
> provider only when we prepare and release and make it ready for them to
> test the RC.
> It's by NOT releasing it earlier we do not give the users to test it
> before 2.2 is released. By releasing it now, we give a chance to test it
> with current beta, later rcs and if the users will find a problem we can
> fix it in the future patchlevel of the provider.
>
>
>> czw., 30 wrz 2021 o 08:32 Kaxil Naik <ka...@gmail.com> napisał(a):
>> >
>> > Agree with Kamil here. We should just adjust the versioning of the
>> docker provider and bump the Minor version to indicate New Feature.
>> >
>> > To XD's question - we can release a patch version in that minor version
>> series for Docker Provider if we find a critical fix.
>> >
>> > Regards,
>> > Kaxil
>> >
>> > On Thu, Sep 30, 2021, 04:03 Jarek Potiuk <ja...@potiuk.com> wrote:
>> >>>
>> >>>
>> >>> I have no strong opinion on this, but would like to have
>> clarification on one question: I understand you highlighted there is no
>> changes nor fixes in the provider, but what if in the future we need to fix
>> something for the “classic” operator which may still be used in Airflow 2.1
>> or even lower version installations? How will releases etc. be managed in
>> such cases?
>> >>
>> >>
>> >> Yep. Good question. This is about the only "potential" issue -
>> currently we have no process for that as we've never had to do it. However
>> it is always possible to do it if and when we need it. If we find that
>> there is a critical fix for Docker Provider that we would need to fix, we
>> can always branch-off from the last released version that works for 2.1 and
>> cherry-pick and release a patched version.
>> >>
>> >> We did not see a need for that so far. We released all providers when
>> we released 2.1.0 which are 2.1+ only (because of apply_defaults) and so
>> far there was not a case where anyone raised a need to release any bugfix
>> for previous versions. With our SemVer approach anyone should be "SAFE"
>> migrating to newer version of Airflow (even 2.2) as they are all supposed
>> to be backwards compatible, so I think our approach should be "If you need
>> anything that we release for the DockerProvider in the future - migrate to
>> 2.2. But this is a "default" approach I think and I see the case of a
>> really critical issue that will result in releasing a patchlevel version
>> for the previous release.
>> >>
>> >> Again - we have not done that yet, and we want to avoid it for as long
>> as we can, but there is nothing (except the overhead on cherry-picking and
>> releasing it) to prevent us from doing so. We have all history, tags,
>> changelogs and mechanisms to selectively release only one provider if
>> needed. But I think it should be really Ad-Hoc and when we REALLY REALLY
>> need it.
>> >>
>> >> J.
>> >>
>> >>> I may have misunderstood or missed something, and be asking a dumb
>> question. Please correct me in that case.
>> >>>
>> >>> Thanks!
>> >>>
>> >>>
>> >>> XD
>> >>>
>> >>> On Wed, Sep 29, 2021 at 22:15 Jarek Potiuk <ja...@potiuk.com> wrote:
>> >>>>
>> >>>> Hello Everyone,
>> >>>>
>> >>>> I am just about to release the September Providers, but Kamil raised
>> a valid concern and he convinced me that it might be a problem if we
>> release a new Docker Provider with minimum airflow version 2.1.
>> >>>>
>> >>>> The new provider has the new "Docker Decorator" feature which will
>> be available only in 2.2. But other than that - it is backwards compatible
>> - so you could install it on Airflow 2.1 and have the "classic" operator
>> work. But if you try to use the decorator, you will be able to write a Dag
>> with this decorator, but it simply won't work (and you will have no message
>> about it).
>> >>>>
>> >>>> The Docker Provider has no other changes nor fixes.
>> >>>>
>> >>>> I am tempted (and quite convinced) to increase the min airflow
>> version of the Docker Provider to 2.2 so that it can be only installed
>> there (and we make it ready for the 2.2 release).
>> >>>>
>> >>>> Does anyone have a strong opinion on that ?
>> >>>>
>> >>>> J.
>> >>>>
>> >>>>
>> >>>>
>>
>
Re: Docker Provider min version ?
Posted by Jarek Potiuk <ja...@potiuk.com>.
On Thu, Sep 30, 2021 at 9:56 AM Kamil Breguła <dz...@gmail.com> wrote:
> If we release this package, the user will not be able to test it in
> any released version as such a version does not exist yet. This could
> also potentially mean that it will never work, as unreleased versions
> of Airflow may have breaking API changes. We only maintain API
> compatibility between released versions of Airflow. Airflow developer
> versions may have breaking changes, and that's fine as long as this
> issue is fixed before a new version is released.
>
>
We will not bump the version now (as task.decorator is not silently
falling - I just double checked it).
But even if we would - we should release it before with >= 2.2. It's quite
the opposite from what you say. Users who are currently testing beta and
RCS of 2.2 will be able to test both RC AND released versions of Docker
provider only when we prepare and release and make it ready for them to
test the RC.
It's by NOT releasing it earlier we do not give the users to test it before
2.2 is released. By releasing it now, we give a chance to test it with
current beta, later rcs and if the users will find a problem we can fix it
in the future patchlevel of the provider.
> czw., 30 wrz 2021 o 08:32 Kaxil Naik <ka...@gmail.com> napisał(a):
> >
> > Agree with Kamil here. We should just adjust the versioning of the
> docker provider and bump the Minor version to indicate New Feature.
> >
> > To XD's question - we can release a patch version in that minor version
> series for Docker Provider if we find a critical fix.
> >
> > Regards,
> > Kaxil
> >
> > On Thu, Sep 30, 2021, 04:03 Jarek Potiuk <ja...@potiuk.com> wrote:
> >>>
> >>>
> >>> I have no strong opinion on this, but would like to have clarification
> on one question: I understand you highlighted there is no changes nor fixes
> in the provider, but what if in the future we need to fix something for the
> “classic” operator which may still be used in Airflow 2.1 or even lower
> version installations? How will releases etc. be managed in such cases?
> >>
> >>
> >> Yep. Good question. This is about the only "potential" issue -
> currently we have no process for that as we've never had to do it. However
> it is always possible to do it if and when we need it. If we find that
> there is a critical fix for Docker Provider that we would need to fix, we
> can always branch-off from the last released version that works for 2.1 and
> cherry-pick and release a patched version.
> >>
> >> We did not see a need for that so far. We released all providers when
> we released 2.1.0 which are 2.1+ only (because of apply_defaults) and so
> far there was not a case where anyone raised a need to release any bugfix
> for previous versions. With our SemVer approach anyone should be "SAFE"
> migrating to newer version of Airflow (even 2.2) as they are all supposed
> to be backwards compatible, so I think our approach should be "If you need
> anything that we release for the DockerProvider in the future - migrate to
> 2.2. But this is a "default" approach I think and I see the case of a
> really critical issue that will result in releasing a patchlevel version
> for the previous release.
> >>
> >> Again - we have not done that yet, and we want to avoid it for as long
> as we can, but there is nothing (except the overhead on cherry-picking and
> releasing it) to prevent us from doing so. We have all history, tags,
> changelogs and mechanisms to selectively release only one provider if
> needed. But I think it should be really Ad-Hoc and when we REALLY REALLY
> need it.
> >>
> >> J.
> >>
> >>> I may have misunderstood or missed something, and be asking a dumb
> question. Please correct me in that case.
> >>>
> >>> Thanks!
> >>>
> >>>
> >>> XD
> >>>
> >>> On Wed, Sep 29, 2021 at 22:15 Jarek Potiuk <ja...@potiuk.com> wrote:
> >>>>
> >>>> Hello Everyone,
> >>>>
> >>>> I am just about to release the September Providers, but Kamil raised
> a valid concern and he convinced me that it might be a problem if we
> release a new Docker Provider with minimum airflow version 2.1.
> >>>>
> >>>> The new provider has the new "Docker Decorator" feature which will be
> available only in 2.2. But other than that - it is backwards compatible -
> so you could install it on Airflow 2.1 and have the "classic" operator
> work. But if you try to use the decorator, you will be able to write a Dag
> with this decorator, but it simply won't work (and you will have no message
> about it).
> >>>>
> >>>> The Docker Provider has no other changes nor fixes.
> >>>>
> >>>> I am tempted (and quite convinced) to increase the min airflow
> version of the Docker Provider to 2.2 so that it can be only installed
> there (and we make it ready for the 2.2 release).
> >>>>
> >>>> Does anyone have a strong opinion on that ?
> >>>>
> >>>> J.
> >>>>
> >>>>
> >>>>
>
Re: Docker Provider min version ?
Posted by Ash Berlin-Taylor <as...@apache.org>.
On Thu, Sep 30 2021 at 09:46:15 +0100, Ash Berlin-Taylor
<as...@apache.org> wrote:
> but that Airflow 2.1 will use
But that Airflow 2.1 *will ignore*, (and the schema allows extra fields
without an error)
Re: Docker Provider min version ?
Posted by Jarek Potiuk <ja...@potiuk.com>.
Yep. I just checked it, and yep. We can't do more than just update the
documentation/faq about that/mention in the changelog. I am going to do
just that
On Thu, Sep 30, 2021 at 12:13 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> There's nothing we can add in to the provider for this case -- as if the
> user tries `@task.docker` on Airflow 2.1 no code in the provider is ever
> run.
>
> -a
>
> On Thu, Sep 30 2021 at 12:03:34 +0200, Jarek Potiuk <ja...@potiuk.com>
> wrote:
>
> On Thu, Sep 30, 2021 at 10:46 AM Ash Berlin-Taylor <as...@apache.org> wrote:
>
>> The new docker provider will still install and work on Airflow 2.1 -- the
>> only change is to add a new field to the provider info dict
>> (`task-decorators`) that Airflow 2.2 will read, but that Airflow 2.1 will
>> use.
>>
>> If you try to use `@task.docker` in Airflow it will fail with
>> "_TaskDecorator has no property docker" which I think is fair -- and the
>> docs about this feature say (
>> http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/tutorial_taskflow_api.html#using-the-taskflow-api-with-docker-or-virtual-environments
>> ):
>>
>>
> Right - if using the task.docker in Airflow 2.1 is not a silent ignore but
> a failure, then I agree - we should not bump the minimum version.
> I guess however that we could give a clearer error message?
>
> We've had too many of those cases where our error messages did not really
> tell the user what is the reason - resulting in the users opening issues
> and asking "I have no idea what this is about" (which I believe will happen
> in this case).
>
> Is it possible to add it quickly before we release the provider? Something
> like ("Note That this decorator only works for Airflow 2.2+ - if you use an
> earlier version please migrate to 2.2 or above)"
>
> J.
>
>
Re: Docker Provider min version ?
Posted by Ash Berlin-Taylor <as...@apache.org>.
There's nothing we can add in to the provider for this case -- as if
the user tries `@task.docker` on Airflow 2.1 no code in the provider is
ever run.
-a
On Thu, Sep 30 2021 at 12:03:34 +0200, Jarek Potiuk <ja...@potiuk.com>
wrote:
> On Thu, Sep 30, 2021 at 10:46 AM Ash Berlin-Taylor <ash@apache.org
> <ma...@apache.org>> wrote:
>> The new docker provider will still install and work on Airflow 2.1
>> -- the only change is to add a new field to the provider info dict
>> (`task-decorators`) that Airflow 2.2 will read, but that Airflow 2.1
>> will use.
>>
>> If you try to use `@task.docker` in Airflow it will fail with
>> "_TaskDecorator has no property docker" which I think is fair -- and
>> the docs about this feature say
>> (<http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/tutorial_taskflow_api.html#using-the-taskflow-api-with-docker-or-virtual-environments>):
>>
>
> Right - if using the task.docker in Airflow 2.1 is not a silent
> ignore but a failure, then I agree - we should not bump the minimum
> version.
> I guess however that we could give a clearer error message?
>
> We've had too many of those cases where our error messages did not
> really tell the user what is the reason - resulting in the users
> opening issues and asking "I have no idea what this is about" (which
> I believe will happen in this case).
>
> Is it possible to add it quickly before we release the provider?
> Something like ("Note That this decorator only works for Airflow 2.2+
> - if you use an earlier version please migrate to 2.2 or above)"
>
> J.
>
>>>>>>>
Re: Docker Provider min version ?
Posted by Jarek Potiuk <ja...@potiuk.com>.
On Thu, Sep 30, 2021 at 10:46 AM Ash Berlin-Taylor <as...@apache.org> wrote:
> The new docker provider will still install and work on Airflow 2.1 -- the
> only change is to add a new field to the provider info dict
> (`task-decorators`) that Airflow 2.2 will read, but that Airflow 2.1 will
> use.
>
> If you try to use `@task.docker` in Airflow it will fail with
> "_TaskDecorator has no property docker" which I think is fair -- and the
> docs about this feature say (
> http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/tutorial_taskflow_api.html#using-the-taskflow-api-with-docker-or-virtual-environments
> ):
>
>
Right - if using the task.docker in Airflow 2.1 is not a silent ignore but
a failure, then I agree - we should not bump the minimum version.
I guess however that we could give a clearer error message?
We've had too many of those cases where our error messages did not really
tell the user what is the reason - resulting in the users opening issues
and asking "I have no idea what this is about" (which I believe will happen
in this case).
Is it possible to add it quickly before we release the provider? Something
like ("Note That this decorator only works for Airflow 2.2+ - if you use an
earlier version please migrate to 2.2 or above)"
J.
Re: Docker Provider min version ?
Posted by Ash Berlin-Taylor <as...@apache.org>.
The new docker provider will still install and work on Airflow 2.1 --
the only change is to add a new field to the provider info dict
(`task-decorators`) that Airflow 2.2 will read, but that Airflow 2.1
will use.
If you try to use `@task.docker` in Airflow it will fail with
"_TaskDecorator has no property docker" which I think is fair -- and
the docs about this feature say
(<http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/tutorial_taskflow_api.html#using-the-taskflow-api-with-docker-or-virtual-environments>):
> If you have tasks that require complex or conflicting requirements
then you will have the ability to use the Taskflow API with either a
Docker container (since version 2.2.0) or Python virtual environment
(since 2.0.2).
So I say we should not change the minimum version of the provider and
just bump the minor version as normal when releasing a new feature --
yes, that new feature isn't available on Airflow 2.1.x but nothing
would break so I think this is fine.
-ash
On Thu, Sep 30 2021 at 09:56:23 +0200, Kamil Breguła
<dz...@gmail.com> wrote:
> If we release this package, the user will not be able to test it in
> any released version as such a version does not exist yet. This could
> also potentially mean that it will never work, as unreleased versions
> of Airflow may have breaking API changes. We only maintain API
> compatibility between released versions of Airflow. Airflow developer
> versions may have breaking changes, and that's fine as long as this
> issue is fixed before a new version is released.
>
> czw., 30 wrz 2021 o 08:32 Kaxil Naik <kaxilnaik@gmail.com
> <ma...@gmail.com>> napisał(a):
>>
>> Agree with Kamil here. We should just adjust the versioning of the
>> docker provider and bump the Minor version to indicate New Feature.
>>
>> To XD's question - we can release a patch version in that minor
>> version series for Docker Provider if we find a critical fix.
>>
>> Regards,
>> Kaxil
>>
>> On Thu, Sep 30, 2021, 04:03 Jarek Potiuk <jarek@potiuk.com
>> <ma...@potiuk.com>> wrote:
>>>>
>>>>
>>>> I have no strong opinion on this, but would like to have
>>>> clarification on one question: I understand you highlighted there
>>>> is no changes nor fixes in the provider, but what if in the future
>>>> we need to fix something for the “classic” operator which may
>>>> still be used in Airflow 2.1 or even lower version installations?
>>>> How will releases etc. be managed in such cases?
>>>
>>>
>>> Yep. Good question. This is about the only "potential" issue -
>>> currently we have no process for that as we've never had to do it.
>>> However it is always possible to do it if and when we need it. If
>>> we find that there is a critical fix for Docker Provider that we
>>> would need to fix, we can always branch-off from the last released
>>> version that works for 2.1 and cherry-pick and release a patched
>>> version.
>>>
>>> We did not see a need for that so far. We released all providers
>>> when we released 2.1.0 which are 2.1+ only (because of
>>> apply_defaults) and so far there was not a case where anyone raised
>>> a need to release any bugfix for previous versions. With our SemVer
>>> approach anyone should be "SAFE" migrating to newer version of
>>> Airflow (even 2.2) as they are all supposed to be backwards
>>> compatible, so I think our approach should be "If you need anything
>>> that we release for the DockerProvider in the future - migrate to
>>> 2.2. But this is a "default" approach I think and I see the case
>>> of a really critical issue that will result in releasing a
>>> patchlevel version for the previous release.
>>>
>>> Again - we have not done that yet, and we want to avoid it for as
>>> long as we can, but there is nothing (except the overhead on
>>> cherry-picking and releasing it) to prevent us from doing so. We
>>> have all history, tags, changelogs and mechanisms to selectively
>>> release only one provider if needed. But I think it should be
>>> really Ad-Hoc and when we REALLY REALLY need it.
>>>
>>> J.
>>>
>>>> I may have misunderstood or missed something, and be asking a
>>>> dumb question. Please correct me in that case.
>>>>
>>>> Thanks!
>>>>
>>>>
>>>> XD
>>>>
>>>> On Wed, Sep 29, 2021 at 22:15 Jarek Potiuk <jarek@potiuk.com
>>>> <ma...@potiuk.com>> wrote:
>>>>>
>>>>> Hello Everyone,
>>>>>
>>>>> I am just about to release the September Providers, but Kamil
>>>>> raised a valid concern and he convinced me that it might be a
>>>>> problem if we release a new Docker Provider with minimum airflow
>>>>> version 2.1.
>>>>>
>>>>> The new provider has the new "Docker Decorator" feature which
>>>>> will be available only in 2.2. But other than that - it is
>>>>> backwards compatible - so you could install it on Airflow 2.1 and
>>>>> have the "classic" operator work. But if you try to use the
>>>>> decorator, you will be able to write a Dag with this decorator,
>>>>> but it simply won't work (and you will have no message about it).
>>>>>
>>>>> The Docker Provider has no other changes nor fixes.
>>>>>
>>>>> I am tempted (and quite convinced) to increase the min airflow
>>>>> version of the Docker Provider to 2.2 so that it can be only
>>>>> installed there (and we make it ready for the 2.2 release).
>>>>>
>>>>> Does anyone have a strong opinion on that ?
>>>>>
>>>>> J.
>>>>>
>>>>>
>>>>>
Re: Docker Provider min version ?
Posted by Kamil Breguła <dz...@gmail.com>.
If we release this package, the user will not be able to test it in
any released version as such a version does not exist yet. This could
also potentially mean that it will never work, as unreleased versions
of Airflow may have breaking API changes. We only maintain API
compatibility between released versions of Airflow. Airflow developer
versions may have breaking changes, and that's fine as long as this
issue is fixed before a new version is released.
czw., 30 wrz 2021 o 08:32 Kaxil Naik <ka...@gmail.com> napisał(a):
>
> Agree with Kamil here. We should just adjust the versioning of the docker provider and bump the Minor version to indicate New Feature.
>
> To XD's question - we can release a patch version in that minor version series for Docker Provider if we find a critical fix.
>
> Regards,
> Kaxil
>
> On Thu, Sep 30, 2021, 04:03 Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>>
>>> I have no strong opinion on this, but would like to have clarification on one question: I understand you highlighted there is no changes nor fixes in the provider, but what if in the future we need to fix something for the “classic” operator which may still be used in Airflow 2.1 or even lower version installations? How will releases etc. be managed in such cases?
>>
>>
>> Yep. Good question. This is about the only "potential" issue - currently we have no process for that as we've never had to do it. However it is always possible to do it if and when we need it. If we find that there is a critical fix for Docker Provider that we would need to fix, we can always branch-off from the last released version that works for 2.1 and cherry-pick and release a patched version.
>>
>> We did not see a need for that so far. We released all providers when we released 2.1.0 which are 2.1+ only (because of apply_defaults) and so far there was not a case where anyone raised a need to release any bugfix for previous versions. With our SemVer approach anyone should be "SAFE" migrating to newer version of Airflow (even 2.2) as they are all supposed to be backwards compatible, so I think our approach should be "If you need anything that we release for the DockerProvider in the future - migrate to 2.2. But this is a "default" approach I think and I see the case of a really critical issue that will result in releasing a patchlevel version for the previous release.
>>
>> Again - we have not done that yet, and we want to avoid it for as long as we can, but there is nothing (except the overhead on cherry-picking and releasing it) to prevent us from doing so. We have all history, tags, changelogs and mechanisms to selectively release only one provider if needed. But I think it should be really Ad-Hoc and when we REALLY REALLY need it.
>>
>> J.
>>
>>> I may have misunderstood or missed something, and be asking a dumb question. Please correct me in that case.
>>>
>>> Thanks!
>>>
>>>
>>> XD
>>>
>>> On Wed, Sep 29, 2021 at 22:15 Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>
>>>> Hello Everyone,
>>>>
>>>> I am just about to release the September Providers, but Kamil raised a valid concern and he convinced me that it might be a problem if we release a new Docker Provider with minimum airflow version 2.1.
>>>>
>>>> The new provider has the new "Docker Decorator" feature which will be available only in 2.2. But other than that - it is backwards compatible - so you could install it on Airflow 2.1 and have the "classic" operator work. But if you try to use the decorator, you will be able to write a Dag with this decorator, but it simply won't work (and you will have no message about it).
>>>>
>>>> The Docker Provider has no other changes nor fixes.
>>>>
>>>> I am tempted (and quite convinced) to increase the min airflow version of the Docker Provider to 2.2 so that it can be only installed there (and we make it ready for the 2.2 release).
>>>>
>>>> Does anyone have a strong opinion on that ?
>>>>
>>>> J.
>>>>
>>>>
>>>>
Re: Docker Provider min version ?
Posted by Kaxil Naik <ka...@gmail.com>.
Agree with Kamil here. We should just adjust the versioning of the docker
provider and bump the Minor version to indicate New Feature.
To XD's question - we can release a patch version in that minor version
series for Docker Provider if we find a critical fix.
Regards,
Kaxil
On Thu, Sep 30, 2021, 04:03 Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> I have no strong opinion on this, but would like to have clarification on
>> one question: I understand you highlighted there is no changes nor fixes in
>> the provider, but what if in the future we need to fix something for the
>> “classic” operator which may still be used in Airflow 2.1 or even lower
>> version installations? How will releases etc. be managed in such cases?
>>
>
> Yep. Good question. This is about the only "potential" issue - currently
> we have no process for that as we've never had to do it. However it is
> always possible to do it if and when we need it. If we find that there is a
> critical fix for Docker Provider that we would need to fix, we can always
> branch-off from the last released version that works for 2.1 and
> cherry-pick and release a patched version.
>
> We did not see a need for that so far. We released all providers when we
> released 2.1.0 which are 2.1+ only (because of apply_defaults) and so far
> there was not a case where anyone raised a need to release any bugfix for
> previous versions. With our SemVer approach anyone should be "SAFE"
> migrating to newer version of Airflow (even 2.2) as they are all supposed
> to be backwards compatible, so I think our approach should be "If you need
> anything that we release for the DockerProvider in the future - migrate to
> 2.2. But this is a "default" approach I think and I see the case of a
> really critical issue that will result in releasing a patchlevel version
> for the previous release.
>
> Again - we have not done that yet, and we want to avoid it for as long as
> we can, but there is nothing (except the overhead on cherry-picking and
> releasing it) to prevent us from doing so. We have all history, tags,
> changelogs and mechanisms to selectively release only one provider if
> needed. But I think it should be really Ad-Hoc and when we REALLY REALLY
> need it.
>
> J.
>
> I may have misunderstood or missed something, and be asking a dumb
>> question. Please correct me in that case.
>>
>> Thanks!
>>
>>
>> XD
>>
>> On Wed, Sep 29, 2021 at 22:15 Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>>> Hello Everyone,
>>>
>>> I am just about to release the September Providers, but Kamil raised a
>>> valid concern and he convinced me that it might be a problem if we release
>>> a new Docker Provider with minimum airflow version 2.1.
>>>
>>> The new provider has the new "Docker Decorator" feature which will be
>>> available only in 2.2. But other than that - it is backwards compatible -
>>> so you could install it on Airflow 2.1 and have the "classic" operator
>>> work. But if you try to use the decorator, you will be able to write a Dag
>>> with this decorator, but it simply won't work (and you will have no message
>>> about it).
>>>
>>> The Docker Provider has no other changes nor fixes.
>>>
>>> I am tempted (and quite convinced) to increase the min airflow version
>>> of the Docker Provider to 2.2 so that it can be only installed there (and
>>> we make it ready for the 2.2 release).
>>>
>>> Does anyone have a strong opinion on that ?
>>>
>>> J.
>>>
>>>
>>>
>>>
Re: Docker Provider min version ?
Posted by Jarek Potiuk <ja...@potiuk.com>.
>
>
> I have no strong opinion on this, but would like to have clarification on
> one question: I understand you highlighted there is no changes nor fixes in
> the provider, but what if in the future we need to fix something for the
> “classic” operator which may still be used in Airflow 2.1 or even lower
> version installations? How will releases etc. be managed in such cases?
>
Yep. Good question. This is about the only "potential" issue - currently we
have no process for that as we've never had to do it. However it is always
possible to do it if and when we need it. If we find that there is a
critical fix for Docker Provider that we would need to fix, we can always
branch-off from the last released version that works for 2.1 and
cherry-pick and release a patched version.
We did not see a need for that so far. We released all providers when we
released 2.1.0 which are 2.1+ only (because of apply_defaults) and so far
there was not a case where anyone raised a need to release any bugfix for
previous versions. With our SemVer approach anyone should be "SAFE"
migrating to newer version of Airflow (even 2.2) as they are all supposed
to be backwards compatible, so I think our approach should be "If you need
anything that we release for the DockerProvider in the future - migrate to
2.2. But this is a "default" approach I think and I see the case of a
really critical issue that will result in releasing a patchlevel version
for the previous release.
Again - we have not done that yet, and we want to avoid it for as long as
we can, but there is nothing (except the overhead on cherry-picking and
releasing it) to prevent us from doing so. We have all history, tags,
changelogs and mechanisms to selectively release only one provider if
needed. But I think it should be really Ad-Hoc and when we REALLY REALLY
need it.
J.
I may have misunderstood or missed something, and be asking a dumb
> question. Please correct me in that case.
>
> Thanks!
>
>
> XD
>
> On Wed, Sep 29, 2021 at 22:15 Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> Hello Everyone,
>>
>> I am just about to release the September Providers, but Kamil raised a
>> valid concern and he convinced me that it might be a problem if we release
>> a new Docker Provider with minimum airflow version 2.1.
>>
>> The new provider has the new "Docker Decorator" feature which will be
>> available only in 2.2. But other than that - it is backwards compatible -
>> so you could install it on Airflow 2.1 and have the "classic" operator
>> work. But if you try to use the decorator, you will be able to write a Dag
>> with this decorator, but it simply won't work (and you will have no message
>> about it).
>>
>> The Docker Provider has no other changes nor fixes.
>>
>> I am tempted (and quite convinced) to increase the min airflow version of
>> the Docker Provider to 2.2 so that it can be only installed there (and we
>> make it ready for the 2.2 release).
>>
>> Does anyone have a strong opinion on that ?
>>
>> J.
>>
>>
>>
>>
Re: Docker Provider min version ?
Posted by Xiaodong Deng <xd...@apache.org>.
Hi Jarek,
I have no strong opinion on this, but would like to have clarification on
one question: I understand you highlighted there is no changes nor fixes in
the provider, but what if in the future we need to fix something for the
“classic” operator which may still be used in Airflow 2.1 or even lower
version installations? How will releases etc. be managed in such cases?
I may have misunderstood or missed something, and be asking a dumb
question. Please correct me in that case.
Thanks!
XD
On Wed, Sep 29, 2021 at 22:15 Jarek Potiuk <ja...@potiuk.com> wrote:
> Hello Everyone,
>
> I am just about to release the September Providers, but Kamil raised a
> valid concern and he convinced me that it might be a problem if we release
> a new Docker Provider with minimum airflow version 2.1.
>
> The new provider has the new "Docker Decorator" feature which will be
> available only in 2.2. But other than that - it is backwards compatible -
> so you could install it on Airflow 2.1 and have the "classic" operator
> work. But if you try to use the decorator, you will be able to write a Dag
> with this decorator, but it simply won't work (and you will have no message
> about it).
>
> The Docker Provider has no other changes nor fixes.
>
> I am tempted (and quite convinced) to increase the min airflow version of
> the Docker Provider to 2.2 so that it can be only installed there (and we
> make it ready for the 2.2 release).
>
> Does anyone have a strong opinion on that ?
>
> J.
>
>
>
>