You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dstandish (via GitHub)" <gi...@apache.org> on 2024/08/30 20:06:38 UTC

[PR] Backcompat for non inheriting executors [airflow]

dstandish opened a new pull request, #41906:
URL: https://github.com/apache/airflow/pull/41906

   Some executors don't inherit from BaseExecutor.  2.10 change assumes they do.
   
   This fixes that.
   
   Resolves https://github.com/apache/airflow/issues/41891


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Backcompat for non inheriting executors [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #41906: Backcompat for non inheriting executors
URL: https://github.com/apache/airflow/pull/41906


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Backcompat for non inheriting executors [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on PR #41906:
URL: https://github.com/apache/airflow/pull/41906#issuecomment-2322289223

   @dstandish Does this not affect main too?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Backcompat for non inheriting executors [airflow]

Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on PR #41906:
URL: https://github.com/apache/airflow/pull/41906#issuecomment-2322262721

   sorry -- had wrong base; fixed now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Backcompat for non inheriting executors [airflow]

Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on PR #41906:
URL: https://github.com/apache/airflow/pull/41906#issuecomment-2322303802

   > or we could remove support for the "old hybrid" executors in Airlfow 3
   
   yeah i feel removing the hybrid executors for airflow 3 is an obvious choice.  not sure exactly how that is enforced since it's in providers...
   
   probably we should just remove them as soon as min airflow version for the provider has airflow version that has hybrid execution.  when was that released, 2.10?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Backcompat for non inheriting executors [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on PR #41906:
URL: https://github.com/apache/airflow/pull/41906#issuecomment-2322319906

   >My answer: theorethically it does, if somoene uses currently released cncf.kubernetes provider and CeleryKubernetesExecutor or LocalKubernetesExecutor in Airflow 3.
   
   If it affects "main" currently, then yeah I think we should fix and merge in main too. And then cherry-pick / separate PR (this can be that PR and you can create one for main too)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Backcompat for non inheriting executors [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #41906:
URL: https://github.com/apache/airflow/pull/41906#issuecomment-2322334735

   >anyway, no strong feeling on it. just ultimately do want this backcompat in 3.0 -- somehow or another.
   
   Yep. then I think we should add it to main and back-port even if it is "such a small change" - I guess in this case "let's do all things the same way" trumps "but this one is so small and likely to be not needed".


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Backcompat for non inheriting executors [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #41906:
URL: https://github.com/apache/airflow/pull/41906#issuecomment-2323071677

   Would be great to get that one for 2.10.1 @ephraimbuddy @utkarsharma2 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Backcompat for non inheriting executors [airflow]

Posted by "utkarsharma2 (via GitHub)" <gi...@apache.org>.
utkarsharma2 commented on PR #41906:
URL: https://github.com/apache/airflow/pull/41906#issuecomment-2323191430

   > Would be great to get that one for 2.10.1 @ephraimbuddy @utkarsharma2
   
   Sure @potiuk I'll include this for 2.10.1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Backcompat for non inheriting executors [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #41906:
URL: https://github.com/apache/airflow/pull/41906#issuecomment-2322279507

   cc: @utkarsharma2 @ephraimbuddy @kaxil @eladkal  -> WDYT? ^^


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Backcompat for non inheriting executors [airflow]

Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish closed pull request #41906: Backcompat for non inheriting executors
URL: https://github.com/apache/airflow/pull/41906


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Backcompat for non inheriting executors [airflow]

Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on PR #41906:
URL: https://github.com/apache/airflow/pull/41906#issuecomment-2322280551

   > I think we have the rule that we generally implement such fixes in main and backport them to v2-*test - this is the first exception so far. Not sure if justified.
   
   I can't speak to that necessarily but this is the type of backcompat thing that I'd want to chop in 3.0 anyway.  If you think it's better, we could add to main, then remove it with changelog.  But it's _such_ a small thing!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Backcompat for non inheriting executors [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #41906:
URL: https://github.com/apache/airflow/pull/41906#issuecomment-2323262189

   Something strange happend when building images -> closing/reopening to rebuild


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Backcompat for non inheriting executors [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on PR #41906:
URL: https://github.com/apache/airflow/pull/41906#issuecomment-2322361028

   >I think individual's convenience in this case is less important that "overall consistency" - and we have to pay the overhead price for it. We already do that by having to run Pull requests for all backports rather than individual cherry-picks as it was in previous v2-* which means that the committers authoring/merging changes have more work to do at the expense of less work by the release manager. I guess that's the sacrifice we all should make now.
   
   Agreed. At the point of the PR being open, if it affects main too, we should ideally fix it in main even if a future change will remove it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Backcompat for non inheriting executors [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #41906:
URL: https://github.com/apache/airflow/pull/41906#issuecomment-2322341101

   I think individual's convenience in this case is less important that "overall consistency" - and we have to pay the overhead price for it. We already do that by having to run Pull requests for all backports rather than individual cherry-picks as it was in previous v2-10* which means that the committers authoring/merging changes have more work to do at the expense of less work by the release manager. I guess that's the sacrifice we all should make now.
   
   And I am not sarcastic - I rigorously follow those rules, it's just the price to pay and we should all be aware why we are paying it and we should share the burden.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Backcompat for non inheriting executors [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #41906:
URL: https://github.com/apache/airflow/pull/41906#issuecomment-2322286436

   > > I think we have the rule that we generally implement such fixes in main and backport them to v2-*test - this is the first exception so far. Not sure if justified.
   > 
   > I can't speak to that necessarily but this is the type of backcompat thing that I'd want to chop in 3.0 anyway. If you think it's better, we could add to main, then remove it with changelog. But it's _such_ a small thing!
   
   We discussed it before and that's the first one that is not cherry-picked. I am raising it - beause It might have some side-effects that might impact relase manager's work. For example there is the tooling that check milestone vs. cherry-picked tasks and it might simply give wrong misleading results. I raised this question before as "academic" question - and we discussed at the last dev call that we will discuss it when we see the first case of it. Which is .... now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Backcompat for non inheriting executors [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #41906:
URL: https://github.com/apache/airflow/pull/41906#issuecomment-2323073018

   (Still - this one needs main + backport as discussed before).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Backcompat for non inheriting executors [airflow]

Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on PR #41906:
URL: https://github.com/apache/airflow/pull/41906#issuecomment-2322326705

   > If it affects "main" currently, then yeah I think we should fix and merge in main too. And then cherry-pick / separate PR (this can be that PR and you can create one for main too)
   
   I guess jarek is saying that "theoretically" it affects main but practicall speaking we will drop support for hybrid executors / executors-that-don't-inherit-from-base-executor in 3.0 and / or somehow indicate that providers are not assumed to be future compat with 3.0.
   
   anyway, no strong feeling on it.  just ultimately do want this backcompat in 3.0 -- somehow or another.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Backcompat for non inheriting executors [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #41906:
URL: https://github.com/apache/airflow/pull/41906#issuecomment-2322295588

   > @dstandish Does this not affect main too?
   
   My answer: theorethically it does, if somoene uses currently released cncf.kubernetes provider and CeleryKubernetesExecutor or LocalKubernetesExecutor in Airflow 3.
   
   Now - we do not know yet which min-versions of providers will be usable for Airflow 3 (we could probably cut-off providers to support Airlfow 3 - or we could remove support for the "old hybrid" executors in Airlfow 3 - in which case that bug does not affect Airflow 3. I think it's quite likely one of these will happen - so likely yes, main (aka. Airlfow 3) is not affected.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Backcompat for non inheriting executors [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #41906:
URL: https://github.com/apache/airflow/pull/41906#issuecomment-2322278340

   I think we have the rule that we generally implement such fixes in main and backport them to v2-*test - this is the first exception so far. Not sure if justified. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Backcompat for non inheriting executors [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #41906:
URL: https://github.com/apache/airflow/pull/41906


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org