You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "o-nikolas (via GitHub)" <gi...@apache.org> on 2025/03/04 00:03:27 UTC

[PR] Remove core Airflow support for static hybrid executors [airflow]

o-nikolas opened a new pull request, #47322:
URL: https://github.com/apache/airflow/pull/47322

   Remove all the handholding and custom logic we have in core airflow which allows the use of static hybrid executors like LocalKubernetesExecutor and CeleryKubernetesExecutor. These executors will still work on 2.X versions of Airflow, but moving forward they will not be supported on Airflow 3
   
   <!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
   
      http://www.apache.org/licenses/LICENSE-2.0
   
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
    -->
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   
   
   <!-- Please keep an empty line above the dashes. -->
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove core Airflow support for static hybrid executors [airflow]

Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #47322:
URL: https://github.com/apache/airflow/pull/47322#discussion_r1978488825


##########
airflow/settings.py:
##########
@@ -687,11 +687,7 @@ def initialize():
 LAZY_LOAD_PROVIDERS: bool = conf.getboolean("core", "lazy_discover_providers", fallback=True)
 
 # Determines if the executor utilizes Kubernetes
-IS_K8S_OR_K8SCELERY_EXECUTOR = conf.get("core", "EXECUTOR") in {
-    executor_constants.KUBERNETES_EXECUTOR,
-    executor_constants.CELERY_KUBERNETES_EXECUTOR,
-    executor_constants.LOCAL_KUBERNETES_EXECUTOR,
-}
+IS_K8S_OR_K8SCELERY_EXECUTOR = conf.get("core", "EXECUTOR") == executor_constants.KUBERNETES_EXECUTOR

Review Comment:
   This variable name probably needs some work. Also probably means this should be a flag on the interface? Not sure what it actually controls on the ui though.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove core Airflow support for static hybrid executors [airflow]

Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #47322:
URL: https://github.com/apache/airflow/pull/47322#discussion_r1982280003


##########
tests/utils/test_log_handlers.py:
##########
@@ -209,8 +207,6 @@ def task_callable(ti):
         {
             ("core", "EXECUTOR"): ",".join(
                 [
-                    executor_constants.LOCAL_KUBERNETES_EXECUTOR,
-                    executor_constants.CELERY_KUBERNETES_EXECUTOR,
                     executor_constants.KUBERNETES_EXECUTOR,
                 ]
             ),

Review Comment:
   Good call.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove core Airflow support for static hybrid executors [airflow]

Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #47322:
URL: https://github.com/apache/airflow/pull/47322#discussion_r1982235120


##########
tests/utils/test_log_handlers.py:
##########
@@ -209,8 +207,6 @@ def task_callable(ti):
         {
             ("core", "EXECUTOR"): ",".join(
                 [
-                    executor_constants.LOCAL_KUBERNETES_EXECUTOR,
-                    executor_constants.CELERY_KUBERNETES_EXECUTOR,
                     executor_constants.KUBERNETES_EXECUTOR,
                 ]
             ),

Review Comment:
   Probably this can be reduced to a single line as we do not need to "join" a single list element array.



##########
airflow/settings.py:
##########
@@ -687,11 +687,7 @@ def initialize():
 LAZY_LOAD_PROVIDERS: bool = conf.getboolean("core", "lazy_discover_providers", fallback=True)
 
 # Determines if the executor utilizes Kubernetes
-IS_K8S_OR_K8SCELERY_EXECUTOR = conf.get("core", "EXECUTOR") in {
-    executor_constants.KUBERNETES_EXECUTOR,
-    executor_constants.CELERY_KUBERNETES_EXECUTOR,
-    executor_constants.LOCAL_KUBERNETES_EXECUTOR,
-}
+IS_K8S_OR_K8SCELERY_EXECUTOR = conf.get("core", "EXECUTOR") == executor_constants.KUBERNETES_EXECUTOR

Review Comment:
   First time that I see this variable - wondering if this would work with multiple executors today... so if you use K8sExecutor and Celery together... then you do not equal. But anyway probably not relevant for this PR...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove core Airflow support for static hybrid executors [airflow]

Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #47322:
URL: https://github.com/apache/airflow/pull/47322#discussion_r1982350812


##########
tests/utils/test_log_handlers.py:
##########
@@ -209,8 +207,6 @@ def task_callable(ti):
         {
             ("core", "EXECUTOR"): ",".join(
                 [
-                    executor_constants.LOCAL_KUBERNETES_EXECUTOR,
-                    executor_constants.CELERY_KUBERNETES_EXECUTOR,
                     executor_constants.KUBERNETES_EXECUTOR,
                 ]
             ),

Review Comment:
   Done! Thanks for the review :smiley:



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove core Airflow support for static hybrid executors [airflow]

Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #47322:
URL: https://github.com/apache/airflow/pull/47322#discussion_r1980173188


##########
airflow/settings.py:
##########
@@ -687,11 +687,7 @@ def initialize():
 LAZY_LOAD_PROVIDERS: bool = conf.getboolean("core", "lazy_discover_providers", fallback=True)
 
 # Determines if the executor utilizes Kubernetes
-IS_K8S_OR_K8SCELERY_EXECUTOR = conf.get("core", "EXECUTOR") in {
-    executor_constants.KUBERNETES_EXECUTOR,
-    executor_constants.CELERY_KUBERNETES_EXECUTOR,
-    executor_constants.LOCAL_KUBERNETES_EXECUTOR,
-}
+IS_K8S_OR_K8SCELERY_EXECUTOR = conf.get("core", "EXECUTOR") == executor_constants.KUBERNETES_EXECUTOR

Review Comment:
   Thanks for the review @jedcunningham!
   
   I decided to stop the scope of the change here, I'll explain why, but it's a fair call out.
   
   I can update the name of the variable at least. But whether or not it requires an interface change I'd like to push back on. That was true even before these changes, all the way back to AIP-51 (here is [the issue](https://github.com/apache/airflow/issues/27933) covering this coupling) and I don't think it should be contingent on this PR being merged.
   
   It takes a lot of code to support it in the interface. We'd have to build a mechanism for executors to vend UI pages, and that's a lot of logic in a space where there's been lots of changes. K8s is the only executor that has a UI page. The solution I proposed the last couple times this has come up is to actually just remove this code altogether since I think that's the easiest path (and we love removing code for Airflow 3.0 :smiley: ). And I'm not sure how many folks really even make use of the UI endpoint to render the config file.
   
   I'd love to hear your thoughts on that. If it sounds agreeable I'll open a separate 3.0 PR for that!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove core Airflow support for static hybrid executors [airflow]

Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #47322:
URL: https://github.com/apache/airflow/pull/47322#discussion_r1980185328


##########
airflow/settings.py:
##########
@@ -687,11 +687,7 @@ def initialize():
 LAZY_LOAD_PROVIDERS: bool = conf.getboolean("core", "lazy_discover_providers", fallback=True)
 
 # Determines if the executor utilizes Kubernetes
-IS_K8S_OR_K8SCELERY_EXECUTOR = conf.get("core", "EXECUTOR") in {
-    executor_constants.KUBERNETES_EXECUTOR,
-    executor_constants.CELERY_KUBERNETES_EXECUTOR,
-    executor_constants.LOCAL_KUBERNETES_EXECUTOR,
-}
+IS_K8S_OR_K8SCELERY_EXECUTOR = conf.get("core", "EXECUTOR") == executor_constants.KUBERNETES_EXECUTOR

Review Comment:
   I'd be for removal. I always thought it was a big weird. If folks do miss it and want it back, it can be done with less duct tape at that point.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove core Airflow support for static hybrid executors [airflow]

Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham merged PR #47322:
URL: https://github.com/apache/airflow/pull/47322


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove core Airflow support for static hybrid executors [airflow]

Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #47322:
URL: https://github.com/apache/airflow/pull/47322#discussion_r1982298710


##########
tests/utils/test_log_handlers.py:
##########
@@ -209,8 +207,6 @@ def task_callable(ti):
         {
             ("core", "EXECUTOR"): ",".join(
                 [
-                    executor_constants.LOCAL_KUBERNETES_EXECUTOR,
-                    executor_constants.CELERY_KUBERNETES_EXECUTOR,
                     executor_constants.KUBERNETES_EXECUTOR,
                 ]
             ),

Review Comment:
   Sure, I can make that change :) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove core Airflow support for static hybrid executors [airflow]

Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #47322:
URL: https://github.com/apache/airflow/pull/47322#discussion_r1980202193


##########
airflow/settings.py:
##########
@@ -687,11 +687,7 @@ def initialize():
 LAZY_LOAD_PROVIDERS: bool = conf.getboolean("core", "lazy_discover_providers", fallback=True)
 
 # Determines if the executor utilizes Kubernetes
-IS_K8S_OR_K8SCELERY_EXECUTOR = conf.get("core", "EXECUTOR") in {
-    executor_constants.KUBERNETES_EXECUTOR,
-    executor_constants.CELERY_KUBERNETES_EXECUTOR,
-    executor_constants.LOCAL_KUBERNETES_EXECUTOR,
-}
+IS_K8S_OR_K8SCELERY_EXECUTOR = conf.get("core", "EXECUTOR") == executor_constants.KUBERNETES_EXECUTOR

Review Comment:
   Cool, sounds great, I'll put up a PR soon!
   
   And given that's the case. I'll just leave this variable name as is here to reduce the churn.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove core Airflow support for static hybrid executors [airflow]

Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on PR #47322:
URL: https://github.com/apache/airflow/pull/47322#issuecomment-2702141821

   Does this look okay to merge @jedcunningham?
   
   Also CC @jscheffl and @amoghrajesh who reviewed the other hybrid executor AF3.0 changes, you may be interested in this one as well. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org