You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/12/06 21:27:54 UTC
[GitHub] [airflow] subkanthi opened a new pull request #19729: Local kubernetes executor
subkanthi opened a new pull request #19729:
URL: https://github.com/apache/airflow/pull/19729
LocalKubernetesExecutor similar to CeleryKubernetesExecutor.
Tested by running webserver, scheduler and connecting to minikube k8s cluster.
Note: This PR does not update the helm charts.
closes: #16185
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi closed pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi closed pull request #19729:
URL: https://github.com/apache/airflow/pull/19729
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-980207459
I think it woudl be good to continue with it "as is" without the explicit spelling out the executor but using queue - but then it could be separate PR where we could introduce the other cases.
I think we will find out that it might be quite an effort to complete the latter, and this one has much higher chance to be merged and available earlier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-1042916898
One comment - before merge (and @subkanthi - maybe you can take a closer look while you are adding it).
There is a but with CeleryKubernetesExecutor reported recently https://github.com/apache/airflow/issues/21548 which might also affect LocalKubernetesExecutor. Would it be possible that you take a look @subkanthi - and maybe you will be able to fix it as general case ? And maybe also take a look if there are other places where we have "hard-coded" Executor names
This was one thing that I was afraid of when we discussed "Generic Combined Excecutor" - seems that there might be at least this one place pointed out in #21548 where we implemented logic based on the type of executor configured and about the executor actually 'used" when the task was executed.
I think that LocalKubernetesExecutor might suffer the same problem actually - because `file_task_handler` was not touched by this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-1059835837
I think we should be good to go for that one then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] BasPH commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
BasPH commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-980254740
@potiuk @subkanthi Would proceed with this PR. Supporting multiple executors can be a new PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-980171954
> > To avoid this situation, what do you think of configuring multiple executors, and have an argument on the BaseOperator to select your executor of choice? The selection of executor with these "multi"-executors via the `queue` argument feels somewhat implicit to me + you avoid ending up with X+Y+Z-executors in the future.
>
> Yep. Might be a good idea to do it explicitly.
@potiuk , Thoughts, does it make sense to continue with this, I wanted to add Helm charts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on a change in pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi commented on a change in pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#discussion_r757820287
##########
File path: airflow/example_dags/example_local_kubernetes_executor.py
##########
@@ -0,0 +1,71 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This is an example dag for using a Kubernetes Executor Configuration.
+"""
+import logging
+from datetime import datetime
+
+from airflow import DAG
+from airflow.configuration import conf
+from airflow.example_dags.libs.helper import print_stuff
+from airflow.operators.python_operator import PythonOperator
+
+log = logging.getLogger(__name__)
+
+worker_container_repository = conf.get('kubernetes', 'worker_container_repository')
+worker_container_tag = conf.get('kubernetes', 'worker_container_tag')
+
+try:
+ from kubernetes.client import models as k8s
+except ImportError:
+ log.warning("Could not import DAGs in example_kubernetes_executor.py", exc_info=True)
+ log.warning("Install Kubernetes dependencies with: pip install apache-airflow[cncf.kubernetes]")
+
+with DAG(
+ dag_id='example_local_kubernetes_executor',
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+ tags=['example3'],
+) as dag:
+ # You can use annotations on your kubernetes pods!
+ start_task_executor_config = {
+ "pod_override": k8s.V1Pod(metadata=k8s.V1ObjectMeta(annotations={"test": "annotation"}))
+ }
+
+ task_with_template = PythonOperator(
+ task_id="task_with_kubernetes_executor",
+ python_callable=print_stuff,
+ pool='kubernetes',
Review comment:
It's the queue , will change it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] BasPH commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
BasPH commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-979216773
> I'm a bit worried about the possible combinations that we run in to if we add this. Is it maybe time to resurrect your multi-executor @potiuk?
To avoid this situation, what do you think of configuring multiple executors, and have an argument on the BaseOperator to select your executor of choice? The selection of executor with these "multi"-executors via the `queue` argument feels somewhat implicit to me + you avoid ending up with X+Y+Z-executors in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-975562022
I'm a bit worried about the possible combinations that we run in to if we add this. Is it maybe time to resurrect your multi-executor @potiuk?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#discussion_r765019302
##########
File path: airflow/executors/local_kubernetes_executor.py
##########
@@ -0,0 +1,204 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Dict, List, Optional, Set, Union
+
+from airflow.configuration import conf
+from airflow.executors.base_executor import CommandType, EventBufferValueType, QueuedTaskInstanceType
+from airflow.executors.kubernetes_executor import KubernetesExecutor
+from airflow.executors.local_executor import LocalExecutor
+from airflow.models.taskinstance import SimpleTaskInstance, TaskInstance, TaskInstanceKey
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class LocalKubernetesExecutor(LoggingMixin):
+ """
+ LocalKubernetesExecutor consists of LocalExecutor and KubernetesExecutor.
+ It chooses an executor to use based on the queue defined on the task.
+ When the queue is the value of ``kubernetes_queue`` in section ``[local_kubernetes_executor]``
+ of the configuration (default value: `kubernetes`), KubernetesExecutor is selected to run the task,
+ otherwise, LocalExecutor is used.
+ """
+
+ supports_ad_hoc_ti_run: bool = True
Review comment:
Yeah
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-1045231490
> Waiting for response from @subkanthi on the potential bug.
Im actually stuck with another issue, Im pretty sure it used to work before, For some reason the k8s_object is str and it ends up in the else block
```
k8s_object = obj.get("pod_override", None)
if k8s_legacy_object and k8s_object:
raise AirflowConfigException(
"Can not have both a legacy and new"
"executor_config object. Please delete the KubernetesExecutor"
"dict and only use the pod_override kubernetes.client.models.V1Pod"
"object."
)
if not k8s_object and not k8s_legacy_object:
return None
print("TYPE" + str(type(k8s_object)))
if isinstance(k8s_object, k8s.V1Pod):
return k8s_object
elif isinstance(k8s_legacy_object, dict):
warnings.warn(
'Using a dictionary for the executor_config is deprecated and will soon be removed.'
'please use a `kubernetes.client.models.V1Pod` class with a "pod_override" key'
' instead. ',
category=DeprecationWarning,
)
return PodGenerator.from_legacy_obj(obj)
else:
raise TypeError(
'Cannot convert a non-kubernetes.client.models.V1Pod object into a KubernetesExecutorConfig'
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-988351445
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-982028264
> Details
Thanks @potiuk , not sure where this document should be linked , the docs Kaxil mentioned, are they in a different repo?
/opt/airflow/docs/apache-airflow/executor/local_kubernetes.rst: WARNING: document isn't included in any toctree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-982043258
Nope. It should be included .. somewhere.. In any of the TOC trees. The way it is now is that it is not mentioned anywhere so it's basically not reachable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on a change in pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
josh-fell commented on a change in pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#discussion_r755219471
##########
File path: airflow/example_dags/example_local_kubernetes_executor.py
##########
@@ -0,0 +1,71 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This is an example dag for using a Kubernetes Executor Configuration.
+"""
+import logging
+from datetime import datetime
+
+from airflow import DAG
+from airflow.configuration import conf
+from airflow.example_dags.libs.helper import print_stuff
+from airflow.operators.python_operator import PythonOperator
+
+log = logging.getLogger(__name__)
+
+worker_container_repository = conf.get('kubernetes', 'worker_container_repository')
+worker_container_tag = conf.get('kubernetes', 'worker_container_tag')
+
+try:
+ from kubernetes.client import models as k8s
+except ImportError:
+ log.warning("Could not import DAGs in example_kubernetes_executor.py", exc_info=True)
+ log.warning("Install Kubernetes dependencies with: pip install apache-airflow[cncf.kubernetes]")
+
+with DAG(
+ dag_id='example_local_kubernetes_executor',
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+ tags=['example3'],
+) as dag:
+ # You can use annotations on your kubernetes pods!
+ start_task_executor_config = {
+ "pod_override": k8s.V1Pod(metadata=k8s.V1ObjectMeta(annotations={"test": "annotation"}))
+ }
+
+ task_with_template = PythonOperator(
+ task_id="task_with_kubernetes_executor",
+ python_callable=print_stuff,
+ pool='kubernetes',
+ queue='kubernetes',
+ executor_config=start_task_executor_config,
+ )
+
+ def print_context(ds, **kwargs):
+ """Print the Airflow context and ds variable from the context."""
+ print(kwargs)
+ print(ds)
+ return 'Whatever you return gets printed in the logs'
+
+ task_with_local = PythonOperator(
+ task_id="task_with_local_executor",
+ python_callable=print_context,
+ )
Review comment:
Let's use the TaskFlow API in the example DAGs rather than the `PythonOperator`. There was a substantial effort to update the example DAGs in this manner and we should try to keep this consistency.
The existing "example_kubernetes_executor" example DAG has some good pointers for how the TaskFlow API was used in a similar context.
##########
File path: docs/apache-airflow/executor/local_kubernetes.rst
##########
@@ -0,0 +1,30 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+.. _executor:LocalKubernetesExecutor:
+
+LocalKubernetes Executor
+=========================
+
+The :class:`~airflow.executors.local_kubernetes_executor.LocalKubernetesExecutor` allows users
+to run simultaneously a ``LocalExecutor`` and a ``KubernetesExecutor``.
Review comment:
```suggestion
to simultaneously run a ``LocalExecutor`` and a ``KubernetesExecutor``.
```
##########
File path: tests/executors/test_local_kubernetes_executor.py
##########
@@ -0,0 +1,64 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from unittest import mock
+
+from airflow.configuration import conf
+from airflow.executors.local_executor import LocalExecutor
+from airflow.executors.local_kubernetes_executor import LocalKubernetesExecutor
+
+KUBERNETES_QUEUE = LocalKubernetesExecutor.KUBERNETES_QUEUE
Review comment:
How is this variable being used?
##########
File path: docs/apache-airflow/executor/local_kubernetes.rst
##########
@@ -0,0 +1,30 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+.. _executor:LocalKubernetesExecutor:
+
+LocalKubernetes Executor
+=========================
+
+The :class:`~airflow.executors.local_kubernetes_executor.LocalKubernetesExecutor` allows users
+to run simultaneously a ``LocalExecutor`` and a ``KubernetesExecutor``.
+An executor is chosen to run a task based on the task's queue.
+
+``LocalKubernetesExecutor`` provides the capability of running low intesive tasks locally
Review comment:
```suggestion
``LocalKubernetesExecutor`` provides the capability of running low-intensive tasks locally
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-1065929107
I can work on this in a week from now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-1053653812
> #21548
@potiuk
Im not sure how to reproduce the bug with this line
` elif conf.get('core', 'executor') == 'KubernetesExecutor': `
I cant find the workflow or configuration where `FileTaskHandler` is used directly, I tried elasticsearch remote logging and with the `_read` function overridden, it doesnt call the `_read` function in `FileTaskHandler`.
It could be either cloudwatch or s3, but I see `_read` function overridden in both those task handlers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-1042548112
> This lookks good. Any other approval?
Can this be merged, it's all green now, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-975608905
> I'm a bit worried about the possible combinations that we run in to if we add this. Is it maybe time to resurrect your multi-executor @potiuk?
I think the benefits of any common code for "multi" executor are not as big - I still think having a few dedicated, specialized executors that you can use straight away and they are well tested and supported by Helm Chart outweitght the "generic" solution.
There are really few combinations that makes sense here - like you woudn't want to support Sequential + Kubernetes or Local + Celery, I think the only "reasonable" combination is `<x>Kubernetes` and we basicallly exhaust all of the combinations already with this one.
The fact that nobody proposed it for about a year since we had CeleryKubernetes, makes it likely that we have about a year again when someoene wants another combination (if at all). I think it's not worth to do generic one in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on a change in pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
josh-fell commented on a change in pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#discussion_r755403897
##########
File path: airflow/example_dags/example_local_kubernetes_executor.py
##########
@@ -0,0 +1,71 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This is an example dag for using a Kubernetes Executor Configuration.
+"""
+import logging
+from datetime import datetime
+
+from airflow import DAG
+from airflow.configuration import conf
+from airflow.example_dags.libs.helper import print_stuff
+from airflow.operators.python_operator import PythonOperator
+
+log = logging.getLogger(__name__)
+
+worker_container_repository = conf.get('kubernetes', 'worker_container_repository')
+worker_container_tag = conf.get('kubernetes', 'worker_container_tag')
+
+try:
+ from kubernetes.client import models as k8s
+except ImportError:
+ log.warning("Could not import DAGs in example_kubernetes_executor.py", exc_info=True)
+ log.warning("Install Kubernetes dependencies with: pip install apache-airflow[cncf.kubernetes]")
+
+with DAG(
+ dag_id='example_local_kubernetes_executor',
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+ tags=['example3'],
+) as dag:
+ # You can use annotations on your kubernetes pods!
+ start_task_executor_config = {
+ "pod_override": k8s.V1Pod(metadata=k8s.V1ObjectMeta(annotations={"test": "annotation"}))
+ }
+
+ task_with_template = PythonOperator(
+ task_id="task_with_kubernetes_executor",
+ python_callable=print_stuff,
+ pool='kubernetes',
+ queue='kubernetes',
+ executor_config=start_task_executor_config,
+ )
+
+ def print_context(ds, **kwargs):
+ """Print the Airflow context and ds variable from the context."""
+ print(kwargs)
+ print(ds)
+ return 'Whatever you return gets printed in the logs'
+
+ task_with_local = PythonOperator(
+ task_id="task_with_local_executor",
+ python_callable=print_context,
+ )
Review comment:
You should be able to pass these types of `BaseOperator` args directly to the decorator like `@task(task_id=..., pool=..., queue=..., executor_config=...)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi edited a comment on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi edited a comment on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-980171954
> > To avoid this situation, what do you think of configuring multiple executors, and have an argument on the BaseOperator to select your executor of choice? The selection of executor with these "multi"-executors via the `queue` argument feels somewhat implicit to me + you avoid ending up with X+Y+Z-executors in the future.
>
> Yep. Might be a good idea to do it explicitly.
@potiuk , Thoughts, does it make sense to continue with this, I wanted to add Helm charts. or if the direction is to work on the multiple executors(I could work there too), Please let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-980208283
@BasPH - WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on a change in pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi commented on a change in pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#discussion_r758836136
##########
File path: airflow/example_dags/example_local_kubernetes_executor.py
##########
@@ -0,0 +1,71 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This is an example dag for using a Kubernetes Executor Configuration.
+"""
+import logging
+from datetime import datetime
+
+from airflow import DAG
+from airflow.configuration import conf
+from airflow.example_dags.libs.helper import print_stuff
+from airflow.operators.python_operator import PythonOperator
+
+log = logging.getLogger(__name__)
+
+worker_container_repository = conf.get('kubernetes', 'worker_container_repository')
+worker_container_tag = conf.get('kubernetes', 'worker_container_tag')
+
+try:
+ from kubernetes.client import models as k8s
+except ImportError:
+ log.warning("Could not import DAGs in example_kubernetes_executor.py", exc_info=True)
+ log.warning("Install Kubernetes dependencies with: pip install apache-airflow[cncf.kubernetes]")
+
+with DAG(
+ dag_id='example_local_kubernetes_executor',
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+ tags=['example3'],
+) as dag:
+ # You can use annotations on your kubernetes pods!
+ start_task_executor_config = {
+ "pod_override": k8s.V1Pod(metadata=k8s.V1ObjectMeta(annotations={"test": "annotation"}))
+ }
+
+ task_with_template = PythonOperator(
+ task_id="task_with_kubernetes_executor",
+ python_callable=print_stuff,
+ pool='kubernetes',
Review comment:
Removed pool.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on a change in pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi commented on a change in pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#discussion_r759656073
##########
File path: airflow/example_dags/example_local_kubernetes_executor.py
##########
@@ -0,0 +1,70 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This is an example dag for using a Local Kubernetes Executor Configuration.
+"""
+import logging
+from datetime import datetime
+
+from airflow import DAG
+from airflow.configuration import conf
+from airflow.decorators import task
+from airflow.example_dags.libs.helper import print_stuff
+
+log = logging.getLogger(__name__)
+
+worker_container_repository = conf.get('kubernetes', 'worker_container_repository')
+worker_container_tag = conf.get('kubernetes', 'worker_container_tag')
+
+try:
+ from kubernetes.client import models as k8s
+except ImportError:
+ log.warning("Could not import DAGs in example_local_kubernetes_executor.py", exc_info=True)
+ log.warning("Install Kubernetes dependencies with: pip install apache-airflow[cncf.kubernetes]")
+
+with DAG(
+ dag_id='example_local_kubernetes_executor',
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+ tags=['example3'],
+) as dag:
+ # You can use annotations on your kubernetes pods!
+ start_task_executor_config = {
+ "pod_override": k8s.V1Pod(metadata=k8s.V1ObjectMeta(annotations={"test": "annotation"}))
+ }
+
+ @task(
+ executor_config='start_task_executor_config',
+ queue='kubernetes',
+ task_id='task_with_kubernetes_executor',
+ )
+ def task_with_template():
+ print_stuff()
+
+ def print_context(ds, **kwargs):
+ """Print the Airflow context and ds variable from the context."""
+ print(kwargs)
+ print(ds)
+ return 'Whatever you return gets printed in the logs'
+
+ @task(task_id='task_with_local_executor')
+ def task_with_local():
+ print_context()
+
+ task_with_local >> task_with_template
Review comment:
Thanks, made the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] arkadiusz-bach commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
arkadiusz-bach commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-1043030837
You will probably have the same issue as the one that I described here: https://github.com/apache/airflow/issues/21225#issuecomment-1035656930
It is causing a lot of pain on CeleryKubernetesExecutor, when for example tasks are waiting for celery to autoscale, because KubernetesExecutor is restarting all of the tasks that are in queued state to scheduled, not only those that has kubernetes qeueue assigned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-974879206
Nice one. I think LocalK8S is quite a useful executor to have :).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-979228822
> To avoid this situation, what do you think of configuring multiple executors, and have an argument on the BaseOperator to select your executor of choice? The selection of executor with these "multi"-executors via the `queue` argument feels somewhat implicit to me + you avoid ending up with X+Y+Z-executors in the future.
Yep. Might be a good idea to do it explicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-980215844
> Airflow 2.3.0
> I think it woudl be good to continue with it "as is" without the explicit spelling out the executors but using queue - but then it could be separate PR where we could introduce the other cases.
>
> I think we will find out that it might be quite an effort to complete the latter, and this one has much higher chance to be merged and available earlier.
Thanks @potiuk I understand the queue part, so if queue is "local" then the localExecutor would be used and if its "kubernetes" , then the kubernetesExecutor, but clarifying this part
**I think it woudl be good to continue with it "as is" without the explicit spelling out the executors**
Question is what executor will be defined in the configuration, MultipleExecutor?
Would config look like this
executor=MutipleExecutor
[multiple_executor]
local_queue=local
kubernetes_queue=kubernetes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a change in pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#discussion_r765018740
##########
File path: airflow/executors/local_kubernetes_executor.py
##########
@@ -0,0 +1,204 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Dict, List, Optional, Set, Union
+
+from airflow.configuration import conf
+from airflow.executors.base_executor import CommandType, EventBufferValueType, QueuedTaskInstanceType
+from airflow.executors.kubernetes_executor import KubernetesExecutor
+from airflow.executors.local_executor import LocalExecutor
+from airflow.models.taskinstance import SimpleTaskInstance, TaskInstance, TaskInstanceKey
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class LocalKubernetesExecutor(LoggingMixin):
+ """
+ LocalKubernetesExecutor consists of LocalExecutor and KubernetesExecutor.
+ It chooses an executor to use based on the queue defined on the task.
+ When the queue is the value of ``kubernetes_queue`` in section ``[local_kubernetes_executor]``
+ of the configuration (default value: `kubernetes`), KubernetesExecutor is selected to run the task,
+ otherwise, LocalExecutor is used.
+ """
+
+ supports_ad_hoc_ti_run: bool = True
Review comment:
> Yeah @dstandish I think running ad-hocs TIs from Webserver is semi-broken -- I would vote for removing or using "clearing" mechanism under the hood so a new instance of Executor does not need to be created.
OK so should @subkanthi just leave this as is for now and we deal with the webserver bit later?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #19729:
URL: https://github.com/apache/airflow/pull/19729
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-1030070450
> This one needs rebase!
Please let me know if there is anything else outstanding for this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-1052740393
> The problem seems to be that for some reason executor_config seems to get unpicked as str. Im trying to figure out the right combination on when it happens.
Let me know when to re-review it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-1019380955
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-980278629
Yes let's please do it in a separate PR - that is a much much bigger change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi edited a comment on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi edited a comment on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-980215844
> Airflow 2.3.0
> I think it woudl be good to continue with it "as is" without the explicit spelling out the executors but using queue - but then it could be separate PR where we could introduce the other cases.
>
> I think we will find out that it might be quite an effort to complete the latter, and this one has much higher chance to be merged and available earlier.
Thanks @potiuk I understand the queue part, so if queue is "local" then the localExecutor would be used and if its "kubernetes" , then the kubernetesExecutor, but clarifying this part
**I think it woudl be good to continue with it "as is" without the explicit spelling out the executors**
Question is what executor will be defined in the configuration, MultipleExecutor?
Would config look like this
executor=MutipleExecutor
[multiple_executor]
local_queue=local
kubernetes_queue=kubernetes
Also the question is where will the defaults be executed, if the queue is not provided, is default always CeleryExecutor?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#discussion_r765014800
##########
File path: airflow/executors/local_kubernetes_executor.py
##########
@@ -0,0 +1,204 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Dict, List, Optional, Set, Union
+
+from airflow.configuration import conf
+from airflow.executors.base_executor import CommandType, EventBufferValueType, QueuedTaskInstanceType
+from airflow.executors.kubernetes_executor import KubernetesExecutor
+from airflow.executors.local_executor import LocalExecutor
+from airflow.models.taskinstance import SimpleTaskInstance, TaskInstance, TaskInstanceKey
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class LocalKubernetesExecutor(LoggingMixin):
+ """
+ LocalKubernetesExecutor consists of LocalExecutor and KubernetesExecutor.
+ It chooses an executor to use based on the queue defined on the task.
+ When the queue is the value of ``kubernetes_queue`` in section ``[local_kubernetes_executor]``
+ of the configuration (default value: `kubernetes`), KubernetesExecutor is selected to run the task,
+ otherwise, LocalExecutor is used.
+ """
+
+ supports_ad_hoc_ti_run: bool = True
Review comment:
Yeah @dstandish I think running ad-hocs TIs from Webserver is semi-broken -- I would vote for removing or using "clearing" mechanism under the hood so a new instance of Executor does not need to be created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on a change in pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi commented on a change in pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#discussion_r758848974
##########
File path: tests/executors/test_local_kubernetes_executor.py
##########
@@ -0,0 +1,64 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from unittest import mock
+
+from airflow.configuration import conf
+from airflow.executors.local_executor import LocalExecutor
+from airflow.executors.local_kubernetes_executor import LocalKubernetesExecutor
+
+KUBERNETES_QUEUE = LocalKubernetesExecutor.KUBERNETES_QUEUE
Review comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on a change in pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi commented on a change in pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#discussion_r758835869
##########
File path: airflow/example_dags/example_local_kubernetes_executor.py
##########
@@ -0,0 +1,71 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This is an example dag for using a Kubernetes Executor Configuration.
+"""
+import logging
+from datetime import datetime
+
+from airflow import DAG
+from airflow.configuration import conf
+from airflow.example_dags.libs.helper import print_stuff
+from airflow.operators.python_operator import PythonOperator
+
+log = logging.getLogger(__name__)
+
+worker_container_repository = conf.get('kubernetes', 'worker_container_repository')
+worker_container_tag = conf.get('kubernetes', 'worker_container_tag')
+
+try:
+ from kubernetes.client import models as k8s
+except ImportError:
+ log.warning("Could not import DAGs in example_kubernetes_executor.py", exc_info=True)
+ log.warning("Install Kubernetes dependencies with: pip install apache-airflow[cncf.kubernetes]")
+
+with DAG(
+ dag_id='example_local_kubernetes_executor',
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+ tags=['example3'],
+) as dag:
+ # You can use annotations on your kubernetes pods!
+ start_task_executor_config = {
+ "pod_override": k8s.V1Pod(metadata=k8s.V1ObjectMeta(annotations={"test": "annotation"}))
+ }
+
+ task_with_template = PythonOperator(
+ task_id="task_with_kubernetes_executor",
+ python_callable=print_stuff,
+ pool='kubernetes',
+ queue='kubernetes',
+ executor_config=start_task_executor_config,
+ )
+
+ def print_context(ds, **kwargs):
+ """Print the Airflow context and ds variable from the context."""
+ print(kwargs)
+ print(ds)
+ return 'Whatever you return gets printed in the logs'
+
+ task_with_local = PythonOperator(
+ task_id="task_with_local_executor",
+ python_callable=print_context,
+ )
Review comment:
Changed to use taskflow api.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a change in pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#discussion_r764477823
##########
File path: airflow/executors/local_kubernetes_executor.py
##########
@@ -0,0 +1,204 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Dict, List, Optional, Set, Union
+
+from airflow.configuration import conf
+from airflow.executors.base_executor import CommandType, EventBufferValueType, QueuedTaskInstanceType
+from airflow.executors.kubernetes_executor import KubernetesExecutor
+from airflow.executors.local_executor import LocalExecutor
+from airflow.models.taskinstance import SimpleTaskInstance, TaskInstance, TaskInstanceKey
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class LocalKubernetesExecutor(LoggingMixin):
+ """
+ LocalKubernetesExecutor consists of LocalExecutor and KubernetesExecutor.
+ It chooses an executor to use based on the queue defined on the task.
+ When the queue is the value of ``kubernetes_queue`` in section ``[local_kubernetes_executor]``
Review comment:
```suggestion
When the task's queue is the value of ``kubernetes_queue`` in section ``[local_kubernetes_executor]``
```
##########
File path: airflow/executors/local_kubernetes_executor.py
##########
@@ -0,0 +1,204 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Dict, List, Optional, Set, Union
+
+from airflow.configuration import conf
+from airflow.executors.base_executor import CommandType, EventBufferValueType, QueuedTaskInstanceType
+from airflow.executors.kubernetes_executor import KubernetesExecutor
+from airflow.executors.local_executor import LocalExecutor
+from airflow.models.taskinstance import SimpleTaskInstance, TaskInstance, TaskInstanceKey
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class LocalKubernetesExecutor(LoggingMixin):
+ """
+ LocalKubernetesExecutor consists of LocalExecutor and KubernetesExecutor.
+ It chooses an executor to use based on the queue defined on the task.
Review comment:
```suggestion
It chooses the executor to use based on the queue defined on the task.
```
##########
File path: docs/apache-airflow/executor/local_kubernetes.rst
##########
@@ -0,0 +1,30 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+.. _executor:LocalKubernetesExecutor:
+
+LocalKubernetes Executor
+=========================
+
+The :class:`~airflow.executors.local_kubernetes_executor.LocalKubernetesExecutor` allows users
+to simultaneously run a ``LocalExecutor`` and a ``KubernetesExecutor``.
+An executor is chosen to run a task based on the task's queue.
+
+``LocalKubernetesExecutor`` provides the capability of running low-intensive tasks locally
+using ``LocalExecutor`` and to handle the high load at the peak time and runtime isolation
+of the ``KubernetesExecutor``.
Review comment:
```suggestion
``LocalKubernetesExecutor`` provides the capability of running tasks with either ``LocalExecutor``,
which runs tasks within the scheduler service, or with ``KubernetesExecutor``, which runs each task
in its own pod on a kubernetes cluster.
```
##########
File path: airflow/executors/local_kubernetes_executor.py
##########
@@ -0,0 +1,204 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Dict, List, Optional, Set, Union
+
+from airflow.configuration import conf
+from airflow.executors.base_executor import CommandType, EventBufferValueType, QueuedTaskInstanceType
+from airflow.executors.kubernetes_executor import KubernetesExecutor
+from airflow.executors.local_executor import LocalExecutor
+from airflow.models.taskinstance import SimpleTaskInstance, TaskInstance, TaskInstanceKey
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class LocalKubernetesExecutor(LoggingMixin):
+ """
+ LocalKubernetesExecutor consists of LocalExecutor and KubernetesExecutor.
+ It chooses an executor to use based on the queue defined on the task.
+ When the queue is the value of ``kubernetes_queue`` in section ``[local_kubernetes_executor]``
+ of the configuration (default value: `kubernetes`), KubernetesExecutor is selected to run the task,
+ otherwise, LocalExecutor is used.
+ """
+
+ supports_ad_hoc_ti_run: bool = True
Review comment:
i think you need also to update the `flash` message [here](https://github.com/apache/airflow/blob/b5c0158b2eb646eb1db5d2c094d3da8f88a08a8b/airflow/www/views.py#L1607-L1609)
##########
File path: airflow/executors/local_kubernetes_executor.py
##########
@@ -0,0 +1,204 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Dict, List, Optional, Set, Union
+
+from airflow.configuration import conf
+from airflow.executors.base_executor import CommandType, EventBufferValueType, QueuedTaskInstanceType
+from airflow.executors.kubernetes_executor import KubernetesExecutor
+from airflow.executors.local_executor import LocalExecutor
+from airflow.models.taskinstance import SimpleTaskInstance, TaskInstance, TaskInstanceKey
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class LocalKubernetesExecutor(LoggingMixin):
+ """
+ LocalKubernetesExecutor consists of LocalExecutor and KubernetesExecutor.
+ It chooses an executor to use based on the queue defined on the task.
+ When the queue is the value of ``kubernetes_queue`` in section ``[local_kubernetes_executor]``
+ of the configuration (default value: `kubernetes`), KubernetesExecutor is selected to run the task,
+ otherwise, LocalExecutor is used.
+ """
+
+ supports_ad_hoc_ti_run: bool = True
Review comment:
on a related note... according to [that message](https://github.com/apache/airflow/blob/b5c0158b2eb646eb1db5d2c094d3da8f88a08a8b/airflow/www/views.py#L1607-L1609), celery / CKE, and KE all support "ad hoc" run. why doesn't LocalExecutor? it looks like it's because it starts up an instance of the executor from within the webserver. i'm surprised this works with k8s because i thought i recalled some issues where the webserver didn't have the right roles to create pods... but that's another story. Anyway, perhaps we need to add some way, more sophisticated now than `supports_ad_hoc_ti_run`, such that the ad hoc trigger will work for tasks with k8s queue but not for local. thoughts @kaxil ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on a change in pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi commented on a change in pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#discussion_r764916183
##########
File path: airflow/executors/local_kubernetes_executor.py
##########
@@ -0,0 +1,204 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Dict, List, Optional, Set, Union
+
+from airflow.configuration import conf
+from airflow.executors.base_executor import CommandType, EventBufferValueType, QueuedTaskInstanceType
+from airflow.executors.kubernetes_executor import KubernetesExecutor
+from airflow.executors.local_executor import LocalExecutor
+from airflow.models.taskinstance import SimpleTaskInstance, TaskInstance, TaskInstanceKey
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class LocalKubernetesExecutor(LoggingMixin):
+ """
+ LocalKubernetesExecutor consists of LocalExecutor and KubernetesExecutor.
+ It chooses an executor to use based on the queue defined on the task.
+ When the queue is the value of ``kubernetes_queue`` in section ``[local_kubernetes_executor]``
+ of the configuration (default value: `kubernetes`), KubernetesExecutor is selected to run the task,
+ otherwise, LocalExecutor is used.
+ """
+
+ supports_ad_hoc_ti_run: bool = True
Review comment:
Will try the supports_ad_hoc_ti_run setting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi closed pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi closed pull request #19729:
URL: https://github.com/apache/airflow/pull/19729
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] XD-DENG commented on a change in pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
XD-DENG commented on a change in pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#discussion_r757818690
##########
File path: airflow/example_dags/example_local_kubernetes_executor.py
##########
@@ -0,0 +1,71 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This is an example dag for using a Kubernetes Executor Configuration.
+"""
+import logging
+from datetime import datetime
+
+from airflow import DAG
+from airflow.configuration import conf
+from airflow.example_dags.libs.helper import print_stuff
+from airflow.operators.python_operator import PythonOperator
+
+log = logging.getLogger(__name__)
+
+worker_container_repository = conf.get('kubernetes', 'worker_container_repository')
+worker_container_tag = conf.get('kubernetes', 'worker_container_tag')
+
+try:
+ from kubernetes.client import models as k8s
+except ImportError:
+ log.warning("Could not import DAGs in example_kubernetes_executor.py", exc_info=True)
+ log.warning("Install Kubernetes dependencies with: pip install apache-airflow[cncf.kubernetes]")
+
+with DAG(
+ dag_id='example_local_kubernetes_executor',
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+ tags=['example3'],
+) as dag:
+ # You can use annotations on your kubernetes pods!
+ start_task_executor_config = {
+ "pod_override": k8s.V1Pod(metadata=k8s.V1ObjectMeta(annotations={"test": "annotation"}))
+ }
+
+ task_with_template = PythonOperator(
+ task_id="task_with_kubernetes_executor",
+ python_callable=print_stuff,
+ pool='kubernetes',
Review comment:
May you please clarify why we need to specify `pool` here? If it's not necessary, we can remove it?
##########
File path: airflow/example_dags/example_local_kubernetes_executor.py
##########
@@ -0,0 +1,71 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This is an example dag for using a Kubernetes Executor Configuration.
Review comment:
I guess it's a copy-paste error here: `Kubernetes Executor Configuration` -> `LocalKubernetesExecutor ...`
##########
File path: airflow/config_templates/config.yml
##########
@@ -1446,6 +1446,21 @@
type: string
example: ~
default: ~
+- name: local_kubernetes_executor
+ description: |
+ This section only applies if you are using the ``LocalKubernetesExecutor`` in
+ ``[core]`` section above
+ options:
+ - name: kubernetes_queue
+ description: |
+ Define when to send a task to ``KubernetesExecutor`` when using ``LocalKubernetesExecutor``.
+ When the queue of a task is the value of ``kubernetes_queue`` (default ``kubernetes``),
+ the task is executed via ``KubernetesExecutor``,
+ otherwise via ``LocalExecutor``
+ version_added: 2.2.0
Review comment:
I saw that @kaxil has added milestone `2.3.0` for this, so let's update this line here as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi closed pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi closed pull request #19729:
URL: https://github.com/apache/airflow/pull/19729
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on a change in pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi commented on a change in pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#discussion_r759718753
##########
File path: airflow/example_dags/example_local_kubernetes_executor.py
##########
@@ -0,0 +1,70 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This is an example dag for using a Local Kubernetes Executor Configuration.
+"""
+import logging
+from datetime import datetime
+
+from airflow import DAG
+from airflow.configuration import conf
+from airflow.decorators import task
+from airflow.example_dags.libs.helper import print_stuff
+
+log = logging.getLogger(__name__)
+
+worker_container_repository = conf.get('kubernetes', 'worker_container_repository')
+worker_container_tag = conf.get('kubernetes', 'worker_container_tag')
+
+try:
+ from kubernetes.client import models as k8s
+except ImportError:
+ log.warning("Could not import DAGs in example_local_kubernetes_executor.py", exc_info=True)
+ log.warning("Install Kubernetes dependencies with: pip install apache-airflow[cncf.kubernetes]")
+
+with DAG(
+ dag_id='example_local_kubernetes_executor',
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+ tags=['example3'],
+) as dag:
+ # You can use annotations on your kubernetes pods!
+ start_task_executor_config = {
+ "pod_override": k8s.V1Pod(metadata=k8s.V1ObjectMeta(annotations={"test": "annotation"}))
+ }
+
+ @task(
+ executor_config='start_task_executor_config',
+ queue='kubernetes',
+ task_id='task_with_kubernetes_executor',
+ )
+ def task_with_template():
+ print_stuff()
+
+ def print_context(ds, **kwargs):
+ """Print the Airflow context and ds variable from the context."""
+ print(kwargs)
+ print(ds)
+ return 'Whatever you return gets printed in the logs'
+
+ @task(task_id='task_with_local_executor')
+ def task_with_local():
+ print_context()
+
+ task_with_local >> task_with_template
Review comment:
THanks, made the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-1065894379
> > Hey, whats the timeline for this to be added to the helm chart, I'd like to test this out right away.
>
> Maybe you can try update it in the chart and provide it as a PR back? That would be great contribution back to the community for the free software and you can become one of the (almost) 2000 contributors then.
Note that this executor will only berelease in 2.3.0 - development only, so this makes perfect sense to add it soon in main BTW. PR with it are most welcome
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-980207459
I think it woudl be good to continue with it "as is" without the explicit spelling out the executors but using queue - but then it could be separate PR where we could introduce the other cases.
I think we will find out that it might be quite an effort to complete the latter, and this one has much higher chance to be merged and available earlier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-982016356
some doca failing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on a change in pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi commented on a change in pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#discussion_r758837078
##########
File path: airflow/example_dags/example_local_kubernetes_executor.py
##########
@@ -0,0 +1,71 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This is an example dag for using a Kubernetes Executor Configuration.
Review comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-980236187
This is where I am sure there will be a lot of discussion. That's why IMHO "MultiExecutor" needs to be separated to another PR and possibly discussed in more detail because exactly those questions you have should be answered first and maybe new ones will appear as well.
For now (in this PR) "LocaKubernetes" executor should behave in similar way to "CeleryKubernetes" one - no "multi executor" yet - simply executor=LocalKubernetesExecutor should be set and about the only confgurable thing should be kubernetes queue. Similarly to:
```
[celery_kubernetes_executor]
This section only applies if you are using the CeleryKubernetesExecutor in [core] section above
kubernetes_queue
Define when to send a task to KubernetesExecutor when using CeleryKubernetesExecutor. When the queue of a task is the value of kubernetes_queue (default kubernetes), the task is executed via KubernetesExecutor, otherwise via CeleryExecutor
Type
string
Default
kubernetes
Environment Variable
AIRFLOW__CELERY_KUBERNETES_EXECUTOR__KUBERNETES_QUEUE
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-1019597187
This one needs rebase!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-1042916898
One comment - before merge (and @subkanthi - maybe you can take a closer look while you are adding it).
There is a bug with CeleryKubernetesExecutor reported recently https://github.com/apache/airflow/issues/21548 which might also affect LocalKubernetesExecutor. Would it be possible that you take a look @subkanthi - and maybe you will be able to fix it as general case ? And maybe also take a look if there are other places where we have "hard-coded" Executor names
This was one thing that I was afraid of when we discussed "Generic Combined Excecutor" - seems that there might be at least this one place pointed out in #21548 where we implemented logic based on the type of executor configured and about the executor actually 'used" when the task was executed.
I think that LocalKubernetesExecutor might suffer the same problem actually - because `file_task_handler` was not touched by this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi edited a comment on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi edited a comment on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-1045231490
> Waiting for response from @subkanthi on the potential bug.
Im actually stuck with another issue, Im pretty sure it used to work before, For some reason the k8s_object is str and it ends up in the else block
```
k8s_object = obj.get("pod_override", None)
if k8s_legacy_object and k8s_object:
raise AirflowConfigException(
"Can not have both a legacy and new"
"executor_config object. Please delete the KubernetesExecutor"
"dict and only use the pod_override kubernetes.client.models.V1Pod"
"object."
)
if not k8s_object and not k8s_legacy_object:
return None
print("TYPE" + str(type(k8s_object)))
if isinstance(k8s_object, k8s.V1Pod):
return k8s_object
elif isinstance(k8s_legacy_object, dict):
warnings.warn(
'Using a dictionary for the executor_config is deprecated and will soon be removed.'
'please use a `kubernetes.client.models.V1Pod` class with a "pod_override" key'
' instead. ',
category=DeprecationWarning,
)
return PodGenerator.from_legacy_obj(obj)
else:
raise TypeError(
'Cannot convert a non-kubernetes.client.models.V1Pod object into a KubernetesExecutorConfig'
)
```
```
kube_exec_config_special = {
"pod_override": k8s.V1Pod(
spec=k8s.V1PodSpec(
containers=[
k8s.V1Container(
name="base", image="python"
),
]
)
)
}
@task(
executor_config=kube_exec_config_special,
queue='kubernetes',
task_id='task_with_kubernetes_executor',
)
def task_with_template():
print_stuff()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-1047050990
> > Waiting for response from @subkanthi on the potential bug.
>
> Im actually stuck with another issue, Im pretty sure it used to work before, For some reason the k8s_object is str and it ends up in the else block
>
> ```
> k8s_object = obj.get("pod_override", None)
>
> if k8s_legacy_object and k8s_object:
> raise AirflowConfigException(
> "Can not have both a legacy and new"
> "executor_config object. Please delete the KubernetesExecutor"
> "dict and only use the pod_override kubernetes.client.models.V1Pod"
> "object."
> )
> if not k8s_object and not k8s_legacy_object:
> return None
>
> print("TYPE" + str(type(k8s_object)))
> if isinstance(k8s_object, k8s.V1Pod):
> return k8s_object
> elif isinstance(k8s_legacy_object, dict):
> warnings.warn(
> 'Using a dictionary for the executor_config is deprecated and will soon be removed.'
> 'please use a `kubernetes.client.models.V1Pod` class with a "pod_override" key'
> ' instead. ',
> category=DeprecationWarning,
> )
> return PodGenerator.from_legacy_obj(obj)
> else:
>
> raise TypeError(
> 'Cannot convert a non-kubernetes.client.models.V1Pod object into a KubernetesExecutorConfig'
> )
> ```
>
> ```
> kube_exec_config_special = {
> "pod_override": k8s.V1Pod(
> spec=k8s.V1PodSpec(
> containers=[
> k8s.V1Container(
> name="base", image="python"
> ),
> ]
> )
> )
> }
> @task(
> executor_config=kube_exec_config_special,
> queue='kubernetes',
> task_id='task_with_kubernetes_executor',
> )
> def task_with_template():
> print_stuff()
> ```
The problem seems to be that for some reason `executor_config` seems to get unpicked as `str`. Im trying to figure out the right combination on when it happens.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on a change in pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi commented on a change in pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#discussion_r755262844
##########
File path: airflow/example_dags/example_local_kubernetes_executor.py
##########
@@ -0,0 +1,71 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This is an example dag for using a Kubernetes Executor Configuration.
+"""
+import logging
+from datetime import datetime
+
+from airflow import DAG
+from airflow.configuration import conf
+from airflow.example_dags.libs.helper import print_stuff
+from airflow.operators.python_operator import PythonOperator
+
+log = logging.getLogger(__name__)
+
+worker_container_repository = conf.get('kubernetes', 'worker_container_repository')
+worker_container_tag = conf.get('kubernetes', 'worker_container_tag')
+
+try:
+ from kubernetes.client import models as k8s
+except ImportError:
+ log.warning("Could not import DAGs in example_kubernetes_executor.py", exc_info=True)
+ log.warning("Install Kubernetes dependencies with: pip install apache-airflow[cncf.kubernetes]")
+
+with DAG(
+ dag_id='example_local_kubernetes_executor',
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+ tags=['example3'],
+) as dag:
+ # You can use annotations on your kubernetes pods!
+ start_task_executor_config = {
+ "pod_override": k8s.V1Pod(metadata=k8s.V1ObjectMeta(annotations={"test": "annotation"}))
+ }
+
+ task_with_template = PythonOperator(
+ task_id="task_with_kubernetes_executor",
+ python_callable=print_stuff,
+ pool='kubernetes',
+ queue='kubernetes',
+ executor_config=start_task_executor_config,
+ )
+
+ def print_context(ds, **kwargs):
+ """Print the Airflow context and ds variable from the context."""
+ print(kwargs)
+ print(ds)
+ return 'Whatever you return gets printed in the logs'
+
+ task_with_local = PythonOperator(
+ task_id="task_with_local_executor",
+ python_callable=print_context,
+ )
Review comment:
I couldn't figure out on how to pass the queue='kubernetes', in the @task decorator. The example_kubernetes_executor doesn't do it. Let me try again. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on a change in pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
josh-fell commented on a change in pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#discussion_r758865806
##########
File path: airflow/example_dags/example_local_kubernetes_executor.py
##########
@@ -0,0 +1,70 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This is an example dag for using a Local Kubernetes Executor Configuration.
+"""
+import logging
+from datetime import datetime
+
+from airflow import DAG
+from airflow.configuration import conf
+from airflow.decorators import task
+from airflow.example_dags.libs.helper import print_stuff
+
+log = logging.getLogger(__name__)
+
+worker_container_repository = conf.get('kubernetes', 'worker_container_repository')
+worker_container_tag = conf.get('kubernetes', 'worker_container_tag')
+
+try:
+ from kubernetes.client import models as k8s
+except ImportError:
+ log.warning("Could not import DAGs in example_local_kubernetes_executor.py", exc_info=True)
+ log.warning("Install Kubernetes dependencies with: pip install apache-airflow[cncf.kubernetes]")
+
+with DAG(
+ dag_id='example_local_kubernetes_executor',
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+ tags=['example3'],
+) as dag:
+ # You can use annotations on your kubernetes pods!
+ start_task_executor_config = {
+ "pod_override": k8s.V1Pod(metadata=k8s.V1ObjectMeta(annotations={"test": "annotation"}))
+ }
+
+ @task(
+ executor_config='start_task_executor_config',
+ queue='kubernetes',
+ task_id='task_with_kubernetes_executor',
+ )
+ def task_with_template():
+ print_stuff()
+
+ def print_context(ds, **kwargs):
+ """Print the Airflow context and ds variable from the context."""
+ print(kwargs)
+ print(ds)
+ return 'Whatever you return gets printed in the logs'
+
+ @task(task_id='task_with_local_executor')
+ def task_with_local():
+ print_context()
+
+ task_with_local >> task_with_template
Review comment:
```suggestion
@task(
executor_config=start_task_executor_config,
queue='kubernetes',
task_id='task_with_kubernetes_executor',
)
def task_with_template():
print_stuff()
@task(task_id='task_with_local_executor')
def task_with_local(ds=None, **kwargs):
"""Print the Airflow context and ds variable from the context."""
print(kwargs)
print(ds)
return 'Whatever you return gets printed in the logs'
task_with_local() >> task_with_template()
```
A few things:
- The `executor_config` in the `task_with_template()` TaskFlow function should reference the `start_task_executor_config` rather a string with the same name.
- Instead of having a TaskFlow Function call `print_context()`, you can simplify this task by having it perform what the `print_context()` function is doing. Alternatively, you could decorate the `print_context()` with `@task(...)` and update the task dependency notation to match. Either way, no need to have a TaskFlow function call a function that's inside the DAG.
- Updating the task dependencies to avoid errors when parsing the DAG.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi edited a comment on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi edited a comment on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-980215844
> Airflow 2.3.0
> I think it woudl be good to continue with it "as is" without the explicit spelling out the executors but using queue - but then it could be separate PR where we could introduce the other cases.
>
> I think we will find out that it might be quite an effort to complete the latter, and this one has much higher chance to be merged and available earlier.
Thanks @potiuk I understand the queue part, so if queue is "local" then the localExecutor would be used and if its "kubernetes" , then the kubernetesExecutor, but clarifying this part
**I think it woudl be good to continue with it "as is" without the explicit spelling out the executors**
Question is what executor will be defined in the configuration, MultipleExecutor?
Would config look like this
executor=MutipleExecutor
[multiple_executor]
local_queue=local
kubernetes_queue=kubernetes
default_queue=celery
Also the question is where will the defaults be executed, if the queue is not provided, is default always CeleryExecutor? or maybe we provide default in config
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a change in pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#discussion_r765219193
##########
File path: airflow/executors/local_kubernetes_executor.py
##########
@@ -0,0 +1,204 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Dict, List, Optional, Set, Union
+
+from airflow.configuration import conf
+from airflow.executors.base_executor import CommandType, EventBufferValueType, QueuedTaskInstanceType
+from airflow.executors.kubernetes_executor import KubernetesExecutor
+from airflow.executors.local_executor import LocalExecutor
+from airflow.models.taskinstance import SimpleTaskInstance, TaskInstance, TaskInstanceKey
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class LocalKubernetesExecutor(LoggingMixin):
+ """
+ LocalKubernetesExecutor consists of LocalExecutor and KubernetesExecutor.
+ It chooses an executor to use based on the queue defined on the task.
+ When the queue is the value of ``kubernetes_queue`` in section ``[local_kubernetes_executor]``
+ of the configuration (default value: `kubernetes`), KubernetesExecutor is selected to run the task,
+ otherwise, LocalExecutor is used.
+ """
+
+ supports_ad_hoc_ti_run: bool = True
Review comment:
yup
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-1065894130
> Hey, whats the timeline for this to be added to the helm chart, I'd like to test this out right away.
Maybe you can try update it in the chart and provide it as a PR back? That would be great contribution back to the community for the free software and you can become one of the (almost) 2000 contributors then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hendoxc commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
hendoxc commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-1065893742
Hey, whats the timeline for this to be added to the helm chart, I'd like to test this out right away.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-974879206
Nice one. I think LocalK8S is quite a useful executor to have :).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-982051315
> Needs adding the executor to https://airflow.apache.org/docs/apache-airflow/stable/executor/index.html#executor-types
>
> and doc on when to use it same as https://airflow.apache.org/docs/apache-airflow/stable/executor/celery_kubernetes.html#when-to-use-celerykubernetesexecutor
Thanks will add it there, was also trying to find out the docs files for this
Needs adding the executor to https://airflow.apache.org/docs/apache-airflow/stable/executor/index.html#executor-types
and doc on when to use it same as https://airflow.apache.org/docs/apache-airflow/stable/executor/celery_kubernetes.html#when-to-use-celerykubernetesexecutor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-982044424
In `docs/apache-airlfow/executor/index.rst` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on a change in pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi commented on a change in pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#discussion_r758836047
##########
File path: airflow/config_templates/config.yml
##########
@@ -1446,6 +1446,21 @@
type: string
example: ~
default: ~
+- name: local_kubernetes_executor
+ description: |
+ This section only applies if you are using the ``LocalKubernetesExecutor`` in
+ ``[core]`` section above
+ options:
+ - name: kubernetes_queue
+ description: |
+ Define when to send a task to ``KubernetesExecutor`` when using ``LocalKubernetesExecutor``.
+ When the queue of a task is the value of ``kubernetes_queue`` (default ``kubernetes``),
+ the task is executed via ``KubernetesExecutor``,
+ otherwise via ``LocalExecutor``
+ version_added: 2.2.0
Review comment:
Updated, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on a change in pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi commented on a change in pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#discussion_r757820287
##########
File path: airflow/example_dags/example_local_kubernetes_executor.py
##########
@@ -0,0 +1,71 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This is an example dag for using a Kubernetes Executor Configuration.
+"""
+import logging
+from datetime import datetime
+
+from airflow import DAG
+from airflow.configuration import conf
+from airflow.example_dags.libs.helper import print_stuff
+from airflow.operators.python_operator import PythonOperator
+
+log = logging.getLogger(__name__)
+
+worker_container_repository = conf.get('kubernetes', 'worker_container_repository')
+worker_container_tag = conf.get('kubernetes', 'worker_container_tag')
+
+try:
+ from kubernetes.client import models as k8s
+except ImportError:
+ log.warning("Could not import DAGs in example_kubernetes_executor.py", exc_info=True)
+ log.warning("Install Kubernetes dependencies with: pip install apache-airflow[cncf.kubernetes]")
+
+with DAG(
+ dag_id='example_local_kubernetes_executor',
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+ tags=['example3'],
+) as dag:
+ # You can use annotations on your kubernetes pods!
+ start_task_executor_config = {
+ "pod_override": k8s.V1Pod(metadata=k8s.V1ObjectMeta(annotations={"test": "annotation"}))
+ }
+
+ task_with_template = PythonOperator(
+ task_id="task_with_kubernetes_executor",
+ python_callable=print_stuff,
+ pool='kubernetes',
Review comment:
I was just trying something, will remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on a change in pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
subkanthi commented on a change in pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#discussion_r765212729
##########
File path: airflow/executors/local_kubernetes_executor.py
##########
@@ -0,0 +1,204 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Dict, List, Optional, Set, Union
+
+from airflow.configuration import conf
+from airflow.executors.base_executor import CommandType, EventBufferValueType, QueuedTaskInstanceType
+from airflow.executors.kubernetes_executor import KubernetesExecutor
+from airflow.executors.local_executor import LocalExecutor
+from airflow.models.taskinstance import SimpleTaskInstance, TaskInstance, TaskInstanceKey
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class LocalKubernetesExecutor(LoggingMixin):
+ """
+ LocalKubernetesExecutor consists of LocalExecutor and KubernetesExecutor.
+ It chooses an executor to use based on the queue defined on the task.
+ When the queue is the value of ``kubernetes_queue`` in section ``[local_kubernetes_executor]``
+ of the configuration (default value: `kubernetes`), KubernetesExecutor is selected to run the task,
+ otherwise, LocalExecutor is used.
+ """
+
+ supports_ad_hoc_ti_run: bool = True
Review comment:
@dstandish i guess I don't have to update the flash message, correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a change in pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#discussion_r765219193
##########
File path: airflow/executors/local_kubernetes_executor.py
##########
@@ -0,0 +1,204 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Dict, List, Optional, Set, Union
+
+from airflow.configuration import conf
+from airflow.executors.base_executor import CommandType, EventBufferValueType, QueuedTaskInstanceType
+from airflow.executors.kubernetes_executor import KubernetesExecutor
+from airflow.executors.local_executor import LocalExecutor
+from airflow.models.taskinstance import SimpleTaskInstance, TaskInstance, TaskInstanceKey
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class LocalKubernetesExecutor(LoggingMixin):
+ """
+ LocalKubernetesExecutor consists of LocalExecutor and KubernetesExecutor.
+ It chooses an executor to use based on the queue defined on the task.
+ When the queue is the value of ``kubernetes_queue`` in section ``[local_kubernetes_executor]``
+ of the configuration (default value: `kubernetes`), KubernetesExecutor is selected to run the task,
+ otherwise, LocalExecutor is used.
+ """
+
+ supports_ad_hoc_ti_run: bool = True
Review comment:
right
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #19729:
URL: https://github.com/apache/airflow/pull/19729#issuecomment-988351165
cc @potiuk @dstandish for a second review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #19729: Local kubernetes executor
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #19729:
URL: https://github.com/apache/airflow/pull/19729
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org