You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "sonmezonur (via GitHub)" <gi...@apache.org> on 2023/08/07 13:53:27 UTC
[GitHub] [airflow] sonmezonur opened a new pull request, #33174: add pod_template_content field to kubernetes pod operator
sonmezonur opened a new pull request, #33174:
URL: https://github.com/apache/airflow/pull/33174
I would like to propose adding a new templated field to kubernetes pod operator. Currently, operator has `pod_template_file` field to create base pod spec but you need to create a file to use this feature. IMHO it might be useful to accept stream or content, thus users might easily configure the base pod spec.
**Use Case**
I would like to write airflow dag which clones git repository and mounts volume for cloned branch. Branch name should be parameterized so that users can specify it while running the DAG. Example code:
```python
templated_pod = k8s.V1Pod(
metadata=k8s.V1ObjectMeta(),
spec=k8s.V1PodSpec(
containers=[],
init_containers=[
k8s.V1Container(
name='git-clone',
image='registry.k8s.io/git-sync:v3.1.1',
args=[
f'--repo=git@github.com:airflow/some_repo.git',
f'--branch={{ params.get("repo_branch", "master") }}',
'--root=/tmp/git',
'--dest=gitclone',
'--ssh=true',
'--wait=120',
'--one-time=true',
],
volume_mounts=[self.git_sync_mount, self.ssh_key_mount],
security_context=k8s.V1SecurityContext(run_as_user=65533),
resources=k8s.V1ResourceRequirements(
requests={
"cpu": "300m",
"memory": "512Mi",
},
limits={
"cpu": "300m",
"memory": "512Mi",
},
),
],
volumes=[],
),
)
serialized_pod = yaml.dump(
pod_generator.PodGenerator.serialize_pod(templated_pod)
)
operator = KubernetesPodOperator(pod_templa
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sonmezonur commented on pull request #33174: Add `pod_template_content` field to `KubernetesPodOperator`
Posted by "sonmezonur (via GitHub)" <gi...@apache.org>.
sonmezonur commented on PR #33174:
URL: https://github.com/apache/airflow/pull/33174#issuecomment-1669554651
> I like the idea, but it needs at least some unit tests to be considered as worthy merging. Without tests it's prone to regressions.
Added testcase
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `pod_template_dict` field to `KubernetesPodOperator` [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #33174:
URL: https://github.com/apache/airflow/pull/33174#discussion_r1429222336
##########
tests/providers/cncf/kubernetes/operators/test_pod.py:
##########
@@ -28,6 +28,7 @@
from urllib3 import HTTPResponse
from airflow.exceptions import AirflowException, AirflowSkipException, TaskDeferred
+from airflow.providers.cncf.kubernetes import pod_generator
from airflow.models import DAG, DagModel, DagRun, TaskInstance
from airflow.models.xcom import XCom
Review Comment:
```suggestion
from airflow.models import DAG, DagModel, DagRun, TaskInstance
from airflow.models.xcom import XCom
from airflow.providers.cncf.kubernetes import pod_generator
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `pod_template_dict` field to `KubernetesPodOperator` [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #33174:
URL: https://github.com/apache/airflow/pull/33174#discussion_r1429096075
##########
tests/providers/cncf/kubernetes/operators/test_pod.py:
##########
@@ -28,6 +28,7 @@
from urllib3 import HTTPResponse
from airflow.exceptions import AirflowException, AirflowSkipException, TaskDeferred
+from airflow.kubernetes import pod_generator
Review Comment:
```suggestion
from airflow.providers.cncf.kubernetes import pod_generator
```
To fix the import error
after that it requires also to rearrange order of imports
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `pod_template_dict` field to `KubernetesPodOperator` [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #33174:
URL: https://github.com/apache/airflow/pull/33174#issuecomment-1859254906
Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `pod_template_dict` field to `KubernetesPodOperator` [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal merged PR #33174:
URL: https://github.com/apache/airflow/pull/33174
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sonmezonur commented on a diff in pull request #33174: Add `pod_template_dict` field to `KubernetesPodOperator`
Posted by "sonmezonur (via GitHub)" <gi...@apache.org>.
sonmezonur commented on code in PR #33174:
URL: https://github.com/apache/airflow/pull/33174#discussion_r1294357934
##########
tests/providers/cncf/kubernetes/operators/test_pod.py:
##########
@@ -920,6 +921,64 @@ def test_pod_template_file_kwargs_override(self, randomize_name, pod_template_fi
"run_id": "test",
}
+ @pytest.mark.parametrize(("randomize_name",), ([True], [False]))
+ def test_pod_template_dict(self, randomize_name):
+ templated_pod = k8s.V1Pod(
+ metadata=k8s.V1ObjectMeta(
+ namespace="templatenamespace",
+ name="hello",
+ labels={"release": "stable"},
+ ),
+ spec=k8s.V1PodSpec(
+ containers=[],
+ init_containers=[
+ k8s.V1Container(
+ name="git-clone",
+ image="registry.k8s.io/git-sync:v3.1.1",
+ args=[
+ "--repo=git@github.com:airflow/some_repo.git",
+ "--branch={{ params.get('repo_branch', 'master') }}",
+ ],
+ ),
+ ],
+ ),
+ )
+ k = KubernetesPodOperator(
+ task_id="task",
+ random_name_suffix=randomize_name,
+ pod_template_dict=pod_generator.PodGenerator.serialize_pod(templated_pod),
Review Comment:
My first intention was to make `full_pod_spec` field templated instead of adding a new field. But templater does not resolve `V1Pod` as you referred. I might have a simple solution, will try once I have some time 🤞
##########
tests/providers/cncf/kubernetes/operators/test_pod.py:
##########
@@ -920,6 +921,64 @@ def test_pod_template_file_kwargs_override(self, randomize_name, pod_template_fi
"run_id": "test",
}
+ @pytest.mark.parametrize(("randomize_name",), ([True], [False]))
+ def test_pod_template_dict(self, randomize_name):
+ templated_pod = k8s.V1Pod(
+ metadata=k8s.V1ObjectMeta(
+ namespace="templatenamespace",
+ name="hello",
+ labels={"release": "stable"},
+ ),
+ spec=k8s.V1PodSpec(
+ containers=[],
+ init_containers=[
+ k8s.V1Container(
+ name="git-clone",
+ image="registry.k8s.io/git-sync:v3.1.1",
+ args=[
+ "--repo=git@github.com:airflow/some_repo.git",
+ "--branch={{ params.get('repo_branch', 'master') }}",
+ ],
+ ),
+ ],
+ ),
+ )
+ k = KubernetesPodOperator(
+ task_id="task",
+ random_name_suffix=randomize_name,
+ pod_template_dict=pod_generator.PodGenerator.serialize_pod(templated_pod),
Review Comment:
My first intention was to make `full_pod_spec` field templated instead of adding a new field. But templater does not resolve `V1Pod` as you referred. I might have a simple solution, will try once I have some time 🤞
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #33174: Add `pod_template_dict` field to `KubernetesPodOperator`
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33174:
URL: https://github.com/apache/airflow/pull/33174#issuecomment-1741596365
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sonmezonur commented on pull request #33174: Add `pod_template_dict` field to `KubernetesPodOperator`
Posted by "sonmezonur (via GitHub)" <gi...@apache.org>.
sonmezonur commented on PR #33174:
URL: https://github.com/apache/airflow/pull/33174#issuecomment-1669580184
My first proposal was to add content string (in YAML format) but while writing unit tests, I realized that a pod template dictionary might be easier to use. What do you think? Refactored the code in my last commit. PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on a diff in pull request #33174: Add `pod_template_dict` field to `KubernetesPodOperator`
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on code in PR #33174:
URL: https://github.com/apache/airflow/pull/33174#discussion_r1289115208
##########
tests/providers/cncf/kubernetes/operators/test_pod.py:
##########
@@ -920,6 +921,64 @@ def test_pod_template_file_kwargs_override(self, randomize_name, pod_template_fi
"run_id": "test",
}
+ @pytest.mark.parametrize(("randomize_name",), ([True], [False]))
+ def test_pod_template_dict(self, randomize_name):
+ templated_pod = k8s.V1Pod(
+ metadata=k8s.V1ObjectMeta(
+ namespace="templatenamespace",
+ name="hello",
+ labels={"release": "stable"},
+ ),
+ spec=k8s.V1PodSpec(
+ containers=[],
+ init_containers=[
+ k8s.V1Container(
+ name="git-clone",
+ image="registry.k8s.io/git-sync:v3.1.1",
+ args=[
+ "--repo=git@github.com:airflow/some_repo.git",
+ "--branch={{ params.get('repo_branch', 'master') }}",
+ ],
+ ),
+ ],
+ ),
+ )
+ k = KubernetesPodOperator(
+ task_id="task",
+ random_name_suffix=randomize_name,
+ pod_template_dict=pod_generator.PodGenerator.serialize_pod(templated_pod),
Review Comment:
I wonder if we should support both types `V1Pod` and dict :thinking:
I tested it and it seems like when we use `V1Pod` the template fields are not resolved. I will check if there is a simple solution for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #33174: add pod_template_content field to kubernetes pod operator
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #33174:
URL: https://github.com/apache/airflow/pull/33174#issuecomment-1667902798
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (ruff, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices).
- Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #33174: add pod_template_content field to kubernetes pod operator
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33174:
URL: https://github.com/apache/airflow/pull/33174#issuecomment-1668562236
I like the idea, but it needs at least some unit tests to be considered as worthy merging. Without tests it's prone to regressions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org