You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/22 18:18:47 UTC
[GitHub] [airflow] pingzh opened a new pull request #10481: Support pass command as args in k8s executor
pingzh opened a new pull request #10481:
URL: https://github.com/apache/airflow/pull/10481
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
The current implementation overrides the image's entrypoint, which is less ideal. Users may want to use their own entrypoint to do additional setup and clean up, before and after the airflow tasks.
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman closed pull request #10481: Support pass command as args in k8s executor
Posted by GitBox <gi...@apache.org>.
dimberman closed pull request #10481:
URL: https://github.com/apache/airflow/pull/10481
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pingzh commented on pull request #10481: Support pass command as args in k8s executor
Posted by GitBox <gi...@apache.org>.
pingzh commented on pull request #10481:
URL: https://github.com/apache/airflow/pull/10481#issuecomment-679371074
@dimberman could you please take a look? thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on pull request #10481: Support pass command as args in k8s executor
Posted by GitBox <gi...@apache.org>.
dimberman commented on pull request #10481:
URL: https://github.com/apache/airflow/pull/10481#issuecomment-679372120
Hi ping, going forward we're actually going to be deleting all of the k8sexecutor airflow.cfg configs involved in pod creation. This is because as of 1.10.12 we allow users to generate pod_templates using yaml. Please see the PR here for simplification of k8sexecutor https://github.com/apache/airflow/pull/10393
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dszakallas commented on pull request #10481: Support pass command as args in k8s executor
Posted by GitBox <gi...@apache.org>.
dszakallas commented on pull request #10481:
URL: https://github.com/apache/airflow/pull/10481#issuecomment-691953749
@dimberman the command and args are dynamic, i.e. they could be different for each run, so the only way to customize this behavior from a pod template is to interpolate these parameters using a jinja template during runtime. I don't know if it is possible currently, but would cover more use cases more cleanly. E.g. a common use case not covered by this when the entrypoint relays the args to `airflow` but sets up some environment variables first, in which case only airflow's arguments should be passed to the script.
If we had a two separate variables for the command (`['airflow']`) and args (`['run', 'tasks', 'abc', '...']`) interpolated into the pod template, we could support that behavior too, by using a static command in the pod spec, and using the dynamic args.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pingzh commented on pull request #10481: Support pass command as args in k8s executor
Posted by GitBox <gi...@apache.org>.
pingzh commented on pull request #10481:
URL: https://github.com/apache/airflow/pull/10481#issuecomment-680266639
> Hi ping, going forward we're actually going to be deleting all of the k8sexecutor airflow.cfg configs involved in pod creation. This is because as of 1.10.12 we allow users to generate pod_templates using yaml. Please see the PR here for simplification of k8sexecutor #10393
Thanks, will keep this in mind.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dszakallas edited a comment on pull request #10481: Support pass command as args in k8s executor
Posted by GitBox <gi...@apache.org>.
dszakallas edited a comment on pull request #10481:
URL: https://github.com/apache/airflow/pull/10481#issuecomment-691953749
@dimberman the command and args are dynamic, i.e. they could be different for each run, so the only way to customize this behavior from a pod template is to interpolate these parameters using a jinja template during runtime. I don't know if it is possible currently, but would cover more use cases more cleanly than this approach. E.g. a common use case not covered by this when the entrypoint relays the args to `airflow` but sets up some environment variables first, in which case only airflow's arguments should be passed to the script.
If we had two separate variables for the command (`['airflow']`) and args (`['run', 'tasks', 'abc', '...']`) interpolated into the pod template, we could support that behavior too, by using a static command in the pod spec, and using the dynamic args.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dszakallas edited a comment on pull request #10481: Support pass command as args in k8s executor
Posted by GitBox <gi...@apache.org>.
dszakallas edited a comment on pull request #10481:
URL: https://github.com/apache/airflow/pull/10481#issuecomment-691953749
@dimberman the command and args are dynamic, i.e. they could be different for each run, so the only way to customize this behavior from a pod template is to interpolate these parameters using a jinja template during runtime. I don't know if it is possible currently, but would cover more use cases more cleanly than this approach. E.g. a common use case not covered by this when the entrypoint relays the args to `airflow` but sets up some environment variables first, in which case only airflow's arguments should be passed to the script.
If we had a two separate variables for the command (`['airflow']`) and args (`['run', 'tasks', 'abc', '...']`) interpolated into the pod template, we could support that behavior too, by using a static command in the pod spec, and using the dynamic args.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org