You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/18 17:36:16 UTC
[GitHub] [airflow] baryluk opened a new pull request #19686: Disclaimer in Kubernetes executor pod template
baryluk opened a new pull request #19686:
URL: https://github.com/apache/airflow/pull/19686
This came in when upgrading from 2.1.4 to 2.2.x, and using non-default dag folder.
Because of https://github.com/apache/airflow/issues/8061
all dags stopped working.
Add a note that the examples for pod template are not final,
and require more configuration variables to be passed explicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19686: Disclaimer in Kubernetes executor pod template example
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #19686:
URL: https://github.com/apache/airflow/pull/19686#issuecomment-973115053
The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a change in pull request #19686: Disclaimer in Kubernetes executor pod template example
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on a change in pull request #19686:
URL: https://github.com/apache/airflow/pull/19686#discussion_r752499335
##########
File path: docs/apache-airflow/executor/kubernetes.rst
##########
@@ -91,6 +91,14 @@ Example pod templates
With these requirements in mind, here are some examples of basic ``pod_template_file`` YAML files.
+.. note::
+
Review comment:
I'm not sure these examples work as-is anywhere - they should be used for inspiration.
I'd be in favor of stating these are for inspiration and that care should be taken to ensure all configuration is consistent across components in your deployment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a change in pull request #19686: Disclaimer in Kubernetes executor pod template example
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on a change in pull request #19686:
URL: https://github.com/apache/airflow/pull/19686#discussion_r754500677
##########
File path: docs/apache-airflow/executor/kubernetes.rst
##########
@@ -91,6 +91,14 @@ Example pod templates
With these requirements in mind, here are some examples of basic ``pod_template_file`` YAML files.
+.. note::
+
Review comment:
I can always open another PR to refactor this. I think this area needs more attention regardless.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #19686: Disclaimer in Kubernetes executor pod template example
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #19686:
URL: https://github.com/apache/airflow/pull/19686
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a change in pull request #19686: Disclaimer in Kubernetes executor pod template example
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on a change in pull request #19686:
URL: https://github.com/apache/airflow/pull/19686#discussion_r752620097
##########
File path: docs/apache-airflow/executor/kubernetes.rst
##########
@@ -91,6 +91,14 @@ Example pod templates
With these requirements in mind, here are some examples of basic ``pod_template_file`` YAML files.
+.. note::
+
Review comment:
Maybe we should use a different word than "configuration". I didn't mean specifically "Airflow configuration", I meant in general. Another example would be `imagePullSecrets` - if you need it for the scheduler etc, you'll need it here too.
I'm _more_ in favor of stating these are for inspiration instead of implying they work if you use the "defaults".
Maybe something along these lines:
The following `pod_template_file` examples are for inspiration. This template will be used to create your workers, so take care to ensure all of your Airflow configuration and Kubernetes secrets/mounts/etc are added so the worker pod can function properly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19686: Disclaimer in Kubernetes executor pod template example
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19686:
URL: https://github.com/apache/airflow/pull/19686#discussion_r754421598
##########
File path: docs/apache-airflow/executor/kubernetes.rst
##########
@@ -91,6 +91,14 @@ Example pod templates
With these requirements in mind, here are some examples of basic ``pod_template_file`` YAML files.
+.. note::
+
Review comment:
Hey @jedcunningham - how strong are you with those :) - Maybe you can propose alternative PR if you are not happy with that :)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19686: Disclaimer in Kubernetes executor pod template example
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19686:
URL: https://github.com/apache/airflow/pull/19686#discussion_r752508035
##########
File path: docs/apache-airflow/executor/kubernetes.rst
##########
@@ -91,6 +91,14 @@ Example pod templates
With these requirements in mind, here are some examples of basic ``pod_template_file`` YAML files.
+.. note::
+
Review comment:
> I'd be in favor of stating these are for inspiration and that care should be taken to ensure all configuration is consistent across components in your deployment.
This is precisely the first thing you see when you follow the "config" link: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html
| Use the same configuration across all the Airflow components. While each component does not require all, some configurations need to be same otherwise they would not work as expected. A good example for that is secret_key which should be same on the Webserver and Worker to allow Webserver to fetch logs from Worker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19686: Disclaimer in Kubernetes executor pod template example
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19686:
URL: https://github.com/apache/airflow/pull/19686#discussion_r752508035
##########
File path: docs/apache-airflow/executor/kubernetes.rst
##########
@@ -91,6 +91,14 @@ Example pod templates
With these requirements in mind, here are some examples of basic ``pod_template_file`` YAML files.
+.. note::
+
Review comment:
> I'd be in favor of stating these are for inspiration and that care should be taken to ensure all configuration is consistent across components in your deployment.
This is precisely the first thing you see when you follow the "config" link: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html
> Use the same configuration across all the Airflow components.
> While each component does not require all, some configurations need to be same otherwise they would not work
> as expected. A good example for that is secret_key which should be same on the Webserver and Worker to allow > Webserver to fetch logs from Worker.
##########
File path: docs/apache-airflow/executor/kubernetes.rst
##########
@@ -91,6 +91,14 @@ Example pod templates
With these requirements in mind, here are some examples of basic ``pod_template_file`` YAML files.
+.. note::
+
Review comment:
> I'd be in favor of stating these are for inspiration and that care should be taken to ensure all configuration is consistent across components in your deployment.
This is precisely the first thing you see when you follow the "config" link: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html
> Use the same configuration across all the Airflow components.
> While each component does not require all, some configurations need to be same otherwise they would not work
> as expected. A good example for that is secret_key which should be same on the Webserver and Worker to allow
> Webserver to fetch logs from Worker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org