You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/07 22:07:45 UTC
[GitHub] [airflow] vikramkoka opened a new issue #12897: upgrade-check: Unnecessary recommendation to create pod_template_file?
vikramkoka opened a new issue #12897:
URL: https://github.com/apache/airflow/issues/12897
On my Airflow 1.10.14rc1 deployment, when I run the airflow upgrade_check (initial version), it reports the following problems, despite the fact that none of my DAGs use the Kubernetes Executor nor the KubePodOperator.
It seems an unnecessary burden on the user to deal with these changes. Is it strictly necessary? Or, can this be eliminated?
Users must delete deprecated configs for KubernetesExecutor
-----------------------------------------------------------
In Airflow 2.0, KubernetesExecutor Users need to set a pod_template_file as a base
value for all pods launched by the KubernetesExecutor. Many Kubernetes configs are no longer
needed once this pod_template_file has been generated.
Problems:
1. The following invalid keys were found in your airflow.cfg:
airflow_configmap
airflow_local_settings_configmap
dags_in_image
dags_volume_subpath
dags_volume_mount_point
dags_volume_claim
logs_volume_subpath
logs_volume_claim
dags_volume_host
logs_volume_host
env_from_configmap_ref
env_from_secret_ref
git_repo
git_branch
git_sync_depth
git_subpath
git_sync_rev
git_user
git_password
git_sync_root
git_sync_dest
git_dags_folder_mount_point
git_ssh_key_secret_name
git_ssh_known_hosts_configmap_name
git_sync_credentials_secret
git_sync_container_repository
git_sync_container_tag
git_sync_init_container_name
git_sync_run_as_user
worker_service_account_name
image_pull_secrets
gcp_service_account_keys
affinity
tolerations
run_as_user
fs_group
Please generate a pod_template_file by running `airflow generate_pod_template` and delete these keys.
Users must set a kubernetes.pod_template_file value
---------------------------------------------------
In Airflow 2.0, KubernetesExecutor Users need to set a pod_template_file as a base
value for all pods launched by the KubernetesExecutor
Problems:
1. Please create a pod_template_file by running `airflow generate_pod_template`.
This will generate a pod using your aiflow.cfg settings
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vikramkoka commented on issue #12897: upgrade-check: Unnecessary recommendation to create pod_template_file?
Posted by GitBox <gi...@apache.org>.
vikramkoka commented on issue #12897:
URL: https://github.com/apache/airflow/issues/12897#issuecomment-741991134
True, that's a good point.
The intent here is to avoid overwhelming users with things that they don't need to do in order to perform the upgrade, because those features are not relevant to them.
Do you think there is a better option?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vikramkoka commented on issue #12897: upgrade-check: Unnecessary recommendation to create pod_template_file?
Posted by GitBox <gi...@apache.org>.
vikramkoka commented on issue #12897:
URL: https://github.com/apache/airflow/issues/12897#issuecomment-740800488
> @vikramkoka so maybe we should check if the user is using the KubernetesExecutor before making this recommendation?
Yes please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #12897: upgrade-check: Unnecessary recommendation to create pod_template_file?
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #12897:
URL: https://github.com/apache/airflow/issues/12897#issuecomment-741087515
> @vikramkoka so maybe we should check if the user is using the KubernetesExecutor before making this recommendation?
I agree. But we have to remember that users checking DAGs locally may not use this executor, or may not even have production config locally, so it's not bullet proof method.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #12897: upgrade-check: Unnecessary recommendation to create pod_template_file?
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #12897:
URL: https://github.com/apache/airflow/issues/12897#issuecomment-740210766
@vikramkoka so maybe we should check if the user is using the KubernetesExecutor before making this recommendation?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #12897: upgrade-check: Unnecessary recommendation to create pod_template_file?
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #12897:
URL: https://github.com/apache/airflow/issues/12897#issuecomment-742125054
@vikramkoka @dimberman should we introduce SKIPPED state for checks? For example, we can introduce additional validation if check applies and if not then mark it as skipped with some small information why. In this way we can use checking executor but if there will be mismatch between prod/dev environment we will left a hint why a check was skipped.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vikramkoka commented on issue #12897: upgrade-check: Unnecessary recommendation to create pod_template_file?
Posted by GitBox <gi...@apache.org>.
vikramkoka commented on issue #12897:
URL: https://github.com/apache/airflow/issues/12897#issuecomment-742132717
@turbaszek I think that [SKIPPED] is a really good idea. It addresses the concern of missing checks because of missing clues from the examined environment or because of "ignore" settings.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #12897: upgrade-check: Unnecessary recommendation to create pod_template_file?
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #12897:
URL: https://github.com/apache/airflow/issues/12897#issuecomment-742850465
Closed via #12981
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek closed issue #12897: upgrade-check: Unnecessary recommendation to create pod_template_file?
Posted by GitBox <gi...@apache.org>.
turbaszek closed issue #12897:
URL: https://github.com/apache/airflow/issues/12897
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org