You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/12/22 18:50:44 UTC
[GitHub] [airflow] potiuk opened a new pull request, #28541: Speed up provider validation pre-commit
potiuk opened a new pull request, #28541:
URL: https://github.com/apache/airflow/pull/28541
The provider validation pre-commit is now a lot slower after #28516 but it turned out that it has been also doing a little too much.
It worked in the way (against the original design) that when any provider.yaml changed, validation was actually performed for all the providers, not only for those changed.
This change improves the regular speed of running this validation when one or few providers are changed in the commit.
It's not only limiting the validation to those provider.yaml files that changes but also performing subset of validations on those changed files if not all providers are changed or --all-files are not used (because some of the validations involving documentation and cross-provider dependencies require all provider.yaml files to be loaded and validated).
The validation is still done in one "docker run" command though.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #28541: Speed up provider validation pre-commit
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #28541:
URL: https://github.com/apache/airflow/pull/28541
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #28541: Speed up provider validation pre-commit
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #28541:
URL: https://github.com/apache/airflow/pull/28541#issuecomment-1363376218
All green.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #28541: Speed up provider validation pre-commit
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #28541:
URL: https://github.com/apache/airflow/pull/28541#issuecomment-1366770329
Anyone needs some speed-ups :) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #28541: Speed up provider validation pre-commit
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #28541:
URL: https://github.com/apache/airflow/pull/28541#discussion_r1056098444
##########
scripts/ci/pre_commit/pre_commit_check_provider_yaml_files.py:
##########
@@ -38,6 +39,9 @@
from airflow_breeze.utils.docker_command_utils import get_extra_docker_flags
from airflow_breeze.utils.run_utils import get_ci_image_for_pre_commits, run_command
+ cmd = "python3 /opt/airflow/scripts/in_container/run_provider_yaml_files_check.py"
+ if len(sys.argv) > 1:
+ cmd = cmd + " " + " ".join([shlex.quote(f) for f in sys.argv[1:]])
Review Comment:
Right 🤦
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #28541: Speed up provider validation pre-commit
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #28541:
URL: https://github.com/apache/airflow/pull/28541#discussion_r1056101696
##########
scripts/ci/pre_commit/pre_commit_check_provider_yaml_files.py:
##########
@@ -38,6 +39,9 @@
from airflow_breeze.utils.docker_command_utils import get_extra_docker_flags
from airflow_breeze.utils.run_utils import get_ci_image_for_pre_commits, run_command
+ cmd = "python3 /opt/airflow/scripts/in_container/run_provider_yaml_files_check.py"
+ if len(sys.argv) > 1:
+ cmd = cmd + " " + " ".join([shlex.quote(f) for f in sys.argv[1:]])
Review Comment:
Ah no... shlex.join is only available in 3.8 (thank yoy mypy for reminding me).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #28541: Speed up provider validation pre-commit
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #28541:
URL: https://github.com/apache/airflow/pull/28541#discussion_r1055922836
##########
scripts/ci/pre_commit/pre_commit_check_provider_yaml_files.py:
##########
@@ -38,6 +39,9 @@
from airflow_breeze.utils.docker_command_utils import get_extra_docker_flags
from airflow_breeze.utils.run_utils import get_ci_image_for_pre_commits, run_command
+ cmd = "python3 /opt/airflow/scripts/in_container/run_provider_yaml_files_check.py"
+ if len(sys.argv) > 1:
+ cmd = cmd + " " + " ".join([shlex.quote(f) for f in sys.argv[1:]])
Review Comment:
Better to use shlex.join
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org