You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/22 17:23:39 UTC
[GitHub] [airflow] szn opened a new pull request #11748: Checks if all the libraries in setup.py are listed in installation.rst file
szn opened a new pull request #11748:
URL: https://github.com/apache/airflow/pull/11748
Checks if all the libraries in setup.py are listed in the installation.rst file
closes: #8088
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] szn commented on a change in pull request #11748: Checks if all the libraries in setup.py are listed in installation.rst file
Posted by GitBox <gi...@apache.org>.
szn commented on a change in pull request #11748:
URL: https://github.com/apache/airflow/pull/11748#discussion_r511612176
##########
File path: scripts/ci/pre_commit/pre_commit_check_setup_installation.rst.py
##########
@@ -0,0 +1,101 @@
+#!/usr/bin/env python
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Checks if all the libraries in setup.py are listed in installation.rst file
+"""
+
+import os
+import re
+import sys
+from os.path import dirname
+
+AIRFLOW_SOURCES_DIR = os.path.join(dirname(__file__), os.pardir, os.pardir, os.pardir)
+SETUP_PY_FILE = 'setup.py'
+DOCS_FILE = 'installation.rst'
+PY_IDENTIFIER = '[a-zA-Z_][a-zA-Z0-9_\.]*'
+
+def get_file_content(*file_path: str) -> str:
+ file_path = os.path.join(AIRFLOW_SOURCES_DIR, *file_path)
+ with open(file_path) as file_to_read:
+ return file_to_read.read()
+
+def get_extras_from_setup() -> {str: bool}:
+ """
+ Returns an array of extras from setup.py file in format:
+ {'package name': is_deprecated, ...]
+ """
+ setup_content = get_file_content(SETUP_PY_FILE)
+
+ extras_section_regex = re.compile(
+ '^EXTRAS_REQUIREMENTS: Dict[^{]+{([^}]+)}', re.MULTILINE)
+ extras_section = extras_section_regex.findall(setup_content)[0]
+
+ extras_regex = re.compile(
+ f'^\s+[\"\']([a-zA-Z_][a-zA-Z0-9_\.]*)[\"\']:\s*([a-zA-Z_][a-zA-Z0-9_\.]*)[^#\n]*(#\s*TODO.*)?$', re.MULTILINE)
+
+ extras_dict = {}
+ for extras in extras_regex.findall(extras_section):
+ package = extras[1]
+ alias = extras[0]
+ if not extras_dict.get(package):
+ extras_dict[package] = []
+ extras_dict[package].append(alias)
+ return extras_dict
+
+def get_extras_from_docs() -> {str: bool}:
+ """
+ Returns an array of extras from installation.rst file in format:
+ {'package name': is_deprecated, ...]
+ """
+ docs_content = get_file_content('docs', DOCS_FILE)
+
+ extras_section_regex = re.compile(f'^\|[^|]+\|.*pip install .apache-airflow\[({PY_IDENTIFIER})\].', re.MULTILINE)
+ return extras_section_regex.findall(docs_content)
+
+if __name__ == '__main__':
+ setup_packages = get_extras_from_setup()
+ docs_packages = get_extras_from_docs()
+
+ output_table = ""
+
+ for extras in sorted(setup_packages.keys()):
+ if not set(setup_packages[extras]).intersection(docs_packages):
+ output_table += "| {:20} | {:^10} | {:^10} |\n".format(extras, "V", "")
+
+ setup_packages_str = str(setup_packages)
+ for extras in sorted(docs_packages):
+ if not f"'{extras}'" in setup_packages_str:
Review comment:
I am not very proud of this assertion but it works, seems safe, and I was unable to find an efficient way of flattening a multidimensional array of strings.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11748: Checks if all the libraries in setup.py are listed in installation.rst file
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11748:
URL: https://github.com/apache/airflow/pull/11748#issuecomment-716170395
[The Workflow run](https://github.com/apache/airflow/actions/runs/327524174) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11748: Checks if all the libraries in setup.py are listed in installation.rst file
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11748:
URL: https://github.com/apache/airflow/pull/11748#issuecomment-716456190
[The Workflow run](https://github.com/apache/airflow/actions/runs/328847779) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] szn commented on a change in pull request #11748: Checks if all the libraries in setup.py are listed in installation.rst file
Posted by GitBox <gi...@apache.org>.
szn commented on a change in pull request #11748:
URL: https://github.com/apache/airflow/pull/11748#discussion_r511611974
##########
File path: setup.py
##########
@@ -565,6 +565,8 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version
"apache.hive": hive,
"apache.kylin": kylin,
"apache.pinot": pinot,
+ "apache.presto": presto,
+ "apache.spark": spark,
Review comment:
My intention was not to touch the setup.py file. But these changes seemed logical and safe. Hope I am right.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11748: Checks if all the libraries in setup.py are listed in installation.rst file
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11748:
URL: https://github.com/apache/airflow/pull/11748#issuecomment-716173618
Some static checks are failing :(
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] szn commented on a change in pull request #11748: Checks if all the libraries in setup.py are listed in installation.rst file
Posted by GitBox <gi...@apache.org>.
szn commented on a change in pull request #11748:
URL: https://github.com/apache/airflow/pull/11748#discussion_r511612624
##########
File path: scripts/ci/pre_commit/pre_commit_check_setup_installation.rst.py
##########
@@ -0,0 +1,101 @@
+#!/usr/bin/env python
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Checks if all the libraries in setup.py are listed in installation.rst file
+"""
+
+import os
+import re
+import sys
+from os.path import dirname
+
+AIRFLOW_SOURCES_DIR = os.path.join(dirname(__file__), os.pardir, os.pardir, os.pardir)
+SETUP_PY_FILE = 'setup.py'
+DOCS_FILE = 'installation.rst'
+PY_IDENTIFIER = '[a-zA-Z_][a-zA-Z0-9_\.]*'
+
+def get_file_content(*file_path: str) -> str:
+ file_path = os.path.join(AIRFLOW_SOURCES_DIR, *file_path)
+ with open(file_path) as file_to_read:
+ return file_to_read.read()
+
+def get_extras_from_setup() -> {str: bool}:
+ """
+ Returns an array EXTRAS_REQUIREMENTS with aliases from setup.py file in format:
+ {'package name': ['alias1', 'alias2], ...}
+ """
+ setup_content = get_file_content(SETUP_PY_FILE)
+
+ extras_section_regex = re.compile(
Review comment:
I was experimenting with `from setup import EXTRAS_REQUIREMENTS`, but this was I wasn't able to detect aliases.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #11748: Checks if all the libraries in setup.py are listed in installation.rst file
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #11748:
URL: https://github.com/apache/airflow/pull/11748#issuecomment-714642313
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, itโs a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better ๐.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11748: Checks if all the libraries in setup.py are listed in installation.rst file
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11748:
URL: https://github.com/apache/airflow/pull/11748#issuecomment-716049929
[The Workflow run](https://github.com/apache/airflow/actions/runs/326379487) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] szn commented on pull request #11748: Checks if all the libraries in setup.py are listed in installation.rst file
Posted by GitBox <gi...@apache.org>.
szn commented on pull request #11748:
URL: https://github.com/apache/airflow/pull/11748#issuecomment-716189110
> Some static checks are failing :(
Fixed except the failing check of newly introduced pre-commit-check (as described in **warning** section above).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11748: Checks if all the libraries in setup.py are listed in installation.rst file
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11748:
URL: https://github.com/apache/airflow/pull/11748#issuecomment-714727934
[The Workflow run](https://github.com/apache/airflow/actions/runs/322779431) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11748: Checks if all the libraries in setup.py are listed in installation.rst file
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11748:
URL: https://github.com/apache/airflow/pull/11748#issuecomment-716192799
[The Workflow run](https://github.com/apache/airflow/actions/runs/327700368) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org