You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "pankajkoti (via GitHub)" <gi...@apache.org> on 2023/08/23 02:37:39 UTC
[GitHub] [airflow] pankajkoti opened a new pull request, #33637: Use a trimmed version of README.md for PyPI
pankajkoti opened a new pull request, #33637:
URL: https://github.com/apache/airflow/pull/33637
closes: #33505
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33637:
URL: https://github.com/apache/airflow/pull/33637#issuecomment-1698635956
Really tiny NIT. in `generated/README.md` I explained how various files in that folder are generated. Would be great to add a paragrapgh descreiption on where it is from. When you look at that file, it's not obvious which commit generates it and how, so adding a short explanation will aid future contributors (and even future selves) to know where the hell that fille is from
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33637:
URL: https://github.com/apache/airflow/pull/33637#issuecomment-1700561303
> Am I missing something obvious or there's more to do π€
Yep. I think you found the fix in the last commit :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #33637:
URL: https://github.com/apache/airflow/pull/33637#discussion_r1309878809
##########
setup.cfg:
##########
@@ -22,7 +22,7 @@ author = Apache Software Foundation
author_email = dev@airflow.apache.org
url = https://airflow.apache.org/
version = attr: airflow.__version__
-long_description = file: README.md
+long_description = file: generated/PYPI_README.md
Review Comment:
You can build the package with `breeze release-management prepare-airflow-package --package-format sdist` and extract it to check if itβs in the package.
##########
setup.cfg:
##########
@@ -22,7 +22,7 @@ author = Apache Software Foundation
author_email = dev@airflow.apache.org
url = https://airflow.apache.org/
version = attr: airflow.__version__
-long_description = file: README.md
+long_description = file: generated/PYPI_README.md
Review Comment:
You can build the package with `breeze release-management prepare-airflow-package --package-format sdist` and extract it to check if itβs in the package.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on PR #33637:
URL: https://github.com/apache/airflow/pull/33637#issuecomment-1701914689
> Well. We need to read them in setup.py using nothing by stdlib of python in all versions we support. Otherwise I'd just use YAML.
We just need wait a bit (3 years π ) for min Python 3.11 and we could use tomllib
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajkoti commented on a diff in pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on code in PR #33637:
URL: https://github.com/apache/airflow/pull/33637#discussion_r1310522414
##########
scripts/ci/pre_commit/pre_commit_generate_pypi_readme.py:
##########
@@ -0,0 +1,84 @@
+#!/usr/bin/env python
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import re
+import sys
+from pathlib import Path
+
+from rich.console import Console
+
+console = Console(width=400, color_system="standard")
+
+AIRFLOW_SOURCES = Path(__file__).parents[3].resolve()
+README_SECTIONS_TO_EXTRACT = [
+ "Apache Airflow",
+ "Requirements",
+ "Getting started",
+ "Installing from PyPI",
+ "Official source code",
+ "Contributing",
+ "Who uses Apache Airflow",
+ "Who maintains Apache Airflow",
+]
+
+PYPI_README_HEADER = (
+ "<!-- "
+ "PLEASE DO NOT MODIFY THIS FILE. IT HAS BEEN GENERATED AUTOMATICALLY FROM THE `README.md` FILE OF THE\n"
+ "PROJECT BY THE `generate-pypi-readme` PRE-COMMIT. YOUR CHANGES HERE WILL BE AUTOMATICALLY OVERWRITTEN."
+ "-->\n"
+)
+
+# Function to extract sections based on start and end comments
+def extract_section(content, section_name):
+ start_comment = (
+ f"<!-- START {section_name}, please keep comment here to allow auto update of PyPI readme.md -->"
+ )
+ end_comment = (
+ f"<!-- END {section_name}, please keep comment here to allow auto update of PyPI readme.md -->"
+ )
+ section_match = re.search(
+ rf"{re.escape(start_comment)}(.*?)\n{re.escape(end_comment)}", content, re.DOTALL
+ )
+ if section_match:
+ return section_match.group(1)
+ else:
+ raise Exception(f"Cannot find section {section_name} in README.md")
+
+
+if __name__ == "__main__":
+ readme_file = AIRFLOW_SOURCES / "README.md"
+ pypi_readme_file = AIRFLOW_SOURCES / "generated" / "PYPI_README.md"
+
+ if not pypi_readme_file.exists():
+ pypi_readme_content = ""
+ else:
+ pypi_readme_content = pypi_readme_file.read_text()
+ with readme_file.open("r") as readme:
+ readme_content = readme.read()
+ generated_pypi_readme_content = PYPI_README_HEADER
+ for section in README_SECTIONS_TO_EXTRACT:
+ section_content = extract_section(readme_content, section)
+ generated_pypi_readme_content += section_content
+ if pypi_readme_content != generated_pypi_readme_content:
+ with pypi_readme_file.open("w") as generated_readme:
+ generated_readme.write(generated_pypi_readme_content)
+ console.print("\n[yellow]PyPI README.md content regenerated after update..\n")
+ console.print("[red]Aborting the commit")
+ console.print("You should check the changes made. Then simply 'git add --update .' and re-commit")
+ sys.exit(1)
Review Comment:
yes, this works very well. Very clean, thank you :)
Yes, I had all the ugly code earlier and the rich dependency for the console output & failing the commit with sys.exit which is no longer needed π
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #33637:
URL: https://github.com/apache/airflow/pull/33637#discussion_r1309882203
##########
setup.cfg:
##########
@@ -22,7 +22,7 @@ author = Apache Software Foundation
author_email = dev@airflow.apache.org
url = https://airflow.apache.org/
version = attr: airflow.__version__
-long_description = file: README.md
+long_description = file: generated/PYPI_README.md
Review Comment:
Yes. And "both" to generate both packages and you can also check if it is in METADATA of teh .whl package
##########
setup.cfg:
##########
@@ -22,7 +22,7 @@ author = Apache Software Foundation
author_email = dev@airflow.apache.org
url = https://airflow.apache.org/
version = attr: airflow.__version__
-long_description = file: README.md
+long_description = file: generated/PYPI_README.md
Review Comment:
Yes. And "both" to generate both packages and you can also check if it is in METADATA of the .whl package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33637:
URL: https://github.com/apache/airflow/pull/33637#issuecomment-1698646085
BTW. I REALLY wanted to add such comment in the .json files - but unfortunately, json specification does not allow comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajkoti commented on pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on PR #33637:
URL: https://github.com/apache/airflow/pull/33637#issuecomment-1702455775
I added the license back to the generated file as the the exclude is not working.
Static checks seems to run fine now.
One failing test of [MSSQL2017-latest, Py3.8: Core Providers[-amazon,google] Other Providers[amazon] WWW API Always CLI Providers[google]](https://github.com/apache/airflow/actions/runs/6046265544/job/16408363681?pr=33637#logs) seems to be unrelated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] phanikumv commented on pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "phanikumv (via GitHub)" <gi...@apache.org>.
phanikumv commented on PR #33637:
URL: https://github.com/apache/airflow/pull/33637#issuecomment-1702469236
> I added the license back to the generated file as the the exclude is not working. Static checks seems to run fine now.
>
> One failing test of [MSSQL2017-latest, Py3.8: Core Providers[-amazon,google] Other Providers[amazon] WWW API Always CLI Providers[google]](https://github.com/apache/airflow/actions/runs/6046265544/job/16408363681?pr=33637#logs) seems to be unrelated.
just re-triggered the failed test, lets see if its a flaky one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #33637:
URL: https://github.com/apache/airflow/pull/33637
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33637:
URL: https://github.com/apache/airflow/pull/33637#issuecomment-1701851268
> > BTW. I REALLY wanted to add such comment in the .json files - but unfortunately, json specification does not allow comments (hence the README.md).
>
> @potiuk maybe it is it possible to switch these files to .jsonc, "JSON with Comments"? VSCode supports it for example (https://code.visualstudio.com/docs/languages/json#_json-with-comments), I am not sure if it is possible to easily include support in this project.
Well. We need to read them in setup.py using nothing by stdlib of python in all versions we support. Otherwise I'd just use YAML. The file is read by `python setup.py` that should work without being in any specifc venv, no libraries installed, no IDEs or anything. Nothing beats simplicity of `import json; provider_dependencies_dict =json.loads( Path(__name__.parent) / "generated" /"provider_dependencies.json").text)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajkoti commented on pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on PR #33637:
URL: https://github.com/apache/airflow/pull/33637#issuecomment-1699142821
Static check is failing with the below error:
```
Check if licenses are OK for Apache................................................Failed
- hook id: check-apache-license-rat
- exit code: 1
Running command:
docker run -v /home/runner/work/airflow/airflow:/opt/airflow -t --user 1001:999 --rm --platform linux/amd64 ghcr.io/apache/airflow-apache-rat:0.13-2021.07.04 -d /opt/airflow --exclude-file /opt/airflow/.rat-excludes
ERROR: Could not find Apache licences in some files:
!????? /opt/airflow/generated/PYPI_README.md
```
I tried adding `generated/PYPI_README.md` (this is already committed and fails in this PR checks), `./generated/PYPI_README.md`, `/opt/airflow/generated/PYPI_README.md ` to the `.rat-excludes` file, also tried to rebuild the image, but the error persists when I run `breeze static-checks --type check-apache-license-rat --all-files --force-build` locally.
Am I missing something obvious or there's more to do π€
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a diff in pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #33637:
URL: https://github.com/apache/airflow/pull/33637#discussion_r1310397895
##########
setup.cfg:
##########
@@ -43,11 +43,12 @@ classifiers =
Topic :: System :: Monitoring
Framework :: Apache Airflow
project_urls =
+ Bug Tracker=https://github.com/apache/airflow/issues
+ Changelog=https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html
Review Comment:
```suggestion
Release Notes=https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html
```
And maybe reorder it as well...
##########
scripts/ci/pre_commit/pre_commit_generate_pypi_readme.py:
##########
@@ -0,0 +1,84 @@
+#!/usr/bin/env python
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import re
+import sys
+from pathlib import Path
+
+from rich.console import Console
+
+console = Console(width=400, color_system="standard")
+
+AIRFLOW_SOURCES = Path(__file__).parents[3].resolve()
+README_SECTIONS_TO_EXTRACT = [
+ "Apache Airflow",
+ "Requirements",
+ "Getting started",
+ "Installing from PyPI",
+ "Official source code",
+ "Contributing",
+ "Who uses Apache Airflow",
+ "Who maintains Apache Airflow",
+]
+
+PYPI_README_HEADER = (
+ "<!-- "
+ "PLEASE DO NOT MODIFY THIS FILE. IT HAS BEEN GENERATED AUTOMATICALLY FROM THE `README.md` FILE OF THE\n"
+ "PROJECT BY THE `generate-pypi-readme` PRE-COMMIT. YOUR CHANGES HERE WILL BE AUTOMATICALLY OVERWRITTEN."
+ "-->\n"
+)
+
+# Function to extract sections based on start and end comments
+def extract_section(content, section_name):
+ start_comment = (
+ f"<!-- START {section_name}, please keep comment here to allow auto update of PyPI readme.md -->"
+ )
+ end_comment = (
+ f"<!-- END {section_name}, please keep comment here to allow auto update of PyPI readme.md -->"
+ )
+ section_match = re.search(
+ rf"{re.escape(start_comment)}(.*?)\n{re.escape(end_comment)}", content, re.DOTALL
+ )
+ if section_match:
+ return section_match.group(1)
+ else:
+ raise Exception(f"Cannot find section {section_name} in README.md")
+
+
+if __name__ == "__main__":
+ readme_file = AIRFLOW_SOURCES / "README.md"
+ pypi_readme_file = AIRFLOW_SOURCES / "generated" / "PYPI_README.md"
+
+ if not pypi_readme_file.exists():
+ pypi_readme_content = ""
+ else:
+ pypi_readme_content = pypi_readme_file.read_text()
+ with readme_file.open("r") as readme:
+ readme_content = readme.read()
+ generated_pypi_readme_content = PYPI_README_HEADER
+ for section in README_SECTIONS_TO_EXTRACT:
+ section_content = extract_section(readme_content, section)
+ generated_pypi_readme_content += section_content
+ if pypi_readme_content != generated_pypi_readme_content:
+ with pypi_readme_file.open("w") as generated_readme:
+ generated_readme.write(generated_pypi_readme_content)
+ console.print("\n[yellow]PyPI README.md content regenerated after update..\n")
+ console.print("[red]Aborting the commit")
+ console.print("You should check the changes made. Then simply 'git add --update .' and re-commit")
+ sys.exit(1)
Review Comment:
How about we just write the file every time? That simplifies this code a bit, and still shows a good enough message to the user:
```
Generate PyPI README...............................................................Failed
- hook id: generate-pypi-readme
- files were modified by this hook
```
```suggestion
readme_content = readme_file.read_text()
generated_pypi_readme_content = PYPI_README_HEADER
for section in README_SECTIONS_TO_EXTRACT:
section_content = extract_section(readme_content, section)
generated_pypi_readme_content += section_content
with pypi_readme_file.open("w") as generated_readme:
generated_readme.write(generated_pypi_readme_content)
```
(If not, I have a number of comments on the original code, so lmk π)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a diff in pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #33637:
URL: https://github.com/apache/airflow/pull/33637#discussion_r1310432046
##########
scripts/ci/pre_commit/pre_commit_generate_pypi_readme.py:
##########
@@ -0,0 +1,84 @@
+#!/usr/bin/env python
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import re
+import sys
+from pathlib import Path
+
+from rich.console import Console
+
+console = Console(width=400, color_system="standard")
+
+AIRFLOW_SOURCES = Path(__file__).parents[3].resolve()
+README_SECTIONS_TO_EXTRACT = [
+ "Apache Airflow",
+ "Requirements",
+ "Getting started",
+ "Installing from PyPI",
+ "Official source code",
+ "Contributing",
+ "Who uses Apache Airflow",
+ "Who maintains Apache Airflow",
+]
+
+PYPI_README_HEADER = (
+ "<!-- "
+ "PLEASE DO NOT MODIFY THIS FILE. IT HAS BEEN GENERATED AUTOMATICALLY FROM THE `README.md` FILE OF THE\n"
+ "PROJECT BY THE `generate-pypi-readme` PRE-COMMIT. YOUR CHANGES HERE WILL BE AUTOMATICALLY OVERWRITTEN."
+ "-->\n"
+)
+
+# Function to extract sections based on start and end comments
+def extract_section(content, section_name):
+ start_comment = (
+ f"<!-- START {section_name}, please keep comment here to allow auto update of PyPI readme.md -->"
+ )
+ end_comment = (
+ f"<!-- END {section_name}, please keep comment here to allow auto update of PyPI readme.md -->"
+ )
+ section_match = re.search(
+ rf"{re.escape(start_comment)}(.*?)\n{re.escape(end_comment)}", content, re.DOTALL
+ )
+ if section_match:
+ return section_match.group(1)
+ else:
+ raise Exception(f"Cannot find section {section_name} in README.md")
+
+
+if __name__ == "__main__":
+ readme_file = AIRFLOW_SOURCES / "README.md"
+ pypi_readme_file = AIRFLOW_SOURCES / "generated" / "PYPI_README.md"
+
+ if not pypi_readme_file.exists():
+ pypi_readme_content = ""
+ else:
+ pypi_readme_content = pypi_readme_file.read_text()
+ with readme_file.open("r") as readme:
+ readme_content = readme.read()
+ generated_pypi_readme_content = PYPI_README_HEADER
+ for section in README_SECTIONS_TO_EXTRACT:
+ section_content = extract_section(readme_content, section)
+ generated_pypi_readme_content += section_content
+ if pypi_readme_content != generated_pypi_readme_content:
+ with pypi_readme_file.open("w") as generated_readme:
+ generated_readme.write(generated_pypi_readme_content)
+ console.print("\n[yellow]PyPI README.md content regenerated after update..\n")
+ console.print("[red]Aborting the commit")
+ console.print("You should check the changes made. Then simply 'git add --update .' and re-commit")
+ sys.exit(1)
Review Comment:
How about we just write the file every time? That simplifies this code a bit, and still shows a good enough message to the user:
```
Generate PyPI README...............................................................Failed
- hook id: generate-pypi-readme
- files were modified by this hook
```
```suggestion
readme_content = readme_file.read_text()
generated_pypi_readme_content = PYPI_README_HEADER
for section in README_SECTIONS_TO_EXTRACT:
section_content = extract_section(readme_content, section)
generated_pypi_readme_content += section_content
with pypi_readme_file.open("w") as generated_readme:
generated_readme.write(generated_pypi_readme_content)
```
(If not, I have a number of comments on the original code, so lmk π)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #33637:
URL: https://github.com/apache/airflow/pull/33637#discussion_r1309882203
##########
setup.cfg:
##########
@@ -22,7 +22,7 @@ author = Apache Software Foundation
author_email = dev@airflow.apache.org
url = https://airflow.apache.org/
version = attr: airflow.__version__
-long_description = file: README.md
+long_description = file: generated/PYPI_README.md
Review Comment:
Yes. And use "both" to generate both packages and you can also check if it is in METADATA of the .whl package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33637:
URL: https://github.com/apache/airflow/pull/33637#issuecomment-1698644069
Super - Super NIT. It would also be great if the first line of the generated PYPIREADME.md contains something like that
```
<!-- PLEASE DO NOT MODIFY THIS FILE. IT HAS BEEN GENERATED AUTOMATICALY FROM THE `README.md` IN MAIN AIRFLOW REPO BY `....` PRE-COMMIT. YOUR CHANGES HERE WILL BE AUTOMATICALLY OVERWRITTEN -->
```
You can also safely remove the Licence header when you generate it (though this might be a bit complex so we can leave it). The ASF requirement is that licence headers are present in files that are human-prepared, it does not have to be added to files which are automatically generated from other files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #33637:
URL: https://github.com/apache/airflow/pull/33637#discussion_r1310637539
##########
scripts/ci/pre_commit/pre_commit_generate_pypi_readme.py:
##########
@@ -0,0 +1,84 @@
+#!/usr/bin/env python
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import re
+import sys
+from pathlib import Path
+
+from rich.console import Console
+
+console = Console(width=400, color_system="standard")
+
+AIRFLOW_SOURCES = Path(__file__).parents[3].resolve()
+README_SECTIONS_TO_EXTRACT = [
+ "Apache Airflow",
+ "Requirements",
+ "Getting started",
+ "Installing from PyPI",
+ "Official source code",
+ "Contributing",
+ "Who uses Apache Airflow",
+ "Who maintains Apache Airflow",
+]
+
+PYPI_README_HEADER = (
+ "<!-- "
+ "PLEASE DO NOT MODIFY THIS FILE. IT HAS BEEN GENERATED AUTOMATICALLY FROM THE `README.md` FILE OF THE\n"
+ "PROJECT BY THE `generate-pypi-readme` PRE-COMMIT. YOUR CHANGES HERE WILL BE AUTOMATICALLY OVERWRITTEN."
+ "-->\n"
+)
+
+# Function to extract sections based on start and end comments
+def extract_section(content, section_name):
+ start_comment = (
+ f"<!-- START {section_name}, please keep comment here to allow auto update of PyPI readme.md -->"
+ )
+ end_comment = (
+ f"<!-- END {section_name}, please keep comment here to allow auto update of PyPI readme.md -->"
+ )
+ section_match = re.search(
+ rf"{re.escape(start_comment)}(.*?)\n{re.escape(end_comment)}", content, re.DOTALL
+ )
+ if section_match:
+ return section_match.group(1)
+ else:
+ raise Exception(f"Cannot find section {section_name} in README.md")
+
+
+if __name__ == "__main__":
+ readme_file = AIRFLOW_SOURCES / "README.md"
+ pypi_readme_file = AIRFLOW_SOURCES / "generated" / "PYPI_README.md"
+
+ if not pypi_readme_file.exists():
+ pypi_readme_content = ""
+ else:
+ pypi_readme_content = pypi_readme_file.read_text()
+ with readme_file.open("r") as readme:
+ readme_content = readme.read()
+ generated_pypi_readme_content = PYPI_README_HEADER
+ for section in README_SECTIONS_TO_EXTRACT:
+ section_content = extract_section(readme_content, section)
+ generated_pypi_readme_content += section_content
+ if pypi_readme_content != generated_pypi_readme_content:
+ with pypi_readme_file.open("w") as generated_readme:
+ generated_readme.write(generated_pypi_readme_content)
+ console.print("\n[yellow]PyPI README.md content regenerated after update..\n")
+ console.print("[red]Aborting the commit")
+ console.print("You should check the changes made. Then simply 'git add --update .' and re-commit")
+ sys.exit(1)
Review Comment:
Yeah. Writing file every time is fine. Now. The only thing that's left is just to ... run the pre-commit to fix some discrepancies :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajkoti commented on pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on PR #33637:
URL: https://github.com/apache/airflow/pull/33637#issuecomment-1689784032
> I think we do NOT want to maintain yet another README. My vision of is that the README will be automatically extracted from the main README by pre-commit (so ideally it should be placed in `generated` dir and used from there,
>
> We could do it dynamically as well, but that would not fly with mostly declarative way how we want to build packages. So automatically generated file which is just truncated version of README.md commited in `generated` seems a much better approach.
yes, good idea, will rather work on the extraction via pre-commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33637:
URL: https://github.com/apache/airflow/pull/33637#issuecomment-1689764964
I think we do NOT want to maintain yet another README. My vision of is that the README will py automatically extracted from the main README by pre-commit (so ideally it should be placed in `generated` dir and used from there,
We could do it dynamically as well, but that would not fly with mostly declarative way how we want to build packages. So automatically generated file whic is just truncated version of README.md commited in `generated` seems a much better approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #33637:
URL: https://github.com/apache/airflow/pull/33637#discussion_r1311819935
##########
.rat-excludes:
##########
@@ -132,3 +132,6 @@ chart/*.iml
# Sha files
.*sha256
+
+# Generated PyPI README file
+generated/PYPI_README.md
Review Comment:
```suggestion
./generated/PYPI_README.md
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajkoti commented on a diff in pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on code in PR #33637:
URL: https://github.com/apache/airflow/pull/33637#discussion_r1311252198
##########
scripts/ci/pre_commit/pre_commit_generate_pypi_readme.py:
##########
@@ -0,0 +1,84 @@
+#!/usr/bin/env python
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import re
+import sys
+from pathlib import Path
+
+from rich.console import Console
+
+console = Console(width=400, color_system="standard")
+
+AIRFLOW_SOURCES = Path(__file__).parents[3].resolve()
+README_SECTIONS_TO_EXTRACT = [
+ "Apache Airflow",
+ "Requirements",
+ "Getting started",
+ "Installing from PyPI",
+ "Official source code",
+ "Contributing",
+ "Who uses Apache Airflow",
+ "Who maintains Apache Airflow",
+]
+
+PYPI_README_HEADER = (
+ "<!-- "
+ "PLEASE DO NOT MODIFY THIS FILE. IT HAS BEEN GENERATED AUTOMATICALLY FROM THE `README.md` FILE OF THE\n"
+ "PROJECT BY THE `generate-pypi-readme` PRE-COMMIT. YOUR CHANGES HERE WILL BE AUTOMATICALLY OVERWRITTEN."
+ "-->\n"
+)
+
+# Function to extract sections based on start and end comments
+def extract_section(content, section_name):
+ start_comment = (
+ f"<!-- START {section_name}, please keep comment here to allow auto update of PyPI readme.md -->"
+ )
+ end_comment = (
+ f"<!-- END {section_name}, please keep comment here to allow auto update of PyPI readme.md -->"
+ )
+ section_match = re.search(
+ rf"{re.escape(start_comment)}(.*?)\n{re.escape(end_comment)}", content, re.DOTALL
+ )
+ if section_match:
+ return section_match.group(1)
+ else:
+ raise Exception(f"Cannot find section {section_name} in README.md")
+
+
+if __name__ == "__main__":
+ readme_file = AIRFLOW_SOURCES / "README.md"
+ pypi_readme_file = AIRFLOW_SOURCES / "generated" / "PYPI_README.md"
+
+ if not pypi_readme_file.exists():
+ pypi_readme_content = ""
+ else:
+ pypi_readme_content = pypi_readme_file.read_text()
+ with readme_file.open("r") as readme:
+ readme_content = readme.read()
+ generated_pypi_readme_content = PYPI_README_HEADER
+ for section in README_SECTIONS_TO_EXTRACT:
+ section_content = extract_section(readme_content, section)
+ generated_pypi_readme_content += section_content
+ if pypi_readme_content != generated_pypi_readme_content:
+ with pypi_readme_file.open("w") as generated_readme:
+ generated_readme.write(generated_pypi_readme_content)
+ console.print("\n[yellow]PyPI README.md content regenerated after update..\n")
+ console.print("[red]Aborting the commit")
+ console.print("You should check the changes made. Then simply 'git add --update .' and re-commit")
+ sys.exit(1)
Review Comment:
I am still seeing static failures wrt https://github.com/apache/airflow/pull/33637#issuecomment-1699142821. Any suggestion for me that what we could do here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajkoti commented on a diff in pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on code in PR #33637:
URL: https://github.com/apache/airflow/pull/33637#discussion_r1310522414
##########
scripts/ci/pre_commit/pre_commit_generate_pypi_readme.py:
##########
@@ -0,0 +1,84 @@
+#!/usr/bin/env python
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import re
+import sys
+from pathlib import Path
+
+from rich.console import Console
+
+console = Console(width=400, color_system="standard")
+
+AIRFLOW_SOURCES = Path(__file__).parents[3].resolve()
+README_SECTIONS_TO_EXTRACT = [
+ "Apache Airflow",
+ "Requirements",
+ "Getting started",
+ "Installing from PyPI",
+ "Official source code",
+ "Contributing",
+ "Who uses Apache Airflow",
+ "Who maintains Apache Airflow",
+]
+
+PYPI_README_HEADER = (
+ "<!-- "
+ "PLEASE DO NOT MODIFY THIS FILE. IT HAS BEEN GENERATED AUTOMATICALLY FROM THE `README.md` FILE OF THE\n"
+ "PROJECT BY THE `generate-pypi-readme` PRE-COMMIT. YOUR CHANGES HERE WILL BE AUTOMATICALLY OVERWRITTEN."
+ "-->\n"
+)
+
+# Function to extract sections based on start and end comments
+def extract_section(content, section_name):
+ start_comment = (
+ f"<!-- START {section_name}, please keep comment here to allow auto update of PyPI readme.md -->"
+ )
+ end_comment = (
+ f"<!-- END {section_name}, please keep comment here to allow auto update of PyPI readme.md -->"
+ )
+ section_match = re.search(
+ rf"{re.escape(start_comment)}(.*?)\n{re.escape(end_comment)}", content, re.DOTALL
+ )
+ if section_match:
+ return section_match.group(1)
+ else:
+ raise Exception(f"Cannot find section {section_name} in README.md")
+
+
+if __name__ == "__main__":
+ readme_file = AIRFLOW_SOURCES / "README.md"
+ pypi_readme_file = AIRFLOW_SOURCES / "generated" / "PYPI_README.md"
+
+ if not pypi_readme_file.exists():
+ pypi_readme_content = ""
+ else:
+ pypi_readme_content = pypi_readme_file.read_text()
+ with readme_file.open("r") as readme:
+ readme_content = readme.read()
+ generated_pypi_readme_content = PYPI_README_HEADER
+ for section in README_SECTIONS_TO_EXTRACT:
+ section_content = extract_section(readme_content, section)
+ generated_pypi_readme_content += section_content
+ if pypi_readme_content != generated_pypi_readme_content:
+ with pypi_readme_file.open("w") as generated_readme:
+ generated_readme.write(generated_pypi_readme_content)
+ console.print("\n[yellow]PyPI README.md content regenerated after update..\n")
+ console.print("[red]Aborting the commit")
+ console.print("You should check the changes made. Then simply 'git add --update .' and re-commit")
+ sys.exit(1)
Review Comment:
yes, this works very well. Very clean, thank you :)
Yes, I had all the ugly code earlier and the rich dependency for the console output which is no longer needed π
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajkoti commented on pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on PR #33637:
URL: https://github.com/apache/airflow/pull/33637#issuecomment-1700595540
> Yep. I think you found the fix in the last commit :)
I am not sure. I did add the file to .rat-excludes, but it still fails, waiting for another run in the CI now, but I'm not hopeful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #33637:
URL: https://github.com/apache/airflow/pull/33637#discussion_r1311819935
##########
.rat-excludes:
##########
@@ -132,3 +132,6 @@ chart/*.iml
# Sha files
.*sha256
+
+# Generated PyPI README file
+generated/PYPI_README.md
Review Comment:
```suggestion
./generated/*
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexbegg commented on pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "alexbegg (via GitHub)" <gi...@apache.org>.
alexbegg commented on PR #33637:
URL: https://github.com/apache/airflow/pull/33637#issuecomment-1701665078
> BTW. I REALLY wanted to add such comment in the .json files - but unfortunately, json specification does not allow comments (hence the README.md).
@potiuk maybe it is it possible to switch these files to .jsonc, "JSON with Comments"? VSCode supports it for example (https://code.visualstudio.com/docs/languages/json#_json-with-comments), I am not sure if it is possible to easily include support in this project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajkoti commented on a diff in pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on code in PR #33637:
URL: https://github.com/apache/airflow/pull/33637#discussion_r1310522414
##########
scripts/ci/pre_commit/pre_commit_generate_pypi_readme.py:
##########
@@ -0,0 +1,84 @@
+#!/usr/bin/env python
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import re
+import sys
+from pathlib import Path
+
+from rich.console import Console
+
+console = Console(width=400, color_system="standard")
+
+AIRFLOW_SOURCES = Path(__file__).parents[3].resolve()
+README_SECTIONS_TO_EXTRACT = [
+ "Apache Airflow",
+ "Requirements",
+ "Getting started",
+ "Installing from PyPI",
+ "Official source code",
+ "Contributing",
+ "Who uses Apache Airflow",
+ "Who maintains Apache Airflow",
+]
+
+PYPI_README_HEADER = (
+ "<!-- "
+ "PLEASE DO NOT MODIFY THIS FILE. IT HAS BEEN GENERATED AUTOMATICALLY FROM THE `README.md` FILE OF THE\n"
+ "PROJECT BY THE `generate-pypi-readme` PRE-COMMIT. YOUR CHANGES HERE WILL BE AUTOMATICALLY OVERWRITTEN."
+ "-->\n"
+)
+
+# Function to extract sections based on start and end comments
+def extract_section(content, section_name):
+ start_comment = (
+ f"<!-- START {section_name}, please keep comment here to allow auto update of PyPI readme.md -->"
+ )
+ end_comment = (
+ f"<!-- END {section_name}, please keep comment here to allow auto update of PyPI readme.md -->"
+ )
+ section_match = re.search(
+ rf"{re.escape(start_comment)}(.*?)\n{re.escape(end_comment)}", content, re.DOTALL
+ )
+ if section_match:
+ return section_match.group(1)
+ else:
+ raise Exception(f"Cannot find section {section_name} in README.md")
+
+
+if __name__ == "__main__":
+ readme_file = AIRFLOW_SOURCES / "README.md"
+ pypi_readme_file = AIRFLOW_SOURCES / "generated" / "PYPI_README.md"
+
+ if not pypi_readme_file.exists():
+ pypi_readme_content = ""
+ else:
+ pypi_readme_content = pypi_readme_file.read_text()
+ with readme_file.open("r") as readme:
+ readme_content = readme.read()
+ generated_pypi_readme_content = PYPI_README_HEADER
+ for section in README_SECTIONS_TO_EXTRACT:
+ section_content = extract_section(readme_content, section)
+ generated_pypi_readme_content += section_content
+ if pypi_readme_content != generated_pypi_readme_content:
+ with pypi_readme_file.open("w") as generated_readme:
+ generated_readme.write(generated_pypi_readme_content)
+ console.print("\n[yellow]PyPI README.md content regenerated after update..\n")
+ console.print("[red]Aborting the commit")
+ console.print("You should check the changes made. Then simply 'git add --update .' and re-commit")
+ sys.exit(1)
Review Comment:
yes, this works very well. Very clean, thank you :)
Yes, I had all the ugly code earlier and the rich dependency for the console output which is no longer needed π
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajkoti commented on a diff in pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on code in PR #33637:
URL: https://github.com/apache/airflow/pull/33637#discussion_r1309842843
##########
scripts/ci/pre_commit/pre_commit_generate_pypi_readme.py:
##########
@@ -0,0 +1,84 @@
+#!/usr/bin/env python
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import re
+import sys
+from pathlib import Path
+
+from rich.console import Console
+
+console = Console(width=400, color_system="standard")
+
+AIRFLOW_SOURCES = Path(__file__).parents[3].resolve()
+README_SECTIONS_TO_EXTRACT = [
+ "Apache Airflow",
+ "Requirements",
+ "Getting started",
+ "Installing from PyPI",
+ "Official source code",
+ "Contributing",
+ "Who uses Apache Airflow",
+ "Who maintains Apache Airflow",
+]
+
+PYPI_README_HEADER = (
+ "<!-- "
+ "PLEASE DO NOT MODIFY THIS FILE. IT HAS BEEN GENERATED AUTOMATICALLY FROM THE `README.md` FILE OF THE\n"
+ "PROJECT BY THE `generate-pypi-readme` PRE-COMMIT. YOUR CHANGES HERE WILL BE AUTOMATICALLY OVERWRITTEN."
+ "-->\n"
+)
+
+# Function to extract sections based on start and end comments
+def extract_section(content, section_name):
+ start_comment = (
+ f"<!-- START {section_name}, please keep comment here to allow auto update of PyPI readme.md -->"
+ )
+ end_comment = (
+ f"<!-- END {section_name}, please keep comment here to allow auto update of PyPI readme.md -->"
+ )
+ section_match = re.search(
+ rf"{re.escape(start_comment)}(.*?)\n{re.escape(end_comment)}", content, re.DOTALL
+ )
+ if section_match:
+ return section_match.group(1)
+ else:
+ raise Exception(f"Cannot find section {section_name} in README.md")
+
+
+if __name__ == "__main__":
+ readme_file = AIRFLOW_SOURCES / "README.md"
+ pypi_readme_file = AIRFLOW_SOURCES / "generated" / "PYPI_README.md"
+
+ if not pypi_readme_file.exists():
+ pypi_readme_content = PYPI_README_HEADER
Review Comment:
```suggestion
pypi_readme_content = ""
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajkoti commented on a diff in pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on code in PR #33637:
URL: https://github.com/apache/airflow/pull/33637#discussion_r1309851818
##########
setup.cfg:
##########
@@ -22,7 +22,7 @@ author = Apache Software Foundation
author_email = dev@airflow.apache.org
url = https://airflow.apache.org/
version = attr: airflow.__version__
-long_description = file: README.md
+long_description = file: generated/PYPI_README.md
Review Comment:
The generated folder is included in the `MANIFEST.in `https://github.com/apache/airflow/blob/main/MANIFEST.in#L42
So I am guessing this file will be available correctly and be used for the PyPI project listing description?
If someone can confirm my guess, would be great.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajkoti commented on a diff in pull request #33637: Use a trimmed version of README.md for PyPI
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on code in PR #33637:
URL: https://github.com/apache/airflow/pull/33637#discussion_r1309958647
##########
setup.cfg:
##########
@@ -22,7 +22,7 @@ author = Apache Software Foundation
author_email = dev@airflow.apache.org
url = https://airflow.apache.org/
version = attr: airflow.__version__
-long_description = file: README.md
+long_description = file: generated/PYPI_README.md
Review Comment:
Thank you.
Yes, confirmed that the file is included both in the `.tar.gz` and in the wheel's pkg description
<img width="857" alt="Screenshot 2023-08-30 at 2 44 28 PM" src="https://github.com/apache/airflow/assets/10206082/a833ded7-7a59-4cc4-883a-e8687fd8db33">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org