You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "amoghrajesh (via GitHub)" <gi...@apache.org> on 2023/09/01 11:30:43 UTC
[GitHub] [airflow] amoghrajesh opened a new pull request, #34004: Introducing short_version flag for building docs
amoghrajesh opened a new pull request, #34004:
URL: https://github.com/apache/airflow/pull/34004
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
Introducing short_version flag so that the users who build docs can now provide a short hand name instead of the whole name while building the docs for specific packages.
Some examples:
```
(new-env) ➜ airflow git:(shortenPackageFilter) ✗ breeze build-docs --help
Usage: breeze build-docs [OPTIONS]
Build documents.
╭─ Doc flags ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --docs-only -d Only build documentation. │
│ --spellcheck-only -s Only run spell checking. │
│ --clean-build Clean inventories of Inter-Sphinx documentation and generated APIs and sphinx artifacts before the build - useful for a clean build. │
│ --one-pass-only Builds documentation in one pass only. This is useful for debugging sphinx errors. │
│ --package-filter List of packages to consider. │
│ (apache-airflow | docker-stack | helm-chart | apache-airflow-providers-airbyte | apache-airflow-providers-alibaba | apache-airflow-providers-amazon | apache-airflow-providers-apache-beam | │
│ apache-airflow-providers-apache-cassandra | apache-airflow-providers-apache-drill | apache-airflow-providers-apache-druid | apache-airflow-providers-apache-flink | apache-airflow-providers-apache-hdfs | │
│ apache-airflow-providers-apache-hive | apache-airflow-providers-apache-impala | apache-airflow-providers-apache-kafka | apache-airflow-providers-apache-kylin | apache-airflow-providers-apache-livy | │
│ apache-airflow-providers-apache-pig | apache-airflow-providers-apache-pinot | apache-airflow-providers-apache-spark | apache-airflow-providers-apache-sqoop | apache-airflow-providers-apprise | │
│ apache-airflow-providers-arangodb | apache-airflow-providers-asana | apache-airflow-providers-atlassian-jira | apache-airflow-providers-celery | apache-airflow-providers-cloudant | │
│ apache-airflow-providers-cncf-kubernetes | apache-airflow-providers-common-sql | apache-airflow-providers-daskexecutor | apache-airflow-providers-databricks | apache-airflow-providers-datadog | │
│ apache-airflow-providers-dbt-cloud | apache-airflow-providers-dingding | apache-airflow-providers-discord | apache-airflow-providers-docker | apache-airflow-providers-elasticsearch | │
│ apache-airflow-providers-exasol | apache-airflow-providers-facebook | apache-airflow-providers-ftp | apache-airflow-providers-github | apache-airflow-providers-google | apache-airflow-providers-grpc | │
│ apache-airflow-providers-hashicorp | apache-airflow-providers-http | apache-airflow-providers-imap | apache-airflow-providers-influxdb | apache-airflow-providers-jdbc | apache-airflow-providers-jenkins | │
│ apache-airflow-providers-microsoft-azure | apache-airflow-providers-microsoft-mssql | apache-airflow-providers-microsoft-psrp | apache-airflow-providers-microsoft-winrm | apache-airflow-providers-mongo | │
│ apache-airflow-providers-mysql | apache-airflow-providers-neo4j | apache-airflow-providers-odbc | apache-airflow-providers-openfaas | apache-airflow-providers-openlineage | │
│ apache-airflow-providers-opsgenie | apache-airflow-providers-oracle | apache-airflow-providers-pagerduty | apache-airflow-providers-papermill | apache-airflow-providers-plexus | │
│ apache-airflow-providers-postgres | apache-airflow-providers-presto | apache-airflow-providers-redis | apache-airflow-providers-salesforce | apache-airflow-providers-samba | │
│ apache-airflow-providers-segment | apache-airflow-providers-sendgrid | apache-airflow-providers-sftp | apache-airflow-providers-singularity | apache-airflow-providers-slack | │
│ apache-airflow-providers-smtp | apache-airflow-providers-snowflake | apache-airflow-providers-sqlite | apache-airflow-providers-ssh | apache-airflow-providers-tableau | apache-airflow-providers-tabular | │
│ apache-airflow-providers-telegram | apache-airflow-providers-trino | apache-airflow-providers-vertica | apache-airflow-providers-yandex | apache-airflow-providers-zendesk) │
│ --github-repository -g GitHub repository used to pull, push run images. (TEXT) [default: apache/airflow] │
│ --builder Buildx builder used to perform `docker buildx build` commands. (TEXT) [default: autodetect] │
│ --short-version Build docs by providing short hand for provider names. Example: Short hand for apache-airflow-providers-cncf-kubernetes will be cncf.kubernetes │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Common options ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --verbose -v Print verbose information about performed steps. │
│ --dry-run -D If dry-run is set, commands are only printed, not executed. │
│ --help -h Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Selecting two packages - cncf kubernetes and airbyte using short hand
```
(new-env) ➜ airflow git:(shortenPackageFilter) ✗ breeze build-docs --short-version --package-filter cncf.kubernetes --package-filter airbyte
Good version of Docker: 24.0.2.
Good version of docker-compose: 2.18.1
The following important files are modified in /Users/adesai/Documents/OSS/airflow since last time image was built:
* setup.py
* setup.cfg
* Dockerfile.ci
* .dockerignore
* generated/provider_dependencies.json
* scripts/docker/common.sh
* scripts/docker/install_additional_dependencies.sh
* scripts/docker/install_airflow.sh
* scripts/docker/install_airflow_dependencies_from_branch_tip.sh
* scripts/docker/install_from_docker_context_files.sh
* scripts/docker/install_mysql.sh
Likely CI image needs rebuild
Do you want to build the image (this works best when you have good connection and can take usually from 20 seconds to few minutes depending how old your image is)?
Press y/N/q. Auto-select n in 10 seconds (add `--answer n` to avoid delay next time): n
The CI image for Python version 3.8 may be outdated
Please run at the earliest convenience:
breeze ci-image build --python 3.8
Running Initialization. Your basic configuration is:
* Airflow home: /root/airflow
* Airflow sources: /opt/airflow
* Airflow core SQL connection:
Using airflow version from current sources
Checking backend and integrations.
Skip fixing ownership of generated files as Host OS is darwin
#################### Available packages ####################
- apache-airflow
- apache-airflow-providers
- apache-airflow-providers-airbyte
- apache-airflow-providers-alibaba
- apache-airflow-providers-amazon
- apache-airflow-providers-apache-beam
- apache-airflow-providers-apache-cassandra
- apache-airflow-providers-apache-drill
- apache-airflow-providers-apache-druid
- apache-airflow-providers-apache-flink
- apache-airflow-providers-apache-hdfs
- apache-airflow-providers-apache-hive
- apache-airflow-providers-apache-impala
- apache-airflow-providers-apache-kafka
- apache-airflow-providers-apache-kylin
- apache-airflow-providers-apache-livy
- apache-airflow-providers-apache-pig
- apache-airflow-providers-apache-pinot
- apache-airflow-providers-apache-spark
- apache-airflow-providers-apache-sqoop
- apache-airflow-providers-apprise
- apache-airflow-providers-arangodb
- apache-airflow-providers-asana
- apache-airflow-providers-atlassian-jira
- apache-airflow-providers-celery
- apache-airflow-providers-cloudant
- apache-airflow-providers-cncf-kubernetes
- apache-airflow-providers-common-sql
- apache-airflow-providers-daskexecutor
- apache-airflow-providers-databricks
- apache-airflow-providers-datadog
- apache-airflow-providers-dbt-cloud
- apache-airflow-providers-dingding
- apache-airflow-providers-discord
- apache-airflow-providers-docker
- apache-airflow-providers-elasticsearch
- apache-airflow-providers-exasol
- apache-airflow-providers-facebook
- apache-airflow-providers-ftp
- apache-airflow-providers-github
- apache-airflow-providers-google
- apache-airflow-providers-grpc
- apache-airflow-providers-hashicorp
- apache-airflow-providers-http
- apache-airflow-providers-imap
- apache-airflow-providers-influxdb
- apache-airflow-providers-jdbc
- apache-airflow-providers-jenkins
- apache-airflow-providers-microsoft-azure
- apache-airflow-providers-microsoft-mssql
- apache-airflow-providers-microsoft-psrp
- apache-airflow-providers-microsoft-winrm
- apache-airflow-providers-mongo
- apache-airflow-providers-mysql
- apache-airflow-providers-neo4j
- apache-airflow-providers-odbc
- apache-airflow-providers-openfaas
- apache-airflow-providers-openlineage
- apache-airflow-providers-opsgenie
- apache-airflow-providers-oracle
- apache-airflow-providers-pagerduty
- apache-airflow-providers-papermill
- apache-airflow-providers-plexus
- apache-airflow-providers-postgres
- apache-airflow-providers-presto
- apache-airflow-providers-redis
- apache-airflow-providers-salesforce
- apache-airflow-providers-samba
- apache-airflow-providers-segment
- apache-airflow-providers-sendgrid
- apache-airflow-providers-sftp
- apache-airflow-providers-singularity
- apache-airflow-providers-slack
- apache-airflow-providers-smtp
- apache-airflow-providers-snowflake
- apache-airflow-providers-sqlite
- apache-airflow-providers-ssh
- apache-airflow-providers-tableau
- apache-airflow-providers-tabular
- apache-airflow-providers-telegram
- apache-airflow-providers-trino
- apache-airflow-providers-vertica
- apache-airflow-providers-yandex
- apache-airflow-providers-zendesk
- docker-stack
- helm-chart
Current package filters:
['apache-airflow-providers-cncf-kubernetes', 'apache-airflow-providers-airbyte']
#################### Fetching inventories ####################
Nothing to do
#################### Documentation will be built for 2 package(s) with 5 parallel jobs ####################
1. apache-airflow-providers-airbyte
2. apache-airflow-providers-cncf-kubernetes
#################### Cleaning documentation files ####################
apache-airflow-providers-airbyte : Cleaning files
apache-airflow-providers-cncf-kubernetes : Cleaning files
#################### Scheduling documentation to build ####################
apache-airflow-providers-airbyte : Scheduling documentation to build
apache-airflow-providers-cncf-kubernetes : Scheduling documentation to build
#################### Running docs building ####################
apache-airflow-providers-airbyte : Building documentation
apache-airflow-providers-cncf-kubernetes : Building documentation
apache-airflow-providers-airbyte : Running sphinx. The output is hidden until an error occurs.
apache-airflow-providers-cncf-kubernetes : Running sphinx. The output is hidden until an error occurs.
^CProcess ForkPoolWorker-4:
Process ForkPoolWorker-3:
Process ForkPoolWorker-5:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/opt/airflow/docs/build_docs.py", line 572, in <module>
main()
File "/opt/airflow/docs/build_docs.py", line 478, in main
package_build_errors, package_spelling_errors = build_docs_for_packages(
File "/opt/airflow/docs/build_docs.py", line 234, in build_docs_for_packages
run_in_parallel(
File "/opt/airflow/docs/build_docs.py", line 300, in run_in_parallel
run_docs_build_in_parallel(
File "/opt/airflow/docs/build_docs.py", line 345, in run_docs_build_in_parallel
result_list = pool.map(perform_docs_build_for_single_package, doc_build_specifications)
File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 765, in get
self.wait(timeout)
File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 762, in wait
self._event.wait(timeout)
File "/usr/local/lib/python3.8/threading.py", line 558, in wait
signaled = self._cond.wait(timeout)
File "/usr/local/lib/python3.8/threading.py", line 302, in wait
waiter.acquire()
KeyboardInterrupt
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/opt/airflow/airflow/settings.py", line 362, in dispose_orm
engine.dispose()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3076, in dispose
self.pool = self.pool.recreate()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 262, in recreate
self._creator,
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 219, in _creator
return self.__dict__["_creator"]
KeyboardInterrupt
Aborted.
```
closes: #33876
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #34004: Allowing short package names in build-docs breeze command
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #34004:
URL: https://github.com/apache/airflow/pull/34004#discussion_r1314233982
##########
dev/breeze/src/airflow_breeze/commands/developer_commands.py:
##########
@@ -357,6 +358,7 @@ def start_airflow(
@main.command(name="build-docs")
@click.option("-d", "--docs-only", help="Only build documentation.", is_flag=True)
@click.option("-s", "--spellcheck-only", help="Only run spell checking.", is_flag=True)
+@argument_packages_plus_all_providers
@option_builder
@click.option(
"--package-filter",
Review Comment:
I think we can change the type of the `--package-filter` to be just `string` instead of the list of package filter values available and instead explain in the help that you can use `apache-airflow-providers-<provider>` or `shorthand package id` or `glob pattern matching the full package name`. There is no need any more to do auto-complete for those or to display available values - since we have shorthand names which are auto-complete'able and we have the list of those in `--help`` output.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #34004: Allowing short package names in build-docs breeze command
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #34004:
URL: https://github.com/apache/airflow/pull/34004#issuecomment-1705639391
Random error. Merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] amoghrajesh commented on a diff in pull request #34004: Allowing short package names in build-docs breeze command
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #34004:
URL: https://github.com/apache/airflow/pull/34004#discussion_r1314561056
##########
dev/README_RELEASE_PROVIDER_PACKAGES.md:
##########
@@ -366,6 +374,14 @@ breeze build-docs --clean-build \
...
```
+You can also use shorthand names as arguments instead of using the full names
+for airflow providers. Example:
+
+```shell script
+cd "${AIRFLOW_REPO_ROOT}"
+breeze build-docs package1 package2 --clean-build
Review Comment:
> How does this work for building apache-airflow-providers ?
I do not think there is a special rule for this. Might need to add one? We can follow the instructions above for this too, no need of any args
>Not that now you can remove the build_provider_documentation.sh part from the guide. Is this script being used somewhere? If not I guess we can remove it
Yeah only here. I think we can safely remove it.
##########
dev/README_RELEASE_PROVIDER_PACKAGES.md:
##########
@@ -366,6 +374,14 @@ breeze build-docs --clean-build \
...
```
+You can also use shorthand names as arguments instead of using the full names
+for airflow providers. Example:
+
+```shell script
+cd "${AIRFLOW_REPO_ROOT}"
+breeze build-docs package1 package2 --clean-build
Review Comment:
> How does this work for building apache-airflow-providers ?
I do not think there is a special rule for this. Might need to add one? We can follow the instructions above for this too, no need of any args
>Not that now you can remove the build_provider_documentation.sh part from the guide. Is this script being used somewhere? If not I guess we can remove it
Yeah only here. I think we can safely remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] amoghrajesh commented on a diff in pull request #34004: Allowing short package names in build-docs breeze command
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #34004:
URL: https://github.com/apache/airflow/pull/34004#discussion_r1314561056
##########
dev/README_RELEASE_PROVIDER_PACKAGES.md:
##########
@@ -366,6 +374,14 @@ breeze build-docs --clean-build \
...
```
+You can also use shorthand names as arguments instead of using the full names
+for airflow providers. Example:
+
+```shell script
+cd "${AIRFLOW_REPO_ROOT}"
+breeze build-docs package1 package2 --clean-build
Review Comment:
> How does this work for building apache-airflow-providers ?
I do not think there is a special rule for this. Might need to add one? We can follow the instructions above for this too, no need of any args
>Not that now you can remove the build_provider_documentation.sh part from the guide. Is this script being used somewhere? If not I guess we can remove it
Yeah only here. I think we can safely remove it.
##########
dev/README_RELEASE_PROVIDER_PACKAGES.md:
##########
@@ -366,6 +374,14 @@ breeze build-docs --clean-build \
...
```
+You can also use shorthand names as arguments instead of using the full names
+for airflow providers. Example:
+
+```shell script
+cd "${AIRFLOW_REPO_ROOT}"
+breeze build-docs package1 package2 --clean-build
Review Comment:
> How does this work for building apache-airflow-providers ?
I do not think there is a special rule for this. Might need to add one? We can follow the instructions above for this too, no need of any args
Yeah only here. I think we can safely remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #34004: Allowing short package names in build-docs breeze command
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #34004:
URL: https://github.com/apache/airflow/pull/34004#discussion_r1315171503
##########
dev/README_RELEASE_PROVIDER_PACKAGES.md:
##########
@@ -366,6 +374,14 @@ breeze build-docs --clean-build \
...
```
+You can also use shorthand names as arguments instead of using the full names
+for airflow providers. Example:
+
+```shell script
+cd "${AIRFLOW_REPO_ROOT}"
+breeze build-docs package1 package2 --clean-build
Review Comment:
Yes. We should make it as part of the follow-up. (@eladkal - it's backwards compatible so old ways will continue to work).
This could be done together with the changes I described for the CI.
However - this is not THE RIGHT script/command :)
This is `build-docs` command and the script is `build_provider_docuemntation.sh`. There is yet ANOTHER change to make in similar way - i.e. same change in `publish-docs` command. And only then we will be able to replace `publish_provider_documentation.sh`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #34004: Allowing short package names in build-docs breeze command
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #34004:
URL: https://github.com/apache/airflow/pull/34004
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] amoghrajesh commented on pull request #34004: Allowing short package names in build-docs breeze command
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #34004:
URL: https://github.com/apache/airflow/pull/34004#issuecomment-1704687477
@potiuk Made the changes. Can you take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] amoghrajesh commented on a diff in pull request #34004: Allowing short package names in build-docs breeze command
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #34004:
URL: https://github.com/apache/airflow/pull/34004#discussion_r1314563088
##########
dev/README_RELEASE_PROVIDER_PACKAGES.md:
##########
@@ -366,6 +374,14 @@ breeze build-docs --clean-build \
...
```
+You can also use shorthand names as arguments instead of using the full names
+for airflow providers. Example:
+
+```shell script
+cd "${AIRFLOW_REPO_ROOT}"
+breeze build-docs package1 package2 --clean-build
Review Comment:
But there is an issue: the script runs the add back references command as a part of it.
```
breeze release-management add-back-references "${@}"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] amoghrajesh commented on pull request #34004: Allowing short package names in build-docs breeze command
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #34004:
URL: https://github.com/apache/airflow/pull/34004#issuecomment-1703870529
@eladkal @potiuk with the current changes, we have not retired the older `package-filter` at all.
One test I tried was to try a hybrid of it, since some users might not find it easy to migrate suddenly:
```
(new-env) ➜ airflow git:(shortenPackageFilter) ✗ breeze build-docs cncf.kubernetes sftp --package-filter apache-airflow-providers-airbyte
Good version of Docker: 24.0.2.
Good version of docker-compose: 2.18.1
The following important files are modified in /Users/adesai/Documents/OSS/airflow since last time image was built:
* setup.py
* setup.cfg
* Dockerfile.ci
* .dockerignore
* generated/provider_dependencies.json
* scripts/docker/common.sh
* scripts/docker/install_additional_dependencies.sh
* scripts/docker/install_airflow.sh
* scripts/docker/install_airflow_dependencies_from_branch_tip.sh
* scripts/docker/install_from_docker_context_files.sh
* scripts/docker/install_mysql.sh
Likely CI image needs rebuild
Do you want to build the image (this works best when you have good connection and can take usually from 20 seconds to few minutes depending how old your image is)?
Press y/N/q. Auto-select n in 10 seconds (add `--answer n` to avoid delay next time): n
The CI image for Python version 3.8 may be outdated
Please run at the earliest convenience:
breeze ci-image build --python 3.8
Running Initialization. Your basic configuration is:
* Airflow home: /root/airflow
* Airflow sources: /opt/airflow
* Airflow core SQL connection:
Using airflow version from current sources
Checking backend and integrations.
Skip fixing ownership of generated files as Host OS is darwin
#################### Available packages ####################
- apache-airflow
- apache-airflow-providers
- apache-airflow-providers-airbyte
- apache-airflow-providers-alibaba
- apache-airflow-providers-amazon
- apache-airflow-providers-apache-beam
- apache-airflow-providers-apache-cassandra
- apache-airflow-providers-apache-drill
- apache-airflow-providers-apache-druid
- apache-airflow-providers-apache-flink
- apache-airflow-providers-apache-hdfs
- apache-airflow-providers-apache-hive
- apache-airflow-providers-apache-impala
- apache-airflow-providers-apache-kafka
- apache-airflow-providers-apache-kylin
- apache-airflow-providers-apache-livy
- apache-airflow-providers-apache-pig
- apache-airflow-providers-apache-pinot
- apache-airflow-providers-apache-spark
- apache-airflow-providers-apache-sqoop
- apache-airflow-providers-apprise
- apache-airflow-providers-arangodb
- apache-airflow-providers-asana
- apache-airflow-providers-atlassian-jira
- apache-airflow-providers-celery
- apache-airflow-providers-cloudant
- apache-airflow-providers-cncf-kubernetes
- apache-airflow-providers-common-sql
- apache-airflow-providers-daskexecutor
- apache-airflow-providers-databricks
- apache-airflow-providers-datadog
- apache-airflow-providers-dbt-cloud
- apache-airflow-providers-dingding
- apache-airflow-providers-discord
- apache-airflow-providers-docker
- apache-airflow-providers-elasticsearch
- apache-airflow-providers-exasol
- apache-airflow-providers-facebook
- apache-airflow-providers-ftp
- apache-airflow-providers-github
- apache-airflow-providers-google
- apache-airflow-providers-grpc
- apache-airflow-providers-hashicorp
- apache-airflow-providers-http
- apache-airflow-providers-imap
- apache-airflow-providers-influxdb
- apache-airflow-providers-jdbc
- apache-airflow-providers-jenkins
- apache-airflow-providers-microsoft-azure
- apache-airflow-providers-microsoft-mssql
- apache-airflow-providers-microsoft-psrp
- apache-airflow-providers-microsoft-winrm
- apache-airflow-providers-mongo
- apache-airflow-providers-mysql
- apache-airflow-providers-neo4j
- apache-airflow-providers-odbc
- apache-airflow-providers-openfaas
- apache-airflow-providers-openlineage
- apache-airflow-providers-opsgenie
- apache-airflow-providers-oracle
- apache-airflow-providers-pagerduty
- apache-airflow-providers-papermill
- apache-airflow-providers-plexus
- apache-airflow-providers-postgres
- apache-airflow-providers-presto
- apache-airflow-providers-redis
- apache-airflow-providers-salesforce
- apache-airflow-providers-samba
- apache-airflow-providers-segment
- apache-airflow-providers-sendgrid
- apache-airflow-providers-sftp
- apache-airflow-providers-singularity
- apache-airflow-providers-slack
- apache-airflow-providers-smtp
- apache-airflow-providers-snowflake
- apache-airflow-providers-sqlite
- apache-airflow-providers-ssh
- apache-airflow-providers-tableau
- apache-airflow-providers-tabular
- apache-airflow-providers-telegram
- apache-airflow-providers-trino
- apache-airflow-providers-vertica
- apache-airflow-providers-yandex
- apache-airflow-providers-zendesk
- docker-stack
- helm-chart
Current package filters:
['apache-airflow-providers-cncf-kubernetes', 'apache-airflow-providers-sftp', 'apache-airflow-providers-airbyte']
#################### Fetching inventories ####################
Nothing to do
#################### Documentation will be built for 3 package(s) with 5 parallel jobs ####################
1. apache-airflow-providers-airbyte
2. apache-airflow-providers-cncf-kubernetes
3. apache-airflow-providers-sftp
#################### Cleaning documentation files ####################
apache-airflow-providers-airbyte : Cleaning files
apache-airflow-providers-cncf-kubernetes : Cleaning files
apache-airflow-providers-sftp : Cleaning files
#################### Scheduling documentation to build ####################
apache-airflow-providers-airbyte : Scheduling documentation to build
apache-airflow-providers-cncf-kubernetes : Scheduling documentation to build
apache-airflow-providers-sftp : Scheduling documentation to build
#################### Running docs building ####################
apache-airflow-providers-airbyte : Building documentation
apache-airflow-providers-cncf-kubernetes : Building documentation
apache-airflow-providers-sftp : Building documentation
apache-airflow-providers-sftp : Running sphinx. The output is hidden until an error occurs.
apache-airflow-providers-cncf-kubernetes : Running sphinx. The output is hidden until an error occurs.
apache-airflow-providers-airbyte : Running sphinx. The output is hidden until an error occurs.
^CProcess ForkPoolWorker-5:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 114, in worker
task = get()
File "/usr/local/lib/python3.8/multiprocessing/queues.py", line 355, in get
with self._rlock:
File "/usr/local/lib/python3.8/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
Process ForkPoolWorker-4:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 114, in worker
task = get()
File "/usr/local/lib/python3.8/multiprocessing/queues.py", line 356, in get
res = self._reader.recv_bytes()
File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
KeyboardInterrupt
Traceback (most recent call last):
File "/usr/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/opt/airflow/docs/build_docs.py", line 572, in <module>
main()
File "/opt/airflow/docs/build_docs.py", line 478, in main
package_build_errors, package_spelling_errors = build_docs_for_packages(
File "/opt/airflow/docs/build_docs.py", line 234, in build_docs_for_packages
run_in_parallel(
File "/opt/airflow/docs/build_docs.py", line 300, in run_in_parallel
run_docs_build_in_parallel(
File "/opt/airflow/docs/build_docs.py", line 345, in run_docs_build_in_parallel
result_list = pool.map(perform_docs_build_for_single_package, doc_build_specifications)
File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 765, in get
self.wait(timeout)
File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 762, in wait
self._event.wait(timeout)
File "/usr/local/lib/python3.8/threading.py", line 558, in wait
signaled = self._cond.wait(timeout)
File "/usr/local/lib/python3.8/threading.py", line 302, in wait
Aborted.
```
It takes combination of args + option from package-filter together!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #34004: Allowing short package names in build-docs breeze command
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #34004:
URL: https://github.com/apache/airflow/pull/34004#discussion_r1315197970
##########
dev/README_RELEASE_PROVIDER_PACKAGES.md:
##########
@@ -366,6 +374,14 @@ breeze build-docs --clean-build \
...
```
+You can also use shorthand names as arguments instead of using the full names
+for airflow providers. Example:
+
+```shell script
+cd "${AIRFLOW_REPO_ROOT}"
+breeze build-docs package1 package2 --clean-build
Review Comment:
> there is no issue... the script handled both doc build and add back referenced but when I didn't use the script I simply ran both commands. I actually prefer it that way. it gives better control over the process.
(just to clarify - this is a different script :) ... it was about publishing, not building :)
> What I meant is that now we don't need the build_provider_documentation.sh any more so it can be removed.
Correct. But I want to really do that in the follow-up when we also switch the CI to use it, to separte "breeze" changes with "using breeze" changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #34004: Allowing short package names in build-docs breeze command
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #34004:
URL: https://github.com/apache/airflow/pull/34004#issuecomment-1704191390
One small thing also to add: https://github.com/apache/airflow/blob/0d93cc5cab8cef56faf3be1705ec6784e2d8a74a/docs/build_docs.py#L92 - contains instructions displayed after docs are build and we should promote new arguments there, not ``--package-filter``.
There is also a follow-up after that one, but it should be a separate PR. We should adapt the current CI/selective checks to use the list of short package ids rather than `--package-filter` . Our 'selective checks` produce the list of packages to build in the form of `--package-filter` but - with this change it should not be needed any more - we could just use remove `docs-filter-list-as-string` output completely and pass the `affected-providers-list-as-string` to the `build-docs` commmand in CI. But this should be a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #34004: Allowing short package names in build-docs breeze command
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #34004:
URL: https://github.com/apache/airflow/pull/34004#discussion_r1315167232
##########
BREEZE.rst:
##########
@@ -478,6 +481,16 @@ These are all available flags of ``build-docs`` command:
:width: 100%
:alt: Breeze build documentation
+.. _generating_short_form_names:
+
+Generating short form names for Providers
+-----------------------------------------
+
+Skip the ``apache-airflow-providers-`` from the usual provider full names.
+Now with the remaining part, replace every ``dash("-")`` with a ``dot(".")``.
Review Comment:
It's just explanation for parmeter names in Breeze. Historicallly we use "apache-airflow-providers-cncf-kubernetes` and `--package-filter` to choose which packages to build. But we already use "short version" of those packages when we run some breeze command and this change is purely UX change so that you can specify them easier when you run command line. Instead of:
```
breeze build-docs --package-filter apache-airflow-providers-cncf-kubernetes --package-filter apache-airflow-providers-google --package-filter apache-airflow
```
you wil be able to specify:
```
breeze build-docs cncf.kubernetes google apache-airflow
```
This explanation is there to explain the user how to derive the short id from the long one used so far.
It's part of `build-docs` command description,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #34004: Allowing short package names in build-docs breeze command
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #34004:
URL: https://github.com/apache/airflow/pull/34004#discussion_r1315197970
##########
dev/README_RELEASE_PROVIDER_PACKAGES.md:
##########
@@ -366,6 +374,14 @@ breeze build-docs --clean-build \
...
```
+You can also use shorthand names as arguments instead of using the full names
+for airflow providers. Example:
+
+```shell script
+cd "${AIRFLOW_REPO_ROOT}"
+breeze build-docs package1 package2 --clean-build
Review Comment:
> there is no issue... the script handled both doc build and add back referenced but when I didn't use the script I simply ran both commands. I actually prefer it that way. it gives better control over the process.
(just to clarify - this is a different script :) ... it was about publishing, not building :)
> What I meant is that now we don't need the build_provider_documentation.sh any more so it can be removed.
Correct. But I want to really do that in the follow-up when we also switch the CI to use the new arguments rather than `--package-filter`, to separte "breeze" changes with "using breeze" changes.
##########
dev/README_RELEASE_PROVIDER_PACKAGES.md:
##########
@@ -366,6 +374,14 @@ breeze build-docs --clean-build \
...
```
+You can also use shorthand names as arguments instead of using the full names
+for airflow providers. Example:
+
+```shell script
+cd "${AIRFLOW_REPO_ROOT}"
+breeze build-docs package1 package2 --clean-build
Review Comment:
> there is no issue... the script handled both doc build and add back referenced but when I didn't use the script I simply ran both commands. I actually prefer it that way. it gives better control over the process.
(just to clarify - this is a different script :) ... it was about publishing, not building :)
> What I meant is that now we don't need the build_provider_documentation.sh any more so it can be removed.
Correct. But I want to really do that in the follow-up when we also switch the CI to use the new arguments rather than `--package-filter`, to separate "breeze" changes with "using breeze" changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] amoghrajesh commented on a diff in pull request #34004: Allowing short package names in build-docs breeze command
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #34004:
URL: https://github.com/apache/airflow/pull/34004#discussion_r1314484831
##########
dev/breeze/src/airflow_breeze/commands/developer_commands.py:
##########
@@ -357,6 +358,7 @@ def start_airflow(
@main.command(name="build-docs")
@click.option("-d", "--docs-only", help="Only build documentation.", is_flag=True)
@click.option("-s", "--spellcheck-only", help="Only run spell checking.", is_flag=True)
+@argument_packages_plus_all_providers
@option_builder
@click.option(
"--package-filter",
Review Comment:
Makes sense. Made the change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] amoghrajesh commented on a diff in pull request #34004: Allowing short package names in build-docs breeze command
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #34004:
URL: https://github.com/apache/airflow/pull/34004#discussion_r1314561054
##########
dev/README_RELEASE_PROVIDER_PACKAGES.md:
##########
@@ -366,6 +374,14 @@ breeze build-docs --clean-build \
...
```
+You can also use shorthand names as arguments instead of using the full names
+for airflow providers. Example:
+
+```shell script
+cd "${AIRFLOW_REPO_ROOT}"
+breeze build-docs package1 package2 --clean-build
Review Comment:
> How does this work for building apache-airflow-providers ?
I do not think there is a special rule for this. Might need to add one? We can follow the instructions above for this too, no need of any args
Yeah only here. I think we can safely remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on pull request #34004: Introducing short_version flag for building docs
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #34004:
URL: https://github.com/apache/airflow/pull/34004#issuecomment-1702692819
We need also to update README_RELEASE_PROVIDER_PACKAGES.md
in several places. For example:
https://github.com/apache/airflow/blob/ffc9854a81ce3195b2f3e8cdeb4ea90462e112f0/dev/README_RELEASE_PROVIDER_PACKAGES.md#L362-L366
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #34004: Allowing short package names in build-docs breeze command
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #34004:
URL: https://github.com/apache/airflow/pull/34004#discussion_r1314233982
##########
dev/breeze/src/airflow_breeze/commands/developer_commands.py:
##########
@@ -357,6 +358,7 @@ def start_airflow(
@main.command(name="build-docs")
@click.option("-d", "--docs-only", help="Only build documentation.", is_flag=True)
@click.option("-s", "--spellcheck-only", help="Only run spell checking.", is_flag=True)
+@argument_packages_plus_all_providers
@option_builder
@click.option(
"--package-filter",
Review Comment:
I think we can change the type of the "--package-filter" to be just 'string` and instead explain in the help that you can use `apache-airflow-providers-<provider>` or `shorthand package id` or `glob pattern matching the full package name`. There is no need any more to do auto-complete for those or to display available values - since we have shorthand names which are auto-complete'able and we have the list of those in `--help`` output.
##########
dev/breeze/src/airflow_breeze/commands/developer_commands.py:
##########
@@ -357,6 +358,7 @@ def start_airflow(
@main.command(name="build-docs")
@click.option("-d", "--docs-only", help="Only build documentation.", is_flag=True)
@click.option("-s", "--spellcheck-only", help="Only run spell checking.", is_flag=True)
+@argument_packages_plus_all_providers
@option_builder
@click.option(
"--package-filter",
Review Comment:
I think we can change the type of the "--package-filter" to be just `string` and instead explain in the help that you can use `apache-airflow-providers-<provider>` or `shorthand package id` or `glob pattern matching the full package name`. There is no need any more to do auto-complete for those or to display available values - since we have shorthand names which are auto-complete'able and we have the list of those in `--help`` output.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #34004: Allowing short package names in build-docs breeze command
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #34004:
URL: https://github.com/apache/airflow/pull/34004#discussion_r1315167232
##########
BREEZE.rst:
##########
@@ -478,6 +481,16 @@ These are all available flags of ``build-docs`` command:
:width: 100%
:alt: Breeze build documentation
+.. _generating_short_form_names:
+
+Generating short form names for Providers
+-----------------------------------------
+
+Skip the ``apache-airflow-providers-`` from the usual provider full names.
+Now with the remaining part, replace every ``dash("-")`` with a ``dot(".")``.
Review Comment:
It's just explanation for parmeter names in Breeze. Historicallly we use "apache-airflow-providers-cncf-kubernetes` and `--package-filter` to choose which packages to build. But we already use "short version" of those packages when we run some breeze command and this change is purely UX change so that you can specify them easier when you run command line. Instead of:
```
breeze build-docs --package-filter apache-airflow-providers-cncf-kubernetes --package-filter apache-airflow-providers-google
```
you wil be able to specify:
```
breeze build-docs cncf.kubernetes google
```
This explanation is there to explain the user how to derive the short id from the long one used so far.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #34004: Allowing short package names in build-docs breeze command
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #34004:
URL: https://github.com/apache/airflow/pull/34004#discussion_r1315167232
##########
BREEZE.rst:
##########
@@ -478,6 +481,16 @@ These are all available flags of ``build-docs`` command:
:width: 100%
:alt: Breeze build documentation
+.. _generating_short_form_names:
+
+Generating short form names for Providers
+-----------------------------------------
+
+Skip the ``apache-airflow-providers-`` from the usual provider full names.
+Now with the remaining part, replace every ``dash("-")`` with a ``dot(".")``.
Review Comment:
It's just explanation for parmeter names in Breeze. Historicallly we use "apache-airflow-providers-cncf-kubernetes` and `--package-filter` to choose which packages to build. But we already use "short version" of those packages when we run some breeze command and this change is purely UX change so that you can specify them easier when you run command line. Instead of:
```
breeze build-docs --package-filter apache-airflow-providers-cncf-kubernetes --package-filter apache-airflow-providers-google --package-filter
```
you wil be able to specify:
```
breeze build-docs cncf.kubernetes google
```
This explanation is there to explain the user how to derive the short id from the long one used so far.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] amoghrajesh commented on pull request #34004: Introducing short_version flag for building docs
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #34004:
URL: https://github.com/apache/airflow/pull/34004#issuecomment-1702603975
@eladkal @potiuk I took an initial spin at this, what do you think? I think we can do this phase by phase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] amoghrajesh commented on a diff in pull request #34004: Introducing short_version flag for building docs
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #34004:
URL: https://github.com/apache/airflow/pull/34004#discussion_r1313873764
##########
dev/breeze/src/airflow_breeze/params/doc_build_params.py:
##########
@@ -42,6 +55,9 @@ def args_doc_builder(self) -> list[str]:
doc_args.append("--one-pass-only")
if AIRFLOW_BRANCH != "main":
doc_args.append("--disable-provider-checks")
+ # if short_version is set, the self.package_filter is preprocessed
Review Comment:
Sounds good. Making some modifications to accomodate the situation. I think we should not retire the older behaviour too. Hence, we support the short hands as arguments but `--package-filter` remains. Trying this out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #34004: Allowing short package names in build-docs breeze command
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #34004:
URL: https://github.com/apache/airflow/pull/34004#discussion_r1314486152
##########
BREEZE.rst:
##########
@@ -478,6 +481,16 @@ These are all available flags of ``build-docs`` command:
:width: 100%
:alt: Breeze build documentation
+.. _generating_short_form_names:
+
+Generating short form names for Providers
+-----------------------------------------
+
+Skip the ``apache-airflow-providers-`` from the usual provider full names.
+Now with the remaining part, replace every ``dash("-")`` with a ``dot(".")``.
Review Comment:
It’s weird to require the user to do this. Dots and dashes (and underscores) are equivalent in Python distribution names so the tool should be able to normalise them automatically.
##########
BREEZE.rst:
##########
@@ -478,6 +481,16 @@ These are all available flags of ``build-docs`` command:
:width: 100%
:alt: Breeze build documentation
+.. _generating_short_form_names:
+
+Generating short form names for Providers
+-----------------------------------------
+
+Skip the ``apache-airflow-providers-`` from the usual provider full names.
+Now with the remaining part, replace every ``dash("-")`` with a ``dot(".")``.
Review Comment:
It’s weird to require the user to do this. Dots and dashes (and underscores) are equivalent in Python distribution names so the tool should be able to normalise them automatically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] amoghrajesh commented on a diff in pull request #34004: Allowing short package names in build-docs breeze command
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #34004:
URL: https://github.com/apache/airflow/pull/34004#discussion_r1314557932
##########
dev/README_RELEASE_PROVIDER_PACKAGES.md:
##########
@@ -351,6 +351,14 @@ breeze build-docs --clean-build --package-filter apache-airflow-providers \
--package-filter 'apache-airflow-providers-*'
```
+You can also use shorthand names as arguments instead of using the full names
+for airflow providers. Example:
+
+```shell script
+cd "${AIRFLOW_REPO_ROOT}"
+breeze build-docs cncf.kubernetes sftp --clean-build
+```
Review Comment:
I see. Sure removing it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] amoghrajesh commented on a diff in pull request #34004: Allowing short package names in build-docs breeze command
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #34004:
URL: https://github.com/apache/airflow/pull/34004#discussion_r1314563373
##########
dev/README_RELEASE_PROVIDER_PACKAGES.md:
##########
@@ -366,6 +374,14 @@ breeze build-docs --clean-build \
...
```
+You can also use shorthand names as arguments instead of using the full names
+for airflow providers. Example:
+
+```shell script
+cd "${AIRFLOW_REPO_ROOT}"
+breeze build-docs package1 package2 --clean-build
Review Comment:
cc @potiuk can we remove the script safely and document that elsewhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a diff in pull request #34004: Allowing short package names in build-docs breeze command
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #34004:
URL: https://github.com/apache/airflow/pull/34004#discussion_r1315196704
##########
dev/README_RELEASE_PROVIDER_PACKAGES.md:
##########
@@ -366,6 +374,14 @@ breeze build-docs --clean-build \
...
```
+You can also use shorthand names as arguments instead of using the full names
+for airflow providers. Example:
+
+```shell script
+cd "${AIRFLOW_REPO_ROOT}"
+breeze build-docs package1 package2 --clean-build
Review Comment:
What I meant is that now we don't need the `build_provider_documentation.sh` any more so it can be removed.
I rarely used this script for released
> But there is an issue: the script runs the add back references command as a part of it.
there is no issue... the script handled both doc build and add back referenced but when I didn't use the script I simply ran both commands. I actually prefer it that way. it gives better control over the process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #34004: Allowing short package names in build-docs breeze command
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #34004:
URL: https://github.com/apache/airflow/pull/34004#discussion_r1315167232
##########
BREEZE.rst:
##########
@@ -478,6 +481,16 @@ These are all available flags of ``build-docs`` command:
:width: 100%
:alt: Breeze build documentation
+.. _generating_short_form_names:
+
+Generating short form names for Providers
+-----------------------------------------
+
+Skip the ``apache-airflow-providers-`` from the usual provider full names.
+Now with the remaining part, replace every ``dash("-")`` with a ``dot(".")``.
Review Comment:
It's just explanation for parmeter names in Breeze. Historicallly we use "apache-airflow-providers-cncf-kubernetes` and `--package-filter` to choose which packages to build. But we already use "short version" of those packages when we run some breeze command and this change is purely UX change so that you can specify them easier when you run command line. Instead of:
```
breeze build-docs --package-filter apache-airflow-providers-cncf-kubernetes --package-filter apache-airflow-providers-google
```
you wil be able to specify:
```
breeze build-docs cncf.kubernetes google
```
This explanation is there to explain the user how to derive the short id from the long one used so far.
It's part of `build-docs` command description,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a diff in pull request #34004: Allowing short package names in build-docs breeze command
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #34004:
URL: https://github.com/apache/airflow/pull/34004#discussion_r1314494026
##########
dev/README_RELEASE_PROVIDER_PACKAGES.md:
##########
@@ -351,6 +351,14 @@ breeze build-docs --clean-build --package-filter apache-airflow-providers \
--package-filter 'apache-airflow-providers-*'
```
+You can also use shorthand names as arguments instead of using the full names
+for airflow providers. Example:
+
+```shell script
+cd "${AIRFLOW_REPO_ROOT}"
+breeze build-docs cncf.kubernetes sftp --clean-build
+```
Review Comment:
No need to mention this here. This part of the doc is about building for all packages
##########
dev/README_RELEASE_PROVIDER_PACKAGES.md:
##########
@@ -366,6 +374,14 @@ breeze build-docs --clean-build \
...
```
+You can also use shorthand names as arguments instead of using the full names
+for airflow providers. Example:
+
+```shell script
+cd "${AIRFLOW_REPO_ROOT}"
+breeze build-docs package1 package2 --clean-build
Review Comment:
How does this work for building `apache-airflow-providers` ?
Not that now you can remove the build_provider_documentation.sh part from the guide. Is this script being used somewhere? If not I guess we can remove it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #34004: Introducing short_version flag for building docs
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #34004:
URL: https://github.com/apache/airflow/pull/34004#discussion_r1313073041
##########
dev/breeze/src/airflow_breeze/params/doc_build_params.py:
##########
@@ -42,6 +55,9 @@ def args_doc_builder(self) -> list[str]:
doc_args.append("--one-pass-only")
if AIRFLOW_BRANCH != "main":
doc_args.append("--disable-provider-checks")
+ # if short_version is set, the self.package_filter is preprocessed
Review Comment:
I think it would be better to use the `argument` instead of `--short-version`. And you already have this one:
```
argument_packages_plus_all_providers = click.argument(
"packages_plus_all_providers",
nargs=-1,
required=False,
type=BetterChoice(["all-providers"] + get_available_documentation_packages(short_version=True)),
)
```
See where it us used.
The advantage of using argument over option is tha tyou can use it without anything like this:
```
breeze build.docs cncf.kubernetes apache-airflow sftp
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org