You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2021/03/25 19:41:26 UTC
[airflow] branch v2-0-test updated: Synchronize Provider templates
with master
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/v2-0-test by this push:
new d09187c Synchronize Provider templates with master
d09187c is described below
commit d09187cd67c016b08d646121b2dc56ae8333e764
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Thu Mar 25 20:40:37 2021 +0100
Synchronize Provider templates with master
---
....jinja2 => PROVIDER_README_TEMPLATE.rst.jinja2} | 53 ++++++++++------------
dev/provider_packages/README.md | 25 +++++++---
dev/provider_packages/SETUP_TEMPLATE.py.jinja2 | 6 +--
.../build_provider_documentation.sh | 35 ++++++++++++++
.../publish_provider_documentation.sh | 34 ++++++++++++++
dev/provider_packages/remove_old_releases.py | 11 ++---
dev/provider_packages/tag_providers.sh | 29 ++++++++++++
7 files changed, 147 insertions(+), 46 deletions(-)
diff --git a/dev/provider_packages/PROVIDER_README_TEMPLATE.md.jinja2 b/dev/provider_packages/PROVIDER_README_TEMPLATE.rst.jinja2
similarity index 61%
rename from dev/provider_packages/PROVIDER_README_TEMPLATE.md.jinja2
rename to dev/provider_packages/PROVIDER_README_TEMPLATE.rst.jinja2
index 92fe13f..9975925 100644
--- a/dev/provider_packages/PROVIDER_README_TEMPLATE.md.jinja2
+++ b/dev/provider_packages/PROVIDER_README_TEMPLATE.rst.jinja2
@@ -20,69 +20,64 @@
OVERWRITTEN WHEN PREPARING PACKAGES.
IF YOU WANT TO MODIFY IT, YOU SHOULD MODIFY THE TEMPLATE
- `PROVIDER_README_TEMPLATE.md.jinja2` IN the `dev/provider_packages` DIRECTORY
+ `PROVIDER_INDEX_TEMPLATE.rst.jinja2` IN the `dev/provider_packages` DIRECTORY
#}
-# Package {{ PACKAGE_PIP_NAME }}
+Package ``{{ PACKAGE_PIP_NAME }}``
-{{ PROVIDER_DESCRIPTION | safe }}
-
-Release: {{ RELEASE }}{{ VERSION_SUFFIX }}
+Release: ``{{ RELEASE }}{{ VERSION_SUFFIX }}``
-**Table of contents**
-- [Provider package](#provider-package)
-- [Installation](#installation)
-{%- if PIP_REQUIREMENTS %}
-- [PIP requirements](#pip-requirements)
-{%- endif %}
-{%- if CROSS_PROVIDERS_DEPENDENCIES %}
-- [Cross provider package dependencies](#cross-provider-package-dependencies)
-{%- endif %}
+{{ PROVIDER_DESCRIPTION | safe }}
-## Provider package
+Provider package
+================
-This is a provider package for `{{PROVIDER_PACKAGE_ID}}` provider. All classes for this provider package
-are in `{{FULL_PACKAGE_NAME}}` python package.
+This is a provider package for ``{{PROVIDER_PACKAGE_ID}}`` provider. All classes for this provider package
+are in ``{{FULL_PACKAGE_NAME}}`` python package.
You can find package information and changelog for the provider
-in [documentation](https://airflow.apache.org/docs/{{ PACKAGE_PIP_NAME }}/{{RELEASE}}/index.html).
+in the `documentation <https://airflow.apache.org/docs/{{ PACKAGE_PIP_NAME }}/{{RELEASE}}/>`_.
-## Installation
+Installation
+============
NOTE!
On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
-`pip install --upgrade pip==20.2.4` or, in case you use Pip 20.3, you need to add option
-`--use-deprecated legacy-resolver` to your pip install command.
+``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
+``--use-deprecated legacy-resolver`` to your pip install command.
You can install this package on top of an existing airflow 2.* installation via
-`pip install {{PACKAGE_PIP_NAME}}`
+``pip install {{PACKAGE_PIP_NAME}}``
{%- if PIP_REQUIREMENTS %}
-## PIP requirements
+PIP requirements
+================
-{{ PIP_REQUIREMENTS_TABLE | safe }}
+{{ PIP_REQUIREMENTS_TABLE_RST | safe }}
{%- endif %}
{%- if CROSS_PROVIDERS_DEPENDENCIES %}
-## Cross provider package dependencies
+Cross provider package dependencies
+===================================
Those are dependencies that might be needed in order to use all the features of the package.
You need to install the specified provider packages in order to use them.
You can install such cross-provider dependencies when installing from PyPI. For example:
-```bash
-pip install {{ PACKAGE_PIP_NAME }}[{{ CROSS_PROVIDERS_DEPENDENCIES[0] }}]
-```
+.. code-block:: bash
+
+ pip install {{ PACKAGE_PIP_NAME }}[{{ CROSS_PROVIDERS_DEPENDENCIES[0] }}]
+
-{{ CROSS_PROVIDERS_DEPENDENCIES_TABLE | safe }}
+{{ CROSS_PROVIDERS_DEPENDENCIES_TABLE_RST | safe }}
{%- endif %}
diff --git a/dev/provider_packages/README.md b/dev/provider_packages/README.md
index 7aa4fec..7fc3f26 100644
--- a/dev/provider_packages/README.md
+++ b/dev/provider_packages/README.md
@@ -42,7 +42,7 @@
# What the provider packages are
-The Prvider Provider packages are separate packages (one package per provider) that implement
+The Provider packages are separate packages (one package per provider) that implement
integrations with external services for Airflow in the form of installable Python packages.
The Release Manager prepares packages separately from the main Airflow Release, using
@@ -182,6 +182,7 @@ also do not container the leading 0s.
* You can install the .whl packages with `pip install <PACKAGE_FILE>`
+You can add `--verbose` flag if you want to see detailed commands executed by the script.
# Testing and debugging provider preparation
@@ -244,6 +245,9 @@ You can see for example list of all provider packages:
./dev/provider_packages/prepare_provider_packages.py list-providers-packages
```
+
+You can add `--verbose` flag in breeze command if you want to see commands executed.
+
## Debugging import check
The script verifies if all provider's classes can be imported.
@@ -328,8 +332,9 @@ pip install -e ".[devel_all]"
3) Run update documentation (version suffix might be empty):
```shell script
-./dev/provider_packages/prepare_provider_packages.py --version-suffix <SUFFIX> \
- update-package-documentation <PACKAGE>
+./dev/provider_packages/prepare_provider_packages.py update-package-documentation \
+ --version-suffix <SUFFIX> \
+ <PACKAGE>
```
This script will fetch the latest version of airflow from Airflow's repo (it will automatically add
@@ -339,6 +344,8 @@ to setup any credentials for it.
In case version being prepared is already tagged in the repo documentation preparation returns immediately
and prints warning.
+You can add `--verbose` flag if you want to see detailed commands executed by the script.
+
## Debugging preparing setup files
This script prepares the actual packages.
@@ -373,10 +380,13 @@ last time it was generated. In the CI we always add 'dev' suffix, and we never c
TAG for it, so in the CI the setup.py is generated and should never fail.
```shell script
-./dev/provider_packages/prepare_provider_packages.py --version-suffix "<SUFFIX>" \
- generate-setup-files <PACKAGE>
+./dev/provider_packages/prepare_provider_packages.py generate-setup-files \
+ --version-suffix "<SUFFIX>" \
+ <PACKAGE>
```
+You can add `--verbose` flag if you want to see detailed commands executed by the script.
+
## Debugging preparing the packages
The script prepares the package after sources have been copied and setup files generated.
@@ -403,8 +413,9 @@ pip install -e ".[devel_all]"
3) Run update documentation (version suffix might be empty):
```shell script
-./dev/provider_packages/prepare_provider_packages.py --version-suffix <SUFFIX> \
- build-provider-packages <PACKAGE>
+./dev/provider_packages/prepare_provider_packages.py build-provider-packages \
+ --version-suffix <SUFFIX> \
+ <PACKAGE>
```
In case version being prepared is already tagged in the repo documentation preparation returns immediately
diff --git a/dev/provider_packages/SETUP_TEMPLATE.py.jinja2 b/dev/provider_packages/SETUP_TEMPLATE.py.jinja2
index 4af1e1b..9071a7f 100644
--- a/dev/provider_packages/SETUP_TEMPLATE.py.jinja2
+++ b/dev/provider_packages/SETUP_TEMPLATE.py.jinja2
@@ -37,7 +37,7 @@ version = '{{ RELEASE_NO_LEADING_ZEROS }}{{ VERSION_SUFFIX }}'
my_dir = dirname(__file__)
try:
- with open(os.path.join(my_dir, '{{ PROVIDER_PATH }}/{{ README_FILE }}'), encoding='utf-8') as f:
+ with open(os.path.join(my_dir, '{{ README_FILE }}'), encoding='utf-8') as f:
long_description = f.read()
except FileNotFoundError:
long_description = ''
@@ -50,7 +50,7 @@ def do_setup():
description='{{ PROVIDER_TYPE }} package '
'{{ PACKAGE_PIP_NAME }} for Apache Airflow',
long_description=long_description,
- long_description_content_type='text/markdown',
+ long_description_content_type='text/x-rst',
license='Apache License 2.0',
version=version,
packages=find_namespace_packages(
@@ -79,7 +79,7 @@ def do_setup():
download_url='https://archive.apache.org/dist/airflow/{{ PROVIDERS_FOLDER }}',
python_requires='~=3.6',
project_urls={
- 'Documentation': 'https://airflow.apache.org/docs/',
+ 'Documentation': 'https://airflow.apache.org/docs/{{ PACKAGE_PIP_NAME }}/{{RELEASE}}/',
'Bug Tracker': 'https://github.com/apache/airflow/issues',
'Source Code': 'https://github.com/apache/airflow',
},
diff --git a/dev/provider_packages/build_provider_documentation.sh b/dev/provider_packages/build_provider_documentation.sh
new file mode 100755
index 0000000..a98c40d
--- /dev/null
+++ b/dev/provider_packages/build_provider_documentation.sh
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+set -euo pipefail
+
+if [[ $# == "0" ]]; then
+ echo "ERROR: Pass provider ids as list"
+ exit 1
+fi
+
+provider_filters=()
+for provider in "${@}"
+do
+ provider_filters+=("--package-filter" "apache-airflow-providers-${provider//./-}")
+done
+
+./breeze build-docs -- \
+ --for-production \
+ --package-filter apache-airflow-providers \
+ "${provider_filters[@]}"
+cd "${AIRFLOW_SITE_DIRECTORY}"
diff --git a/dev/provider_packages/publish_provider_documentation.sh b/dev/provider_packages/publish_provider_documentation.sh
new file mode 100755
index 0000000..4ebf7f2
--- /dev/null
+++ b/dev/provider_packages/publish_provider_documentation.sh
@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+set -euo pipefail
+
+if [[ $# == "0" ]]; then
+ echo "ERROR: Pass provider ids as list"
+ exit 1
+fi
+
+provider_filters=()
+for provider in "${@}"
+do
+ provider_filters+=("--package-filter" "apache-airflow-providers-${provider//./-}")
+done
+
+./docs/publish_docs.py \
+ --package-filter apache-airflow-providers \
+ "${provider_filters[@]}"
+cd "${AIRFLOW_SITE_DIRECTORY}"
diff --git a/dev/provider_packages/remove_old_releases.py b/dev/provider_packages/remove_old_releases.py
index fb8643d..2c8c62d 100644
--- a/dev/provider_packages/remove_old_releases.py
+++ b/dev/provider_packages/remove_old_releases.py
@@ -67,7 +67,7 @@ def process_all_files(directory: str, suffix: str, execute: bool):
versioned_file = package_types[0]
print(
"Leaving the only version: "
- f"${versioned_file.base + versioned_file.version + versioned_file.suffix}"
+ f"{versioned_file.base + versioned_file.version + versioned_file.suffix}"
)
# Leave only last version from each type
for versioned_file in package_types[:-1]:
@@ -95,12 +95,9 @@ def parse_args() -> argparse.Namespace:
if __name__ == '__main__':
args = parse_args()
- process_all_files(args.directory, "-bin.tar.gz", args.execute)
- process_all_files(args.directory, "-bin.tar.gz.sha512", args.execute)
- process_all_files(args.directory, "-bin.tar.gz.asc", args.execute)
- process_all_files(args.directory, "-source.tar.gz", args.execute)
- process_all_files(args.directory, "-source.tar.gz.sha512", args.execute)
- process_all_files(args.directory, "-source.tar.gz.asc", args.execute)
+ process_all_files(args.directory, ".tar.gz", args.execute)
+ process_all_files(args.directory, ".tar.gz.sha512", args.execute)
+ process_all_files(args.directory, ".tar.gz.asc", args.execute)
process_all_files(args.directory, "-py3-none-any.whl", args.execute)
process_all_files(args.directory, "-py3-none-any.whl.sha512", args.execute)
process_all_files(args.directory, "-py3-none-any.whl.asc", args.execute)
diff --git a/dev/provider_packages/tag_providers.sh b/dev/provider_packages/tag_providers.sh
new file mode 100755
index 0000000..f290945
--- /dev/null
+++ b/dev/provider_packages/tag_providers.sh
@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+set -euo pipefail
+AIRFLOW_SOURCES="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../" && pwd)"
+
+for file in "${AIRFLOW_SOURCES}/dist/"*.whl
+do
+ if [[ ${file} =~ .*airflow_providers_(.*)-(.*)-py3.* ]]; then
+ provider="providers-${BASH_REMATCH[1]}"
+ tag="${provider//_/-}/${BASH_REMATCH[2]}"
+ git tag "${tag}"
+ git push apache "${tag}"
+ fi
+done