You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/11/18 12:23:17 UTC
[GitHub] [airflow] mik-laj opened a new pull request #12444: Separate out documentation building per provider
mik-laj opened a new pull request #12444:
URL: https://github.com/apache/airflow/pull/12444
Part of: https://github.com/apache/airflow/issues/11423
In this PR I made the following changes:
- I split the documentation generation process for each package separately. Now we are not building one documentation for the entire Airflow, but we are building documentation for the core and for each provider separately.
- The documentation for the Python API for each provider has been migrated to the provider's documentation.
- For apache-airflow-providers-googoe all (almost all documentation) was migrated.
- An extension has been prepared, which facilitates the creation of links between documents i.e. if you write `:class: airlfow.providers......` anywhere in the documentation a working link to the documentation in a good package will be generated.
- RTD is not enough anymore, so documentation is built on CI / CD and published on S3.
Preview: http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/
In this PR, I would like to do a few more things:
- clean-up code
- add documentation on building documentation
In next PRs, I will deal with the following things:
- The documentation will use the official Apache Airflow theme
- Improve the process of publishing the documentation on airflow-site
- migrate the content of the documentation to the provider documentations
- move the core documentation to the apache-airflow directory.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12444: Separate out documentation building per provider
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12444:
URL: https://github.com/apache/airflow/pull/12444#issuecomment-729689121
Looks great! I guess we are going to have a separate version for each provider version (we have only the master now)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #12444: Separate out documentation building per provider
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #12444:
URL: https://github.com/apache/airflow/pull/12444#discussion_r527606388
##########
File path: .github/workflows/ci.yml
##########
@@ -314,10 +314,24 @@ jobs:
run: ./scripts/ci/docs/ci_docs.sh --docs-only
- name: "Upload documentation"
uses: actions/upload-artifact@v2
- if: always()
+ if: always() && github.event_name == 'pull_request'
with:
name: airflow-documentation
path: "./files/documentation"
+ - name: Configure AWS credentials
+ uses: aws-actions/configure-aws-credentials@v1
+ if: >
+ github.ref == 'refs/heads/master' && github.repository == 'mik-laj/airflow' &&
Review comment:
```suggestion
github.ref == 'refs/heads/master' && github.repository == 'apache/airflow' &&
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12444: Separate out documentation building per provider
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12444:
URL: https://github.com/apache/airflow/pull/12444#issuecomment-729692953
I could not find where it is from, but the yellow header has "not" where it should not have one:
This is ~**not**~ the documentation for the Apache Airflow developer version. For documentation for stable versions, see: airflow.apache.org.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #12444: Separate out documentation building per provider
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #12444:
URL: https://github.com/apache/airflow/pull/12444#issuecomment-731096375
@turbaszek Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #12444: Separate out documentation building per provider
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #12444:
URL: https://github.com/apache/airflow/pull/12444#discussion_r526046263
##########
File path: .github/workflows/ci.yml
##########
@@ -318,6 +318,20 @@ jobs:
with:
name: airflow-documentation
path: "./files/documentation"
+ - name: Configure AWS credentials from Test account
+ uses: aws-actions/configure-aws-credentials@v1
+ if: >
+ github.ref == 'refs/heads/master' && github.repository == 'mik-laj/airflow' &&
Review comment:
TODO: I need to change it before merging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #12444: Separate out documentation building per provider
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #12444:
URL: https://github.com/apache/airflow/pull/12444#issuecomment-729692942
> Looks great! I guess we are going to have a separate version for each provider version (we have only the master now)?
On S3, we will only have documentation for the latest version. At airlfow.apache.org, we will have documentation for the stable versions, but publishing the documentation on airflow.apache.org is a separate task.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #12444: Separate out documentation building per provider
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #12444:
URL: https://github.com/apache/airflow/pull/12444#discussion_r527607266
##########
File path: .github/workflows/ci.yml
##########
@@ -314,10 +314,24 @@ jobs:
run: ./scripts/ci/docs/ci_docs.sh --docs-only
- name: "Upload documentation"
uses: actions/upload-artifact@v2
- if: always()
+ if: always() && github.event_name == 'pull_request'
with:
name: airflow-documentation
path: "./files/documentation"
+ - name: Configure AWS credentials
+ uses: aws-actions/configure-aws-credentials@v1
+ if: >
+ github.ref == 'refs/heads/master' && github.repository == 'mik-laj/airflow' &&
Review comment:
```suggestion
github.ref == 'refs/heads/master' && github.repository == 'apache/airflow' &&
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj merged pull request #12444: Separate out documentation building per provider
Posted by GitBox <gi...@apache.org>.
mik-laj merged pull request #12444:
URL: https://github.com/apache/airflow/pull/12444
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #12444: Separate out documentation building per provider
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #12444:
URL: https://github.com/apache/airflow/pull/12444#issuecomment-730664494
@kaxil @potiuk It is ready for review.
I also set up secrets in `apache/airflow` so everything should work after merging.
```
bash gh-secrets.sh list https://github.com/apache/airflow
{
"result": {
"total_count": 3,
"secrets": [
{
"name": "ACTIONS_RUNNER_DEBUG",
"created_at": "2020-10-13T08:25:20Z",
"updated_at": "2020-10-13T08:25:21Z"
},
{
"name": "DOCS_AWS_ACCESS_KEY_ID",
"created_at": "2020-11-19T21:59:33Z",
"updated_at": "2020-11-19T21:59:33Z"
},
{
"name": "DOCS_AWS_SECRET_ACCESS_KEY",
"created_at": "2020-11-19T22:00:14Z",
"updated_at": "2020-11-19T22:00:14Z"
}
]
},
"status": "ok",
"statusCode": 200
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12444: Separate out documentation building per provider
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12444:
URL: https://github.com/apache/airflow/pull/12444#issuecomment-730480331
[The Workflow run](https://github.com/apache/airflow/actions/runs/372647097) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12444: Separate out documentation building per provider
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12444:
URL: https://github.com/apache/airflow/pull/12444#issuecomment-730347072
[The Workflow run](https://github.com/apache/airflow/actions/runs/372271492) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12444: Separate out documentation building per provider
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12444:
URL: https://github.com/apache/airflow/pull/12444#issuecomment-731114551
The PR needs to run all tests because it modifies core of Airflow! Please rebase it to latest master or ask committer to re-run it!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12444: Separate out documentation building per provider
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12444:
URL: https://github.com/apache/airflow/pull/12444#issuecomment-730196144
[The Workflow run](https://github.com/apache/airflow/actions/runs/371829672) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #12444: Separate out documentation building per provider
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #12444:
URL: https://github.com/apache/airflow/pull/12444#discussion_r527608013
##########
File path: .github/workflows/ci.yml
##########
@@ -314,10 +314,24 @@ jobs:
run: ./scripts/ci/docs/ci_docs.sh --docs-only
- name: "Upload documentation"
uses: actions/upload-artifact@v2
- if: always()
+ if: always() && github.event_name == 'pull_request'
with:
name: airflow-documentation
path: "./files/documentation"
+ - name: Configure AWS credentials
+ uses: aws-actions/configure-aws-credentials@v1
+ if: >
+ github.ref == 'refs/heads/master' && github.repository == 'apache/airflow' &&
+ github.event_name == 'push'
+ with:
+ aws-access-key-id: ${{ secrets.DOCS_AWS_ACCESS_KEY_ID }}
+ aws-secret-access-key: ${{ secrets.DOCS_AWS_SECRET_ACCESS_KEY }}
+ aws-region: eu-central-1
+ - name: "Upload documentation to AWS S3"
+ if: >
+ github.ref == 'refs/heads/master' && github.repository == 'mik-laj/airflow' &&
Review comment:
```suggestion
github.ref == 'refs/heads/master' && github.repository == 'apache/airflow' &&
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #12444: Separate out documentation building per provider
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #12444:
URL: https://github.com/apache/airflow/pull/12444#discussion_r526103526
##########
File path: airflow/providers/google/provider.yaml
##########
@@ -542,36 +542,36 @@ hooks:
transfers:
- source-integration-name: Presto
target-integration-name: Google Cloud Storage (GCS)
- how-to-guide: /docs/howto/operator/google/transfer/presto_to_gcs.rst
+ how-to-guide: /docs-providers/apache-airflow-providers-google/operators/transfer/presto_to_gcs.rst
python-module: airflow.providers.google.cloud.transfers.presto_to_gcs
- source-integration-name: SQL
target-integration-name: Google Cloud Storage (GCS)
python-module: airflow.providers.google.cloud.transfers.sql_to_gcs
- source-integration-name: Google Cloud Storage (GCS)
target-integration-name: Google Drive
- how-to-guide: /docs/howto/operator/google/transfer/gcs_to_gdrive.rst
+ how-to-guide: /docs-providers/apache-airflow-providers-google/operators/transfer/gcs_to_gdrive.rst
python-module: airflow.providers.google.suite.transfers.gcs_to_gdrive
- source-integration-name: Microsoft SQL Server (MSSQL)
target-integration-name: Google Cloud Storage (GCS)
python-module: airflow.providers.google.cloud.transfers.mssql_to_gcs
- source-integration-name: Microsoft Azure FileShare
target-integration-name: Google Cloud Storage (GCS)
- how-to-guide: /docs/howto/operator/google/transfer/azure_fileshare_to_gcs.rst
+ how-to-guide: /docs-providers/apache-airflow-providers-google/operators/transfer/azure_fileshare_to_gcs.rst
python-module: airflow.providers.google.cloud.transfers.azure_fileshare_to_gcs
- source-integration-name: Apache Cassandra
target-integration-name: Google Cloud Storage (GCS)
python-module: airflow.providers.google.cloud.transfers.cassandra_to_gcs
- source-integration-name: Google Spreadsheet
target-integration-name: Google Cloud Storage (GCS)
- how-to-guide: /docs/howto/operator/google/transfer/sheets_to_gcs.rst
+ how-to-guide: /docs-providers/apache-airflow-providers-google/operators/transfer/sheets_to_gcs.rst
python-module: airflow.providers.google.cloud.transfers.sheets_to_gcs
- source-integration-name: Amazon Simple Storage Service (S3)
target-integration-name: Google Cloud Storage (GCS)
- how-to-guide: /docs/howto/operator/google/transfer/s3_to_gcs.rst
+ how-to-guide: /docs-providers/apache-airflow-providers-google/operators/transfer/s3_to_gcs.rst
Review comment:
Do we need the whole path here? I think just sheets_to_gcs should be needed and all the rest should be a convention
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12444: Separate out documentation building per provider
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12444:
URL: https://github.com/apache/airflow/pull/12444#issuecomment-730243429
[The Workflow run](https://github.com/apache/airflow/actions/runs/371964171) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #12444: Separate out documentation building per provider
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #12444:
URL: https://github.com/apache/airflow/pull/12444#issuecomment-729647124
Just one suggestion, can we have “name of the provider” instead of the package?
i.e. instead of `apache-airflow-providers-amazon just amazon
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #12444: Separate out documentation building per provider
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #12444:
URL: https://github.com/apache/airflow/pull/12444#discussion_r526111604
##########
File path: airflow/providers/google/provider.yaml
##########
@@ -542,36 +542,36 @@ hooks:
transfers:
- source-integration-name: Presto
target-integration-name: Google Cloud Storage (GCS)
- how-to-guide: /docs/howto/operator/google/transfer/presto_to_gcs.rst
+ how-to-guide: /docs-providers/apache-airflow-providers-google/operators/transfer/presto_to_gcs.rst
python-module: airflow.providers.google.cloud.transfers.presto_to_gcs
- source-integration-name: SQL
target-integration-name: Google Cloud Storage (GCS)
python-module: airflow.providers.google.cloud.transfers.sql_to_gcs
- source-integration-name: Google Cloud Storage (GCS)
target-integration-name: Google Drive
- how-to-guide: /docs/howto/operator/google/transfer/gcs_to_gdrive.rst
+ how-to-guide: /docs-providers/apache-airflow-providers-google/operators/transfer/gcs_to_gdrive.rst
python-module: airflow.providers.google.suite.transfers.gcs_to_gdrive
- source-integration-name: Microsoft SQL Server (MSSQL)
target-integration-name: Google Cloud Storage (GCS)
python-module: airflow.providers.google.cloud.transfers.mssql_to_gcs
- source-integration-name: Microsoft Azure FileShare
target-integration-name: Google Cloud Storage (GCS)
- how-to-guide: /docs/howto/operator/google/transfer/azure_fileshare_to_gcs.rst
+ how-to-guide: /docs-providers/apache-airflow-providers-google/operators/transfer/azure_fileshare_to_gcs.rst
python-module: airflow.providers.google.cloud.transfers.azure_fileshare_to_gcs
- source-integration-name: Apache Cassandra
target-integration-name: Google Cloud Storage (GCS)
python-module: airflow.providers.google.cloud.transfers.cassandra_to_gcs
- source-integration-name: Google Spreadsheet
target-integration-name: Google Cloud Storage (GCS)
- how-to-guide: /docs/howto/operator/google/transfer/sheets_to_gcs.rst
+ how-to-guide: /docs-providers/apache-airflow-providers-google/operators/transfer/sheets_to_gcs.rst
python-module: airflow.providers.google.cloud.transfers.sheets_to_gcs
- source-integration-name: Amazon Simple Storage Service (S3)
target-integration-name: Google Cloud Storage (GCS)
- how-to-guide: /docs/howto/operator/google/transfer/s3_to_gcs.rst
+ how-to-guide: /docs-providers/apache-airflow-providers-google/operators/transfer/s3_to_gcs.rst
Review comment:
Now we definitely need a full path because some documentation is in the core and some in providers. In the long term, I also think the full path seems to be helpful to me, as new contributors will find it easier to find this file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org