[GitHub] [airflow] potiuk commented on a change in pull request #12304: Docs installation improvements

[GitBox <gi...@apache.org>]

potiuk commented on a change in pull request #12304:
URL: https://github.com/apache/airflow/pull/12304#discussion_r522306570



##########
File path: docs/autoapi_templates/index.rst
##########
@@ -431,3 +431,74 @@ All task log handlers are derived from :class:`~airflow.utils.log.file_task_hand
   airflow/providers/elasticsearch/log/index
   airflow/providers/google/cloud/log/index
   airflow/providers/microsoft/azure/log/index
+
+Providers
+---------
+
+Third party integrations are in a separate package :mod:`airflow.providers`.
+
+.. toctree::
+  :includehidden:
+  :glob:
+  :maxdepth: 1
+
+  airflow/providers/amazon/index
+  airflow/providers/apache/cassandra/index
+  airflow/providers/apache/druid/index
+  airflow/providers/apache/hdfs/index
+  airflow/providers/apache/hive/index
+  airflow/providers/apache/kylin/index
+  airflow/providers/apache/livy/index
+  airflow/providers/apache/pig/index
+  airflow/providers/apache/pinot/index
+  airflow/providers/apache/spark/index
+  airflow/providers/apache/sqoop/index
+  airflow/providers/celery/index
+  airflow/providers/cloudant/index
+  airflow/providers/cncf/kubernetes/index
+  airflow/providers/databricks/index
+  airflow/providers/datadog/index
+  airflow/providers/dingding/index
+  airflow/providers/discord/index
+  airflow/providers/docker/index
+  airflow/providers/elasticsearch/index
+  airflow/providers/exasol/index
+  airflow/providers/facebook/index
+  airflow/providers/ftp/index
+  airflow/providers/google/index
+  airflow/providers/grpc/index
+  airflow/providers/hashicorp/index
+  airflow/providers/http/index
+  airflow/providers/imap/index
+  airflow/providers/jdbc/index
+  airflow/providers/jenkins/index
+  airflow/providers/jira/index
+  airflow/providers/microsoft/azure/index
+  airflow/providers/microsoft/mssql/index
+  airflow/providers/microsoft/winrm/index
+  airflow/providers/mongo/index
+  airflow/providers/mysql/index
+  airflow/providers/odbc/index
+  airflow/providers/openfaas/index
+  airflow/providers/opsgenie/index
+  airflow/providers/oracle/index
+  airflow/providers/pagerduty/index
+  airflow/providers/papermill/index
+  airflow/providers/plexus/index
+  airflow/providers/postgres/index
+  airflow/providers/presto/index
+  airflow/providers/qubole/index
+  airflow/providers/redis/index
+  airflow/providers/salesforce/index
+  airflow/providers/samba/index
+  airflow/providers/segment/index
+  airflow/providers/sendgrid/index
+  airflow/providers/sftp/index
+  airflow/providers/singularity/index
+  airflow/providers/slack/index
+  airflow/providers/snowflake/index
+  airflow/providers/sqlite/index
+  airflow/providers/ssh/index
+  airflow/providers/vertica/index
+  airflow/providers/yandex/index
+  airflow/providers/zendesk/index

Review comment:
       Same ere - same aitomated check if all providers are here should be adedd.

##########
File path: .pre-commit-config.yaml
##########
@@ -245,11 +245,11 @@ repos:
         pass_filenames: false
         entry: ./scripts/ci/pre_commit/pre_commit_check_order_setup.py
       - id: setup-installation

Review comment:
       Should be ``setup-extra-packages``

##########
File path: dev/package-data/providers.yaml
##########
@@ -0,0 +1,447 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+---
+providers:
+  - package-name: apache-airflow-providers-amazon
+    description: |
+      Amazon integration (including `Amazon Web Services (AWS) <https://aws.amazon.com/>`__).
+    python-module: airflow.providers.amazon
+    versions:
+      - 1.0.0b1

Review comment:
       I think what we really want to do instead of keeping it in one file, this should be a separate provider.yaml in each provider's directory, rather than one big file with all providers. What we really want to do is make sure that ther  are NO specific  references to specific providers in a core part. This should be all automatically discoverable. It should be enough to drop the "provider.yaml" in "providers/google" package and the script should find that file and get provider information from there.
   
   This has the added benefit, that all changes to a provider (including version change) will be done in the right "provider/nnn" directory and we will be able to see it in the list of changes generated there. Also this means that if you will be changing provider and updating it's description or version, there will be no "core" change whatsoever - which means that the "core" tests will not be triggered.
   
   I am happy to approve it now and change it as a follow-up PR if you do not want to change it now.
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org