You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "shashbha14 (via GitHub)" <gi...@apache.org> on 2026/05/15 13:16:23 UTC
[PR] Add Bundle documentation for Amazon S3 and Google GCS providers [airflow]
shashbha14 opened a new pull request, #66993:
URL: https://github.com/apache/airflow/pull/66993
Closes #66986
Added a top-level Bundles section to the Amazon and Google provider docs,
following the same structure as the existing Git provider bundle docs.
## Changes
- `providers/amazon/docs/bundles/index.rst` — documents S3DagBundle with
configuration examples and parameter descriptions
- `providers/google/docs/bundles/index.rst` — documents GCSDagBundle with
configuration examples and parameter descriptions
- Updated `providers/amazon/docs/index.rst` — added Bundles to the Guides toctree
- Updated `providers/google/docs/index.rst` — added Bundles to the Guides toctree
## Screenshots
Rendered doc screenshots pending — building locally with Breeze now, will add shortly.
> I used Claude (claude.ai) as an AI assistant for parts of this implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Bundle documentation for Amazon S3 and Google GCS providers [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #66993:
URL: https://github.com/apache/airflow/pull/66993#discussion_r3249627735
##########
providers/google/docs/bundles/index.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Bundles
+#######
+
+A DAG bundle is a way to load DAGs into Airflow from an external source. For a general overview of
+DAG bundles, see :doc:`apache-airflow:administration-and-deployment/dag-bundles`.
+
+GCSDagBundle
+============
+
+Use the :class:`~airflow.providers.google.cloud.bundles.gcs.GCSDagBundle` to load DAGs directly from
+a Google Cloud Storage bucket. Airflow will periodically sync DAG files from the specified GCS bucket
+and prefix to a local directory and load them from there.
Review Comment:
Should we mentioned that this is unversioned bundle? Same with S3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Bundle documentation for Amazon S3 and Google GCS providers [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #66993:
URL: https://github.com/apache/airflow/pull/66993#discussion_r3249601220
##########
providers/amazon/docs/bundles/index.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Bundles
+#######
+
+A DAG bundle is a way to load DAGs into Airflow from an external source. For a general overview of
Review Comment:
```suggestion
A Dag bundle is a way to load Dags into Airflow from an external source. For a general overview of
```
##########
providers/amazon/docs/bundles/index.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Bundles
+#######
+
+A DAG bundle is a way to load DAGs into Airflow from an external source. For a general overview of
+DAG bundles, see :doc:`apache-airflow:administration-and-deployment/dag-bundles`.
+
+S3DagBundle
+===========
+
+Use the :class:`~airflow.providers.amazon.aws.bundles.s3.S3DagBundle` to load DAGs directly from
Review Comment:
```suggestion
Use the :class:`~airflow.providers.amazon.aws.bundles.s3.S3DagBundle` to load Dags directly from
```
We resolved to use "Dag" instead of "DAG" in documentations. Please update other places. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Bundle documentation for Amazon S3 and Google GCS providers [airflow]
Posted by "shashbha14 (via GitHub)" <gi...@apache.org>.
shashbha14 commented on PR #66993:
URL: https://github.com/apache/airflow/pull/66993#issuecomment-4466993703
> @shashbha14 while git bundle does exist https://airflow.apache.org/docs/apache-airflow-providers-git/stable/bundles/index.html#bundles can you please edit it to cross reference the bundle docs in core? Ideally we should have the same info/format for the providers on that front. So if something is good on git lets reuse it in the other ones and vise-versa
Added a cross-reference to the core dag-bundles doc and simplified the format to match the git provider bundle docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Bundle documentation for Amazon S3 and Google GCS providers [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #66993:
URL: https://github.com/apache/airflow/pull/66993#discussion_r3249619247
##########
providers/amazon/docs/bundles/index.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Bundles
+#######
+
+A DAG bundle is a way to load DAGs into Airflow from an external source. For a general overview of
+DAG bundles, see :doc:`apache-airflow:administration-and-deployment/dag-bundles`.
+
+S3DagBundle
+===========
+
+Use the :class:`~airflow.providers.amazon.aws.bundles.s3.S3DagBundle` to load DAGs directly from
+an Amazon S3 bucket. Airflow will periodically sync DAG files from the specified S3 bucket and prefix
Review Comment:
```suggestion
an Amazon S3 bucket. Airflow will periodically sync Dag files from the specified S3 bucket and prefix
```
##########
providers/amazon/docs/bundles/index.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Bundles
+#######
+
+A DAG bundle is a way to load DAGs into Airflow from an external source. For a general overview of
+DAG bundles, see :doc:`apache-airflow:administration-and-deployment/dag-bundles`.
+
+S3DagBundle
+===========
+
+Use the :class:`~airflow.providers.amazon.aws.bundles.s3.S3DagBundle` to load DAGs directly from
+an Amazon S3 bucket. Airflow will periodically sync DAG files from the specified S3 bucket and prefix
+to a local directory and load them from there.
+
+Prerequisites
+-------------
+
+- An AWS connection configured in Airflow (see :doc:`/connections/aws`).
+- An S3 bucket containing your DAG Python files.
Review Comment:
```suggestion
- An S3 bucket containing your Dag Python files.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Bundle documentation for Amazon S3 and Google GCS providers [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #66993:
URL: https://github.com/apache/airflow/pull/66993#discussion_r3249622098
##########
providers/amazon/docs/bundles/index.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Bundles
+#######
+
+A DAG bundle is a way to load DAGs into Airflow from an external source. For a general overview of
+DAG bundles, see :doc:`apache-airflow:administration-and-deployment/dag-bundles`.
+
+S3DagBundle
+===========
+
+Use the :class:`~airflow.providers.amazon.aws.bundles.s3.S3DagBundle` to load DAGs directly from
+an Amazon S3 bucket. Airflow will periodically sync DAG files from the specified S3 bucket and prefix
+to a local directory and load them from there.
+
+Prerequisites
+-------------
+
+- An AWS connection configured in Airflow (see :doc:`/connections/aws`).
+- An S3 bucket containing your DAG Python files.
+
+Configuration
+-------------
+
+Add the bundle to your ``[dag_processor] dag_bundle_config_list`` configuration:
+
+.. code-block:: json
+
+ [
+ {
+ "name": "my-s3-dags",
+ "classpath": "airflow.providers.amazon.aws.bundles.s3.S3DagBundle",
+ "kwargs": {
+ "aws_conn_id": "aws_default",
+ "bucket_name": "my-airflow-bucket",
+ "prefix": "dags/",
+ "refresh_interval": 60
+ }
+ }
+ ]
+
+Or using an environment variable:
+
+.. code-block:: bash
+
+ export AIRFLOW__DAG_PROCESSOR__DAG_BUNDLE_CONFIG_LIST='[
+ {
+ "name": "my-s3-dags",
+ "classpath": "airflow.providers.amazon.aws.bundles.s3.S3DagBundle",
+ "kwargs": {
+ "aws_conn_id": "aws_default",
+ "bucket_name": "my-airflow-bucket",
+ "prefix": "dags/",
+ "refresh_interval": 60
+ }
+ }
+ ]'
+
+Parameters
+----------
+
+- ``aws_conn_id`` – Airflow connection ID for AWS. Defaults to ``aws_default``.
+- ``bucket_name`` – Name of the S3 bucket containing the DAG files.
+- ``prefix`` – Optional subdirectory prefix within the bucket. If omitted, DAGs are loaded from the root of the bucket.
+- ``refresh_interval`` – How often (in seconds) to sync DAGs from S3.
Review Comment:
```suggestion
- ``refresh_interval`` – How often (in seconds) to sync Dags from S3.
```
##########
providers/amazon/docs/bundles/index.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Bundles
+#######
+
+A DAG bundle is a way to load DAGs into Airflow from an external source. For a general overview of
+DAG bundles, see :doc:`apache-airflow:administration-and-deployment/dag-bundles`.
+
+S3DagBundle
+===========
+
+Use the :class:`~airflow.providers.amazon.aws.bundles.s3.S3DagBundle` to load DAGs directly from
+an Amazon S3 bucket. Airflow will periodically sync DAG files from the specified S3 bucket and prefix
+to a local directory and load them from there.
+
+Prerequisites
+-------------
+
+- An AWS connection configured in Airflow (see :doc:`/connections/aws`).
+- An S3 bucket containing your DAG Python files.
+
+Configuration
+-------------
+
+Add the bundle to your ``[dag_processor] dag_bundle_config_list`` configuration:
+
+.. code-block:: json
+
+ [
+ {
+ "name": "my-s3-dags",
+ "classpath": "airflow.providers.amazon.aws.bundles.s3.S3DagBundle",
+ "kwargs": {
+ "aws_conn_id": "aws_default",
+ "bucket_name": "my-airflow-bucket",
+ "prefix": "dags/",
+ "refresh_interval": 60
+ }
+ }
+ ]
+
+Or using an environment variable:
+
+.. code-block:: bash
+
+ export AIRFLOW__DAG_PROCESSOR__DAG_BUNDLE_CONFIG_LIST='[
+ {
+ "name": "my-s3-dags",
+ "classpath": "airflow.providers.amazon.aws.bundles.s3.S3DagBundle",
+ "kwargs": {
+ "aws_conn_id": "aws_default",
+ "bucket_name": "my-airflow-bucket",
+ "prefix": "dags/",
+ "refresh_interval": 60
+ }
+ }
+ ]'
+
+Parameters
+----------
+
+- ``aws_conn_id`` – Airflow connection ID for AWS. Defaults to ``aws_default``.
+- ``bucket_name`` – Name of the S3 bucket containing the DAG files.
+- ``prefix`` – Optional subdirectory prefix within the bucket. If omitted, DAGs are loaded from the root of the bucket.
Review Comment:
```suggestion
- ``prefix`` – Optional subdirectory prefix within the bucket. If omitted, Dags are loaded from the root of the bucket.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Bundle documentation for Amazon S3 and Google GCS providers [airflow]
Posted by "shashbha14 (via GitHub)" <gi...@apache.org>.
shashbha14 commented on code in PR #66993:
URL: https://github.com/apache/airflow/pull/66993#discussion_r3249694791
##########
providers/amazon/docs/bundles/index.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Bundles
+#######
+
+A DAG bundle is a way to load DAGs into Airflow from an external source. For a general overview of
+DAG bundles, see :doc:`apache-airflow:administration-and-deployment/dag-bundles`.
+
+S3DagBundle
+===========
+
+Use the :class:`~airflow.providers.amazon.aws.bundles.s3.S3DagBundle` to load DAGs directly from
+an Amazon S3 bucket. Airflow will periodically sync DAG files from the specified S3 bucket and prefix
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Bundle documentation for Amazon S3 and Google GCS providers [airflow]
Posted by "shashbha14 (via GitHub)" <gi...@apache.org>.
shashbha14 commented on code in PR #66993:
URL: https://github.com/apache/airflow/pull/66993#discussion_r3249765197
##########
providers/amazon/docs/bundles/index.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Bundles
+#######
+
+A DAG bundle is a way to load DAGs into Airflow from an external source. For a general overview of
+DAG bundles, see :doc:`apache-airflow:administration-and-deployment/dag-bundles`.
+
+S3DagBundle
+===========
+
+Use the :class:`~airflow.providers.amazon.aws.bundles.s3.S3DagBundle` to load DAGs directly from
+an Amazon S3 bucket. Airflow will periodically sync DAG files from the specified S3 bucket and prefix
+to a local directory and load them from there.
+
+Prerequisites
+-------------
+
+- An AWS connection configured in Airflow (see :doc:`/connections/aws`).
+- An S3 bucket containing your DAG Python files.
+
+Configuration
+-------------
+
+Add the bundle to your ``[dag_processor] dag_bundle_config_list`` configuration:
+
+.. code-block:: json
+
+ [
+ {
+ "name": "my-s3-dags",
+ "classpath": "airflow.providers.amazon.aws.bundles.s3.S3DagBundle",
+ "kwargs": {
+ "aws_conn_id": "aws_default",
+ "bucket_name": "my-airflow-bucket",
+ "prefix": "dags/",
+ "refresh_interval": 60
+ }
+ }
+ ]
+
+Or using an environment variable:
+
+.. code-block:: bash
+
+ export AIRFLOW__DAG_PROCESSOR__DAG_BUNDLE_CONFIG_LIST='[
+ {
+ "name": "my-s3-dags",
+ "classpath": "airflow.providers.amazon.aws.bundles.s3.S3DagBundle",
+ "kwargs": {
+ "aws_conn_id": "aws_default",
+ "bucket_name": "my-airflow-bucket",
+ "prefix": "dags/",
+ "refresh_interval": 60
+ }
+ }
+ ]'
+
+Parameters
+----------
+
+- ``aws_conn_id`` – Airflow connection ID for AWS. Defaults to ``aws_default``.
+- ``bucket_name`` – Name of the S3 bucket containing the DAG files.
+- ``prefix`` – Optional subdirectory prefix within the bucket. If omitted, DAGs are loaded from the root of the bucket.
+- ``refresh_interval`` – How often (in seconds) to sync DAGs from S3.
Review Comment:
Replaced
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Bundle documentation for Amazon S3 and Google GCS providers [airflow]
Posted by "shashbha14 (via GitHub)" <gi...@apache.org>.
shashbha14 commented on code in PR #66993:
URL: https://github.com/apache/airflow/pull/66993#discussion_r3249767934
##########
providers/google/docs/bundles/index.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Bundles
+#######
+
+A DAG bundle is a way to load DAGs into Airflow from an external source. For a general overview of
+DAG bundles, see :doc:`apache-airflow:administration-and-deployment/dag-bundles`.
+
+GCSDagBundle
+============
+
+Use the :class:`~airflow.providers.google.cloud.bundles.gcs.GCSDagBundle` to load DAGs directly from
+a Google Cloud Storage bucket. Airflow will periodically sync DAG files from the specified GCS bucket
+and prefix to a local directory and load them from there.
Review Comment:
Good catch — added a note to GCSDagBundle that it does not support versioning. Should I add the same note to S3DagBundle as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Bundle documentation for Amazon S3 and Google GCS providers [airflow]
Posted by "shashbha14 (via GitHub)" <gi...@apache.org>.
shashbha14 commented on code in PR #66993:
URL: https://github.com/apache/airflow/pull/66993#discussion_r3249763083
##########
providers/amazon/docs/bundles/index.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Bundles
+#######
+
+A DAG bundle is a way to load DAGs into Airflow from an external source. For a general overview of
+DAG bundles, see :doc:`apache-airflow:administration-and-deployment/dag-bundles`.
+
+S3DagBundle
+===========
+
+Use the :class:`~airflow.providers.amazon.aws.bundles.s3.S3DagBundle` to load DAGs directly from
+an Amazon S3 bucket. Airflow will periodically sync DAG files from the specified S3 bucket and prefix
+to a local directory and load them from there.
+
+Prerequisites
+-------------
+
+- An AWS connection configured in Airflow (see :doc:`/connections/aws`).
+- An S3 bucket containing your DAG Python files.
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Bundle documentation for Amazon S3 and Google GCS providers [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #66993:
URL: https://github.com/apache/airflow/pull/66993#issuecomment-4461850624
@shashbha14 while git bundle does exist https://airflow.apache.org/docs/apache-airflow-providers-git/stable/bundles/index.html#bundles can you please edit it to cross reference the bundle docs in core? Ideally we should have the same info/format for the providers on that front. So if something is good on git lets reuse it in the other ones and vise-versa
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Bundle documentation for Amazon S3 and Google GCS providers [airflow]
Posted by "shashbha14 (via GitHub)" <gi...@apache.org>.
shashbha14 commented on code in PR #66993:
URL: https://github.com/apache/airflow/pull/66993#discussion_r3249771970
##########
providers/amazon/docs/bundles/index.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Bundles
+#######
+
+A DAG bundle is a way to load DAGs into Airflow from an external source. For a general overview of
+DAG bundles, see :doc:`apache-airflow:administration-and-deployment/dag-bundles`.
+
+S3DagBundle
+===========
+
+Use the :class:`~airflow.providers.amazon.aws.bundles.s3.S3DagBundle` to load DAGs directly from
+an Amazon S3 bucket. Airflow will periodically sync DAG files from the specified S3 bucket and prefix
+to a local directory and load them from there.
+
+Prerequisites
+-------------
+
+- An AWS connection configured in Airflow (see :doc:`/connections/aws`).
+- An S3 bucket containing your DAG Python files.
+
+Configuration
+-------------
+
+Add the bundle to your ``[dag_processor] dag_bundle_config_list`` configuration:
+
+.. code-block:: json
+
+ [
+ {
+ "name": "my-s3-dags",
+ "classpath": "airflow.providers.amazon.aws.bundles.s3.S3DagBundle",
+ "kwargs": {
+ "aws_conn_id": "aws_default",
+ "bucket_name": "my-airflow-bucket",
+ "prefix": "dags/",
+ "refresh_interval": 60
+ }
+ }
+ ]
+
+Or using an environment variable:
+
+.. code-block:: bash
+
+ export AIRFLOW__DAG_PROCESSOR__DAG_BUNDLE_CONFIG_LIST='[
+ {
+ "name": "my-s3-dags",
+ "classpath": "airflow.providers.amazon.aws.bundles.s3.S3DagBundle",
+ "kwargs": {
+ "aws_conn_id": "aws_default",
+ "bucket_name": "my-airflow-bucket",
+ "prefix": "dags/",
+ "refresh_interval": 60
+ }
+ }
+ ]'
+
+Parameters
+----------
+
+- ``aws_conn_id`` – Airflow connection ID for AWS. Defaults to ``aws_default``.
+- ``bucket_name`` – Name of the S3 bucket containing the DAG files.
+- ``prefix`` – Optional subdirectory prefix within the bucket. If omitted, DAGs are loaded from the root of the bucket.
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Bundle documentation for Amazon S3 and Google GCS providers [airflow]
Posted by "shashbha14 (via GitHub)" <gi...@apache.org>.
shashbha14 commented on code in PR #66993:
URL: https://github.com/apache/airflow/pull/66993#discussion_r3249693457
##########
providers/amazon/docs/bundles/index.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Bundles
+#######
+
+A DAG bundle is a way to load DAGs into Airflow from an external source. For a general overview of
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Bundle documentation for Amazon S3 and Google GCS providers [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal merged PR #66993:
URL: https://github.com/apache/airflow/pull/66993
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Bundle documentation for Amazon S3 and Google GCS providers [airflow]
Posted by "shashbha14 (via GitHub)" <gi...@apache.org>.
shashbha14 commented on PR #66993:
URL: https://github.com/apache/airflow/pull/66993#issuecomment-4466994861
> We also need new bundles to be present in provider.yaml otherwise the [registry](https://airflow.apache.org/registry/) won't pick them up.
>
> example for Git provider:
>
> https://github.com/apache/airflow/blob/077821050598e6807c63a497bcb70c420b529d8b/providers/git/provider.yaml#L62-L65
Both providers already have bundles registered in their provider.yaml files (Amazon at line 739, Google at line 934).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org