You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by vi...@apache.org on 2024/03/22 15:10:27 UTC
(airflow) branch main updated: Create AWS auth manager documentation. Part: setup identity center (#38273)
This is an automated email from the ASF dual-hosted git repository.
vincbeck pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 83d62cae69 Create AWS auth manager documentation. Part: setup identity center (#38273)
83d62cae69 is described below
commit 83d62cae695ebea0e537c7bfa8e9cc65de05ec09
Author: Vincent <97...@users.noreply.github.com>
AuthorDate: Fri Mar 22 12:10:16 2024 -0300
Create AWS auth manager documentation. Part: setup identity center (#38273)
---
.../auth-manager/index.rst | 39 ++++++
.../auth-manager/setup/config.rst | 40 ++++++
.../auth-manager/setup/identity-center.rst | 147 +++++++++++++++++++++
.../img/diagram_auth_manager_architecture.md5sum | 1 +
.../img/diagram_auth_manager_architecture.png | Bin 0 -> 104408 bytes
.../img/diagram_auth_manager_architecture.py | 70 ++++++++++
.../img/icons/avp.png | Bin 0 -> 13029 bytes
.../img/icons/idc.png | Bin 0 -> 18332 bytes
docs/apache-airflow-providers-amazon/index.rst | 1 +
9 files changed, 298 insertions(+)
diff --git a/docs/apache-airflow-providers-amazon/auth-manager/index.rst b/docs/apache-airflow-providers-amazon/auth-manager/index.rst
new file mode 100644
index 0000000000..0a74715894
--- /dev/null
+++ b/docs/apache-airflow-providers-amazon/auth-manager/index.rst
@@ -0,0 +1,39 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+================
+AWS auth manager
+================
+
+Before reading this, you should be familiar with the concept of auth manager.
+See :doc:`apache-airflow:core-concepts/auth-manager`.
+
+The AWS auth manager is an auth manager powered by AWS. It uses two services:
+
+* `AWS IAM Identity Center <https://aws.amazon.com/iam/identity-center/>`_ for authentication purposes
+* `Amazon Verified Permissions <https://aws.amazon.com/verified-permissions/>`_ for authorization purposes
+
+.. image:: ../img/diagram_auth_manager_architecture.png
+
+
+**Getting started**
+
+.. toctree::
+ :maxdepth: 2
+
+ setup/config
+ setup/identity-center
diff --git a/docs/apache-airflow-providers-amazon/auth-manager/setup/config.rst b/docs/apache-airflow-providers-amazon/auth-manager/setup/config.rst
new file mode 100644
index 0000000000..ed50641663
--- /dev/null
+++ b/docs/apache-airflow-providers-amazon/auth-manager/setup/config.rst
@@ -0,0 +1,40 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+=============================================
+Configure Airflow to use the AWS auth manager
+=============================================
+
+In order to use the AWS auth manager as auth manager in your Airflow environment, you need to set it in your Airflow configuration.
+You also need to specify the AWS region where both services used by the AWS auth manager
+(AWS IAM Identity Center and Amazon Verified Permissions) are configured.
+
+.. note::
+ Configuration options must be consistent across all the hosts/environments running the Airflow components (Scheduler, Webserver, ECS Task containers, etc). See `here <https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html>`__ for more details on setting configurations.
+
+.. code-block:: ini
+
+ [core]
+ auth_manager = airflow.providers.amazon.aws.auth_manager.aws_auth_manager.AwsAuthManager
+ region_name = <region_name>
+
+or
+
+.. code-block:: bash
+
+ export AIRFLOW__CORE__AUTH_MANAGER='airflow.providers.amazon.aws.auth_manager.aws_auth_manager.AwsAuthManager'
+ export AIRFLOW__AWS_AUTH_MANAGER__REGION_NAME='<region_name>'
diff --git a/docs/apache-airflow-providers-amazon/auth-manager/setup/identity-center.rst b/docs/apache-airflow-providers-amazon/auth-manager/setup/identity-center.rst
new file mode 100644
index 0000000000..09f910997d
--- /dev/null
+++ b/docs/apache-airflow-providers-amazon/auth-manager/setup/identity-center.rst
@@ -0,0 +1,147 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+=================================
+Configure AWS IAM Identity Center
+=================================
+
+In order to use the AWS auth manager, you first need to configure `AWS IAM Identity Center <https://aws.amazon.com/iam/identity-center/>`_.
+AWS IAM Identity Center is used by the AWS auth manager for authentication purposes (login and logout).
+Following configuration, the Airflow environment administrator can manage users and groups with Identity Center service.
+
+Create resources
+================
+
+The AWS auth manager needs two resources in AWS IAM Identity Center: an instance and an application.
+You can create them either through the provided CLI command or manually.
+
+Create resources with CLI
+-------------------------
+
+.. note::
+ The CLI command is not compatible with AWS accounts that are managed through AWS organizations.
+ If your AWS account is managed through an AWS organization, please follow the
+ :ref:`manual configuration <identity_center_manual_configuration>`.
+
+.. note::
+ To create all necessary resources for the AWS Auth Manager, you can utilize the CLI command provided as part of the
+ AWS auth manager. Before executing the command, ensure the AWS auth manager is configured as the auth manager
+ for the Airflow instance. See :doc:`/auth-manager/setup/config`.
+
+To create the resources, please run the following command:
+
+.. code-block:: bash
+
+ airflow aws-auth-manager init-identity-center
+
+The CLI command should exit successfully with the message: ::
+
+ AWS IAM Identity Center resources created successfully.
+
+If the CLI command exited with an error, please look carefully at the CLI command output to understand which resource(s)
+have or have not been created successfully. The resource(s) which have not been successfully created need to be
+:ref:`created manually <identity_center_manual_configuration>`.
+
+If the error message below is raised, please create the AWS IAM Identity Center application through the console
+following :ref:`these instructions <identity_center_manual_configuration_application>`: ::
+
+ Creation of SAML applications is only supported in AWS console today. Please create the application through the console.
+
+.. _identity_center_manual_configuration:
+
+Create resources manually
+-------------------------
+
+Create the instance
+~~~~~~~~~~~~~~~~~~~
+
+Please follow `AWS documentation <https://docs.aws.amazon.com/singlesignon/latest/userguide/identity-center-instances.html>`_
+to create the AWS IAM Identity Center instance.
+
+.. _identity_center_manual_configuration_application:
+
+Create the application
+~~~~~~~~~~~~~~~~~~~~~~
+
+Please follow the instructions below to create the AWS IAM Identity Center application.
+
+1. Open the `IAM Identity Center console <https://console.aws.amazon.com/singlesignon>`_.
+2. Choose **Applications**.
+3. Choose the **Customer managed** tab.
+4. Choose **Add application**.
+5. On the **Select application type** page, under **Setup preference**, choose **I have an application I want to set up**.
+6. Under **Application type**, choose **SAML 2.0**.
+7. Choose **Next**.
+8. On the **Configure application** page, under **Configure application**, enter a **Display name** for the application, such as ``Airflow``. Then, enter a Description.
+9. Under **IAM Identity Center metadata**, copy the address of the **IAM Identity Center SAML metadata file**.
+
+.. note::
+ You will need to set this address in Airflow configuration later.
+
+10. Under **Application metadata**, choose **Manually type your metadata values**. Then, provide the **Application ACS URL** and **Application SAML audience** as follows.
+
+ .. important::
+ Replace ``<base_url>`` by the base URL of your Airflow UI. It should be defined in ``AIRFLOW__WEBSERVER__BASE_URL``
+ (e.g. ``localhost:8080`` if Airflow is running locally).
+
+ * **Application ACS URL**: ``<base_url>/login_callback``
+ * **Application SAML audience**: ``<base_url>/login_metadata``
+
+11. Choose **Submit**. The application is now created.
+
+Attribute mappings configuration
+================================
+
+Once the application is created, you need to configure the attribute mappings.
+
+1. Go to the details page of the application that you just created.
+2. Choose **Actions**.
+3. Under **Actions**, choose **Edit attribute mappings**.
+4. On the **Attribute mappings** page, you need to configure the different attribute mappings between your identity
+ provider and AWS IAM Identity Center. For more information on attribute mappings, see the
+ `IAM Identity Center documentation <https://docs.aws.amazon.com/singlesignon/latest/userguide/attributemappingsconcept.html>`_.
+ The AWS auth manager needs two attributes: **id** and **groups**.
+ If you use the default Identity Center directory as identity source, you can use the configuration below:
+
+ * **id**
+
+ * **User attribute in the application**: ``id``
+ * **Maps to this string value or user attribute in IAM Identity Center**: ``${user:AD_GUID}``
+ * **Format**: ``basic``
+ * **groups**
+
+ * **User attribute in the application**: ``groups``
+ * **Maps to this string value or user attribute in IAM Identity Center**: ``${user:groups}``
+ * **Format**: ``basic``
+
+5. Once both attributes **id** and **groups** are defined, choose **Save changes**.
+
+Configure Airflow
+=================
+
+You need to set in Airflow configuration the IAM Identity Center SAML metadata file created previously.
+
+.. code-block:: ini
+
+ [aws_auth_manager]
+ saml_metadata_url = <saml_metadata_file_url>
+
+or
+
+.. code-block:: bash
+
+ export AIRFLOW__AWS_AUTH_MANAGER__SAML_METADATA_URL='<saml_metadata_file_url>'
diff --git a/docs/apache-airflow-providers-amazon/img/diagram_auth_manager_architecture.md5sum b/docs/apache-airflow-providers-amazon/img/diagram_auth_manager_architecture.md5sum
new file mode 100644
index 0000000000..f5167d2ddf
--- /dev/null
+++ b/docs/apache-airflow-providers-amazon/img/diagram_auth_manager_architecture.md5sum
@@ -0,0 +1 @@
+32e06ab6da0640f2570feac7d974f645
diff --git a/docs/apache-airflow-providers-amazon/img/diagram_auth_manager_architecture.png b/docs/apache-airflow-providers-amazon/img/diagram_auth_manager_architecture.png
new file mode 100644
index 0000000000..87696dc2f3
Binary files /dev/null and b/docs/apache-airflow-providers-amazon/img/diagram_auth_manager_architecture.png differ
diff --git a/docs/apache-airflow-providers-amazon/img/diagram_auth_manager_architecture.py b/docs/apache-airflow-providers-amazon/img/diagram_auth_manager_architecture.py
new file mode 100644
index 0000000000..a8910a01aa
--- /dev/null
+++ b/docs/apache-airflow-providers-amazon/img/diagram_auth_manager_architecture.py
@@ -0,0 +1,70 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from pathlib import Path
+
+from diagrams import Cluster, Diagram, Edge
+from diagrams.aws.general import Users
+from diagrams.aws.security import IAMPermissions
+from diagrams.custom import Custom
+from diagrams.onprem.client import User
+from rich.console import Console
+
+MY_DIR = Path(__file__).parent
+MY_FILENAME = Path(__file__).with_suffix("").name
+PYTHON_MULTIPROCESS_LOGO = MY_DIR.parents[1] / "diagrams" / "python_multiprocess_logo.png"
+IDENTITY_CENTER_LOGO = MY_DIR / "icons" / "idc.png"
+AVP_LOGO = MY_DIR / "icons" / "avp.png"
+
+console = Console(width=400, color_system="standard")
+
+
+def generate_auth_manager_diagram():
+ image_file = (MY_DIR / MY_FILENAME).with_suffix(".png")
+ console.print(f"[bright_blue]Generating architecture image {image_file}")
+ with Diagram(
+ show=False,
+ filename=MY_FILENAME,
+ ):
+ user = User("Airflow user")
+ admin = User("Admin")
+ with Cluster("Airflow environment"):
+ webserver = Custom("Webserver(s)", PYTHON_MULTIPROCESS_LOGO.as_posix())
+
+ with Cluster("Amazon provider"):
+ auth_manager = Custom("AWS auth manager", PYTHON_MULTIPROCESS_LOGO.as_posix())
+
+ identity_center = Custom("AWS IAM Identity Center", IDENTITY_CENTER_LOGO.as_posix())
+ avp = Custom("Amazon Verified Permissions", AVP_LOGO.as_posix())
+
+ user >> Edge(label="Access to the UI/Rest API") >> webserver
+ webserver >> auth_manager
+ auth_manager >> Edge(label="Authentication") >> identity_center
+ auth_manager >> Edge(label="Authorization") >> avp
+
+ Users("Users") >> identity_center
+ IAMPermissions("Permissions") >> avp
+
+ identity_center << Edge(label="Manage users/groups") << admin
+ avp << Edge(label="Manage permissions") << admin
+
+ console.print(f"[green]Generating architecture image {image_file}")
+
+
+if __name__ == "__main__":
+ generate_auth_manager_diagram()
diff --git a/docs/apache-airflow-providers-amazon/img/icons/avp.png b/docs/apache-airflow-providers-amazon/img/icons/avp.png
new file mode 100644
index 0000000000..c24f72b8ce
Binary files /dev/null and b/docs/apache-airflow-providers-amazon/img/icons/avp.png differ
diff --git a/docs/apache-airflow-providers-amazon/img/icons/idc.png b/docs/apache-airflow-providers-amazon/img/icons/idc.png
new file mode 100644
index 0000000000..b7a6a5752a
Binary files /dev/null and b/docs/apache-airflow-providers-amazon/img/icons/idc.png differ
diff --git a/docs/apache-airflow-providers-amazon/index.rst b/docs/apache-airflow-providers-amazon/index.rst
index 1532d02d44..0fa8342546 100644
--- a/docs/apache-airflow-providers-amazon/index.rst
+++ b/docs/apache-airflow-providers-amazon/index.rst
@@ -42,6 +42,7 @@
Logging for Tasks <logging/index>
Configuration <configurations-ref>
Executors <executors/index>
+ Auth manager <auth-manager/index>
CLI <cli-ref>
.. toctree::