[GitHub] [airflow] feluelle commented on a change in pull request #9129: Replaces cloud-provider clis in CI image with docker image aliases

[GitBox <gi...@apache.org>]

feluelle commented on a change in pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#discussion_r435064284



##########
File path: TESTING.rst
##########
@@ -621,12 +621,11 @@ credentials stored in your ``home`` directory. Use this feature with care as it
 visible to anything that you have installed inside the Docker container.
 
 Currently forwarded credentials are:
-  * all credentials stored in ``${HOME}/.config`` (for example, GCP credentials)
-  * credentials stored in ``${HOME}/.gsutil`` for ``gsutil`` tool from GCS
   * credentials stored in ``${HOME}/.aws``, ``${HOME}/.boto``, and ``${HOME}/.s3`` (for AWS authentication)
+  * credentials stored in ``${HOME}/.azure``(for Azure authentication)
+  * credentials stored in ``${HOME}/.config`` (for example, GCP credentials)
+  * credentials stored in ``${HOME}/.gsutil`` for ``gsutil`` tool from GCS
   * credentials stored in ``${HOME}/.docker`` for docker

Review comment:
       Should we sort that alphabetically?

##########
File path: Dockerfile.ci
##########
@@ -338,6 +308,24 @@ RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
         pip install ${ADDITIONAL_PYTHON_DEPS}; \
     fi
 
+ARG AWSCLI_IMAGE="amazon/aws-cli:latest"
+ENV AWSCLI_IMAGE=${AWSCLI_IMAGE}
+
+ARG AZURECLI_IMAGE="mcr.microsoft.com/azure-cli:latest"
+ENV AZURECLI_IMAGE=${AZURECLI_IMAGE}
+
+ARG GCLOUD_IMAGE="gcr.io/google.com/cloudsdktool/cloud-sdk:latest"
+ENV GCLOUD_IMAGE=${GCLOUD_IMAGE}
+
+RUN echo -e "\
+    alias aws=\"docker run --rm -it -v /root/.aws:/root/.aws ${AWSCLI_IMAGE}\" \n\
+    alias aws-update=\"docker pull ${AWSCLI_IMAGE}\" \n\
+    alias az=\"docker run --rm -it -v /root/.azure:/root/.azure ${AZURECLI_IMAGE} az\" \n\
+    alias az-update=\"docker pull ${AZURECLI_IMAGE}\" \n\
+    alias gcloud=\"docker run --rm -it -v /root/.config:/root/.config ${GCLOUD_IMAGE} gcloud\" \n\
+    alias gcloud-update=\"docker pull ${GCLOUD_IMAGE}\" \n\
+    " >> ~/.bashrc
+

Review comment:
       I really like this change - looks cleaner to have seperate docker containers running for those.. but I am wondering how terraform will be able to use those. I am on adding terraform to the CI and I could use [their docker image](https://hub.docker.com/r/hashicorp/terraform/dockerfile) as well.

##########
File path: Dockerfile.ci
##########
@@ -208,8 +178,8 @@ ENV AIRFLOW_SOURCES=${AIRFLOW_SOURCES}
 
 WORKDIR ${AIRFLOW_SOURCES}
 
-RUN mkdir -pv ${AIRFLOW_HOME} \
-    mkdir -pv ${AIRFLOW_HOME}/dags \

Review comment:
       Did that work? 🤔 

##########
File path: Dockerfile.ci
##########
@@ -338,6 +308,24 @@ RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
         pip install ${ADDITIONAL_PYTHON_DEPS}; \
     fi
 
+ARG AWSCLI_IMAGE="amazon/aws-cli:latest"
+ENV AWSCLI_IMAGE=${AWSCLI_IMAGE}
+
+ARG AZURECLI_IMAGE="mcr.microsoft.com/azure-cli:latest"
+ENV AZURECLI_IMAGE=${AZURECLI_IMAGE}
+
+ARG GCLOUD_IMAGE="gcr.io/google.com/cloudsdktool/cloud-sdk:latest"
+ENV GCLOUD_IMAGE=${GCLOUD_IMAGE}

Review comment:
       What is the use-case for having those as `ARG`s and what is the use-case for having those as `ENV`s?

##########
File path: BREEZE.rst
##########
@@ -231,6 +232,41 @@ from your ``logs`` directory in the Airflow sources, so all logs created in the
 visible in the host as well. Every time you enter the container, the ``logs`` directory is
 cleaned so that logs do not accumulate.
 
+CLI for cloud providers
+=======================
+
+For development convenience we installed simple wrappers for the most common cloud providers CLIs. Those
+CLIs are not installed when you build or pull the image - they will be downloaded as docker images
+the first time you attempt to use them. It is downloaded and executed in your host's docker engine so once
+it is downloaded, it will stay until you remove the downloaded images from your host container.
+
+For each of those CLI you need to provide credentials. You can do it in one of the ways:
+
+* You can forward credentials from your host machine when starting breeze with
+  ``--forward-credentials`` flag added when you start ``Breeze``. Just a watch-out - this makes
+  your host credentials available inside the image, so use it with care.
+
+* If you do not want to forward credentials, you can use traditional authentication methods provided by
+  each of the tools (for example ``gcloud auth login``) - the credentials are stored in a configuration
+  directory specific for each CLI tool until you stop breeze session. You will need to re-authenticate
+  when you restart ``Breeze``.
+
+Those are currently installed CLIs (they are available as aliases to the docker commands):
+
++-----------------------+----------+-------------------------------------------------+-------------------+
+| Cloud Provider        | CLI tool | Docker image                                    | Configuration dir |
++=======================+==========+=================================================+===================+
+| Amazon Web Services   | aws      | amazon/aws-cli:latest                           | .aws              |
++-----------------------+----------+-------------------------------------------------+-------------------+
+| Google Cloud Platform | gcloud   | mcr.microsoft.com/azure-cli:latest              | .config/gcloud    |
++-----------------------+----------+-------------------------------------------------+-------------------+
+| Microsoft Azure       | az       | gcr.io/google.com/cloudsdktool/cloud-sdk:latest | .azure            |
++-----------------------+----------+-------------------------------------------------+-------------------+
+
+For each of the CLIs we have also an accompanying ``*-update`` alias (for example aws-update) which
+will pull the latest image for the tool.

Review comment:
       nice!

##########
File path: Dockerfile.ci
##########
@@ -338,6 +308,24 @@ RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
         pip install ${ADDITIONAL_PYTHON_DEPS}; \
     fi
 
+ARG AWSCLI_IMAGE="amazon/aws-cli:latest"
+ENV AWSCLI_IMAGE=${AWSCLI_IMAGE}
+
+ARG AZURECLI_IMAGE="mcr.microsoft.com/azure-cli:latest"
+ENV AZURECLI_IMAGE=${AZURECLI_IMAGE}
+
+ARG GCLOUD_IMAGE="gcr.io/google.com/cloudsdktool/cloud-sdk:latest"
+ENV GCLOUD_IMAGE=${GCLOUD_IMAGE}

Review comment:
       I don't think that is necessary.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org