You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/06/03 23:59:39 UTC
[GitHub] [airflow] potiuk opened a new pull request #9129: Remove gcloud aws
potiuk opened a new pull request #9129:
URL: https://github.com/apache/airflow/pull/9129
NOTE! to reviewed only look at the last commit.. This one is based on #9126
The clis are replaced with aliases that will pull and run
docker images when they are needed.
Added Azure CLI as well.
Closes: #8946 #8947 #8785
---
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Target Github ISSUE in description if exists
- [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #9129: Replaces cloud-provider clis in CI image with docker image aliases
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#discussion_r435242982
##########
File path: Dockerfile.ci
##########
@@ -340,6 +308,23 @@ RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
pip install ${ADDITIONAL_PYTHON_DEPS}; \
fi
+RUN \
+ export AWSCLI_IMAGE="amazon/aws-cli:latest" && \
+ export AZURECLI_IMAGE="mcr.microsoft.com/azure-cli:latest" && \
+ export GCLOUD_IMAGE="gcr.io/google.com/cloudsdktool/cloud-sdk:latest" && \
+ echo -e "\
+ alias aws=\"docker run --rm -it -v /root/.aws:/root/.aws ${AWSCLI_IMAGE}\" \n\
Review comment:
It will work OK. I tested it. There are two ways of running breeze ( I described it with BREEZE.rst)
1) if you run it as normal you would mount the /root/.aws from the container which means that you have to authenticate separately and this authentication will not survive a restart of Breeze.
2) if you run `breeze --forward-credentials` it will mount your "${HOME}/.aws" to the breeze container's /root/.aws and from there it will be further mounted to the "aws" image. This means that you can use your host credentials and they will survive Breeze restart.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #9129: Replaces cloud-provider clis in CI image with docker image aliases
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#discussion_r435219835
##########
File path: Dockerfile.ci
##########
@@ -340,6 +308,23 @@ RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
pip install ${ADDITIONAL_PYTHON_DEPS}; \
fi
+RUN \
+ export AWSCLI_IMAGE="amazon/aws-cli:latest" && \
+ export AZURECLI_IMAGE="mcr.microsoft.com/azure-cli:latest" && \
+ export GCLOUD_IMAGE="gcr.io/google.com/cloudsdktool/cloud-sdk:latest" && \
+ echo -e "\
+ alias aws=\"docker run --rm -it -v /root/.aws:/root/.aws ${AWSCLI_IMAGE}\" \n\
Review comment:
This volume mount is wrong. Since it is using the host's docker daemon, this will mount from /root/.aws on the host, not what /root/.aws is mounted to in this running image.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #9129: Replaces cloud-provider CLIs in CI image with scripts running containers
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#issuecomment-638937158
Anyone tested this change on MacOS?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #9129: Replaces cloud-provider clis in CI image with docker image aliases
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#discussion_r435076765
##########
File path: BREEZE.rst
##########
@@ -231,6 +232,41 @@ from your ``logs`` directory in the Airflow sources, so all logs created in the
visible in the host as well. Every time you enter the container, the ``logs`` directory is
cleaned so that logs do not accumulate.
+CLI for cloud providers
+=======================
+
+For development convenience we installed simple wrappers for the most common cloud providers CLIs. Those
+CLIs are not installed when you build or pull the image - they will be downloaded as docker images
+the first time you attempt to use them. It is downloaded and executed in your host's docker engine so once
+it is downloaded, it will stay until you remove the downloaded images from your host container.
+
+For each of those CLI you need to provide credentials. You can do it in one of the ways:
+
+* You can forward credentials from your host machine when starting breeze with
+ ``--forward-credentials`` flag added when you start ``Breeze``. Just a watch-out - this makes
+ your host credentials available inside the image, so use it with care.
+
+* If you do not want to forward credentials, you can use traditional authentication methods provided by
+ each of the tools (for example ``gcloud auth login``) - the credentials are stored in a configuration
+ directory specific for each CLI tool until you stop breeze session. You will need to re-authenticate
+ when you restart ``Breeze``.
+
+Those are currently installed CLIs (they are available as aliases to the docker commands):
+
++-----------------------+----------+-------------------------------------------------+-------------------+
+| Cloud Provider | CLI tool | Docker image | Configuration dir |
++=======================+==========+=================================================+===================+
+| Amazon Web Services | aws | amazon/aws-cli:latest | .aws |
++-----------------------+----------+-------------------------------------------------+-------------------+
+| Google Cloud Platform | gcloud | mcr.microsoft.com/azure-cli:latest | .config/gcloud |
++-----------------------+----------+-------------------------------------------------+-------------------+
+| Microsoft Azure | az | gcr.io/google.com/cloudsdktool/cloud-sdk:latest | .azure |
++-----------------------+----------+-------------------------------------------------+-------------------+
+
+For each of the CLIs we have also an accompanying ``*-update`` alias (for example aws-update) which
+will pull the latest image for the tool.
Review comment:
Ah yeah. That's really good this way. And also as a side effect if you have 1.10 and master on the same docker engine, they will share those :)
##########
File path: Dockerfile.ci
##########
@@ -338,6 +308,24 @@ RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
pip install ${ADDITIONAL_PYTHON_DEPS}; \
fi
+ARG AWSCLI_IMAGE="amazon/aws-cli:latest"
+ENV AWSCLI_IMAGE=${AWSCLI_IMAGE}
+
+ARG AZURECLI_IMAGE="mcr.microsoft.com/azure-cli:latest"
+ENV AZURECLI_IMAGE=${AZURECLI_IMAGE}
+
+ARG GCLOUD_IMAGE="gcr.io/google.com/cloudsdktool/cloud-sdk:latest"
+ENV GCLOUD_IMAGE=${GCLOUD_IMAGE}
Review comment:
It's not necessary. Agree. Just got used to add those every time. But I can clean it up :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #9129: Replaces cloud-provider clis in CI image with docker image aliases
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#discussion_r435277393
##########
File path: Dockerfile.ci
##########
@@ -340,6 +308,23 @@ RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
pip install ${ADDITIONAL_PYTHON_DEPS}; \
fi
+RUN \
+ export AWSCLI_IMAGE="amazon/aws-cli:latest" && \
+ export AZURECLI_IMAGE="mcr.microsoft.com/azure-cli:latest" && \
+ export GCLOUD_IMAGE="gcr.io/google.com/cloudsdktool/cloud-sdk:latest" && \
+ echo -e "\
+ alias aws=\"docker run --rm -it -v /root/.aws:/root/.aws ${AWSCLI_IMAGE}\" \n\
Review comment:
> If you run `breeze --forward-credentials` it will mount your "${HOME}/.aws" to the breeze container's /root/.aws and from there it will be further mounted to the "aws" image. This means that you can use your host credentials and they will survive Breeze restart.
Is this docker-in-docker, or mounting /var/run/docker.sock from the host.
If it's the latter: this will not work. All volume mounts are resolved from the host:
```
airflow ❯ touch ~/.aws/example
airflow ❯ docker run -ti -v /var/run/docker.sock:/var/run/docker.sock -v $HOME/.aws:/root/.aws docker:latest sh
/ # echo container 1 -- "breeze"
container 1 -- breeze
/ # ls -altr /root/.aws/example
-rw-r--r-- 1 1000 1000 0 Jun 4 13:51 /root/.aws/example
/ # hostname
4b4a523958cb
/ # docker run -ti -v /var/run/docker.sock:/var/run/docker.sock -v /root/.aws/.aws:/root/.aws docker:latest sh
/ # echo container 2 - cloud tools
container 2 - cloud tools
/ # hostname
dcae57c09ea4
/ # ls -altr /root/.aws
total 10
drwxr-xr-x 2 root root 2 Jun 4 13:57 .
drwx------ 3 root root 4 Jun 4 13:57 ..
/ # touch /root/.aws/hello
/ # ls /root/.aws/
example
/ # exit
/ # exit
airflow ❯ sudo ls /root/.aws
hello
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #9129: Replaces cloud-provider clis in CI image with docker image aliases
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#discussion_r435309570
##########
File path: Dockerfile.ci
##########
@@ -340,6 +308,23 @@ RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
pip install ${ADDITIONAL_PYTHON_DEPS}; \
fi
+RUN \
+ export AWSCLI_IMAGE="amazon/aws-cli:latest" && \
+ export AZURECLI_IMAGE="mcr.microsoft.com/azure-cli:latest" && \
+ export GCLOUD_IMAGE="gcr.io/google.com/cloudsdktool/cloud-sdk:latest" && \
+ echo -e "\
+ alias aws=\"docker run --rm -it -v /root/.aws:/root/.aws ${AWSCLI_IMAGE}\" \n\
Review comment:
Yes you are perfectly right, good catch.
I just found out that I actually had /root/.config from earlier authorization apparently on the host. I will update it slightly to forward the HOST_HOME or similar and mount it from there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #9129: Replaces cloud-provider clis in CI image with docker image aliases
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#discussion_r435249164
##########
File path: Dockerfile.ci
##########
@@ -340,6 +308,23 @@ RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
pip install ${ADDITIONAL_PYTHON_DEPS}; \
fi
+RUN \
+ export AWSCLI_IMAGE="amazon/aws-cli:latest" && \
+ export AZURECLI_IMAGE="mcr.microsoft.com/azure-cli:latest" && \
+ export GCLOUD_IMAGE="gcr.io/google.com/cloudsdktool/cloud-sdk:latest" && \
+ echo -e "\
+ alias aws=\"docker run --rm -it -v /root/.aws:/root/.aws ${AWSCLI_IMAGE}\" \n\
Review comment:
Comment from BREEZE.rst
```
For each of those CLI you need to provide credentials. You can do it in one of the ways:
* You can forward credentials from your host machine when starting breeze with
``--forward-credentials`` flag added when you start ``Breeze``. Just a watch-out - this makes
your host credentials available inside the image, so use it with care.
* If you do not want to forward credentials, you can use traditional authentication methods provided by
each of the tools (for example ``gcloud auth login``) - the credentials are stored in a configuration
directory specific for each CLI tool until you stop breeze session. You will need to re-authenticate
when you restart ``Breeze``.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #9129: Replaces cloud-provider clis in CI image with docker image aliases
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#discussion_r435168282
##########
File path: TESTING.rst
##########
@@ -621,12 +621,11 @@ credentials stored in your ``home`` directory. Use this feature with care as it
visible to anything that you have installed inside the Docker container.
Currently forwarded credentials are:
- * all credentials stored in ``${HOME}/.config`` (for example, GCP credentials)
- * credentials stored in ``${HOME}/.gsutil`` for ``gsutil`` tool from GCS
* credentials stored in ``${HOME}/.aws``, ``${HOME}/.boto``, and ``${HOME}/.s3`` (for AWS authentication)
+ * credentials stored in ``${HOME}/.azure``(for Azure authentication)
+ * credentials stored in ``${HOME}/.config`` (for example, GCP credentials)
+ * credentials stored in ``${HOME}/.gsutil`` for ``gsutil`` tool from GCS
* credentials stored in ``${HOME}/.docker`` for docker
Review comment:
Yep. Will do and add a bit more description.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #9129: Replaces cloud-provider CLIs in CI image with scripts running containers
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#issuecomment-638922239
All should be find now @ashb @feluelle
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #9129: Replaces cloud-provider clis in CI image with docker image aliases
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#discussion_r435339194
##########
File path: Dockerfile.ci
##########
@@ -340,6 +308,23 @@ RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
pip install ${ADDITIONAL_PYTHON_DEPS}; \
fi
+RUN \
+ export AWSCLI_IMAGE="amazon/aws-cli:latest" && \
+ export AZURECLI_IMAGE="mcr.microsoft.com/azure-cli:latest" && \
+ export GCLOUD_IMAGE="gcr.io/google.com/cloudsdktool/cloud-sdk:latest" && \
+ echo -e "\
+ alias aws=\"docker run --rm -it -v /root/.aws:/root/.aws ${AWSCLI_IMAGE}\" \n\
Review comment:
I updated the clis (I had to change them from aliases into /usr/bin/script to take into account HOST_HOME variable. But it works like a charm now even without --forward-credentials. Documentation updated as well..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #9129: Replaces cloud-provider CLIs in CI image with scripts running containers
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#issuecomment-639413787
@feluelle I'm working on a fix. We need to create an executable file, not just an alias in bash.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #9129: Replaces cloud-provider clis in CI image with docker image aliases
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#discussion_r435338251
##########
File path: Dockerfile.ci
##########
@@ -340,6 +308,23 @@ RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
pip install ${ADDITIONAL_PYTHON_DEPS}; \
fi
+RUN \
+ export AWSCLI_IMAGE="amazon/aws-cli:latest" && \
+ export AZURECLI_IMAGE="mcr.microsoft.com/azure-cli:latest" && \
+ export GCLOUD_IMAGE="gcr.io/google.com/cloudsdktool/cloud-sdk:latest" && \
+ echo -e "\
+ alias aws=\"docker run --rm -it -v /root/.aws:/root/.aws ${AWSCLI_IMAGE}\" \n\
Review comment:
I think with that, you saved me few hours of debugging of someone's problem :)? Thanks @ashb !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #9129: Replaces cloud-provider clis in CI image with docker image aliases
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#discussion_r435189082
##########
File path: TESTING.rst
##########
@@ -621,12 +621,11 @@ credentials stored in your ``home`` directory. Use this feature with care as it
visible to anything that you have installed inside the Docker container.
Currently forwarded credentials are:
- * all credentials stored in ``${HOME}/.config`` (for example, GCP credentials)
- * credentials stored in ``${HOME}/.gsutil`` for ``gsutil`` tool from GCS
* credentials stored in ``${HOME}/.aws``, ``${HOME}/.boto``, and ``${HOME}/.s3`` (for AWS authentication)
+ * credentials stored in ``${HOME}/.azure``(for Azure authentication)
+ * credentials stored in ``${HOME}/.config`` (for example, GCP credentials)
+ * credentials stored in ``${HOME}/.gsutil`` for ``gsutil`` tool from GCS
* credentials stored in ``${HOME}/.docker`` for docker
Review comment:
Yeah. I also removed .s3 .gsutil and .boto parts - I checked that all the tools got standardized and all of them to use the .config or .aws or .azure and i am only mounting those.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle commented on pull request #9129: Replaces cloud-provider CLIs in CI image with scripts running containers
Posted by GitBox <gi...@apache.org>.
feluelle commented on pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#issuecomment-639408137
@potiuk @mik-laj when I run it via `python -c "import subprocess; subprocess.Popen(['aws']).poll()"` I get `OSError: [Errno 8] Exec format error: 'aws'`.
The shell script runs flawlessy via bash but not in python.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #9129: Replaces cloud-provider clis in CI image with docker image aliases
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#issuecomment-638795541
Hey @feluelle -> comments addressed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #9129: Replaces cloud-provider CLIs in CI image with scripts running containers
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#issuecomment-638980268
I am also testing - just in case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle commented on a change in pull request #9129: Replaces cloud-provider clis in CI image with docker image aliases
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#discussion_r435064284
##########
File path: TESTING.rst
##########
@@ -621,12 +621,11 @@ credentials stored in your ``home`` directory. Use this feature with care as it
visible to anything that you have installed inside the Docker container.
Currently forwarded credentials are:
- * all credentials stored in ``${HOME}/.config`` (for example, GCP credentials)
- * credentials stored in ``${HOME}/.gsutil`` for ``gsutil`` tool from GCS
* credentials stored in ``${HOME}/.aws``, ``${HOME}/.boto``, and ``${HOME}/.s3`` (for AWS authentication)
+ * credentials stored in ``${HOME}/.azure``(for Azure authentication)
+ * credentials stored in ``${HOME}/.config`` (for example, GCP credentials)
+ * credentials stored in ``${HOME}/.gsutil`` for ``gsutil`` tool from GCS
* credentials stored in ``${HOME}/.docker`` for docker
Review comment:
Should we sort that alphabetically?
##########
File path: Dockerfile.ci
##########
@@ -338,6 +308,24 @@ RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
pip install ${ADDITIONAL_PYTHON_DEPS}; \
fi
+ARG AWSCLI_IMAGE="amazon/aws-cli:latest"
+ENV AWSCLI_IMAGE=${AWSCLI_IMAGE}
+
+ARG AZURECLI_IMAGE="mcr.microsoft.com/azure-cli:latest"
+ENV AZURECLI_IMAGE=${AZURECLI_IMAGE}
+
+ARG GCLOUD_IMAGE="gcr.io/google.com/cloudsdktool/cloud-sdk:latest"
+ENV GCLOUD_IMAGE=${GCLOUD_IMAGE}
+
+RUN echo -e "\
+ alias aws=\"docker run --rm -it -v /root/.aws:/root/.aws ${AWSCLI_IMAGE}\" \n\
+ alias aws-update=\"docker pull ${AWSCLI_IMAGE}\" \n\
+ alias az=\"docker run --rm -it -v /root/.azure:/root/.azure ${AZURECLI_IMAGE} az\" \n\
+ alias az-update=\"docker pull ${AZURECLI_IMAGE}\" \n\
+ alias gcloud=\"docker run --rm -it -v /root/.config:/root/.config ${GCLOUD_IMAGE} gcloud\" \n\
+ alias gcloud-update=\"docker pull ${GCLOUD_IMAGE}\" \n\
+ " >> ~/.bashrc
+
Review comment:
I really like this change - looks cleaner to have seperate docker containers running for those.. but I am wondering how terraform will be able to use those. I am on adding terraform to the CI and I could use [their docker image](https://hub.docker.com/r/hashicorp/terraform/dockerfile) as well.
##########
File path: Dockerfile.ci
##########
@@ -208,8 +178,8 @@ ENV AIRFLOW_SOURCES=${AIRFLOW_SOURCES}
WORKDIR ${AIRFLOW_SOURCES}
-RUN mkdir -pv ${AIRFLOW_HOME} \
- mkdir -pv ${AIRFLOW_HOME}/dags \
Review comment:
Did that work? 🤔
##########
File path: Dockerfile.ci
##########
@@ -338,6 +308,24 @@ RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
pip install ${ADDITIONAL_PYTHON_DEPS}; \
fi
+ARG AWSCLI_IMAGE="amazon/aws-cli:latest"
+ENV AWSCLI_IMAGE=${AWSCLI_IMAGE}
+
+ARG AZURECLI_IMAGE="mcr.microsoft.com/azure-cli:latest"
+ENV AZURECLI_IMAGE=${AZURECLI_IMAGE}
+
+ARG GCLOUD_IMAGE="gcr.io/google.com/cloudsdktool/cloud-sdk:latest"
+ENV GCLOUD_IMAGE=${GCLOUD_IMAGE}
Review comment:
What is the use-case for having those as `ARG`s and what is the use-case for having those as `ENV`s?
##########
File path: BREEZE.rst
##########
@@ -231,6 +232,41 @@ from your ``logs`` directory in the Airflow sources, so all logs created in the
visible in the host as well. Every time you enter the container, the ``logs`` directory is
cleaned so that logs do not accumulate.
+CLI for cloud providers
+=======================
+
+For development convenience we installed simple wrappers for the most common cloud providers CLIs. Those
+CLIs are not installed when you build or pull the image - they will be downloaded as docker images
+the first time you attempt to use them. It is downloaded and executed in your host's docker engine so once
+it is downloaded, it will stay until you remove the downloaded images from your host container.
+
+For each of those CLI you need to provide credentials. You can do it in one of the ways:
+
+* You can forward credentials from your host machine when starting breeze with
+ ``--forward-credentials`` flag added when you start ``Breeze``. Just a watch-out - this makes
+ your host credentials available inside the image, so use it with care.
+
+* If you do not want to forward credentials, you can use traditional authentication methods provided by
+ each of the tools (for example ``gcloud auth login``) - the credentials are stored in a configuration
+ directory specific for each CLI tool until you stop breeze session. You will need to re-authenticate
+ when you restart ``Breeze``.
+
+Those are currently installed CLIs (they are available as aliases to the docker commands):
+
++-----------------------+----------+-------------------------------------------------+-------------------+
+| Cloud Provider | CLI tool | Docker image | Configuration dir |
++=======================+==========+=================================================+===================+
+| Amazon Web Services | aws | amazon/aws-cli:latest | .aws |
++-----------------------+----------+-------------------------------------------------+-------------------+
+| Google Cloud Platform | gcloud | mcr.microsoft.com/azure-cli:latest | .config/gcloud |
++-----------------------+----------+-------------------------------------------------+-------------------+
+| Microsoft Azure | az | gcr.io/google.com/cloudsdktool/cloud-sdk:latest | .azure |
++-----------------------+----------+-------------------------------------------------+-------------------+
+
+For each of the CLIs we have also an accompanying ``*-update`` alias (for example aws-update) which
+will pull the latest image for the tool.
Review comment:
nice!
##########
File path: Dockerfile.ci
##########
@@ -338,6 +308,24 @@ RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
pip install ${ADDITIONAL_PYTHON_DEPS}; \
fi
+ARG AWSCLI_IMAGE="amazon/aws-cli:latest"
+ENV AWSCLI_IMAGE=${AWSCLI_IMAGE}
+
+ARG AZURECLI_IMAGE="mcr.microsoft.com/azure-cli:latest"
+ENV AZURECLI_IMAGE=${AZURECLI_IMAGE}
+
+ARG GCLOUD_IMAGE="gcr.io/google.com/cloudsdktool/cloud-sdk:latest"
+ENV GCLOUD_IMAGE=${GCLOUD_IMAGE}
Review comment:
I don't think that is necessary.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle commented on pull request #9129: Replaces cloud-provider CLIs in CI image with scripts running containers
Posted by GitBox <gi...@apache.org>.
feluelle commented on pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#issuecomment-639408882
I found this issue when running `pytest` i.e running `LoggingCommandExecutor.execute_cmd`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #9129: Replaces cloud-provider CLIs in CI image with scripts running containers
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#issuecomment-638987993
Works like a charm on MacOS as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #9129: Replaces cloud-provider clis in CI image with docker image aliases
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#discussion_r435075650
##########
File path: Dockerfile.ci
##########
@@ -338,6 +308,24 @@ RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
pip install ${ADDITIONAL_PYTHON_DEPS}; \
fi
+ARG AWSCLI_IMAGE="amazon/aws-cli:latest"
+ENV AWSCLI_IMAGE=${AWSCLI_IMAGE}
+
+ARG AZURECLI_IMAGE="mcr.microsoft.com/azure-cli:latest"
+ENV AZURECLI_IMAGE=${AZURECLI_IMAGE}
+
+ARG GCLOUD_IMAGE="gcr.io/google.com/cloudsdktool/cloud-sdk:latest"
+ENV GCLOUD_IMAGE=${GCLOUD_IMAGE}
+
+RUN echo -e "\
+ alias aws=\"docker run --rm -it -v /root/.aws:/root/.aws ${AWSCLI_IMAGE}\" \n\
+ alias aws-update=\"docker pull ${AWSCLI_IMAGE}\" \n\
+ alias az=\"docker run --rm -it -v /root/.azure:/root/.azure ${AZURECLI_IMAGE} az\" \n\
+ alias az-update=\"docker pull ${AZURECLI_IMAGE}\" \n\
+ alias gcloud=\"docker run --rm -it -v /root/.config:/root/.config ${GCLOUD_IMAGE} gcloud\" \n\
+ alias gcloud-update=\"docker pull ${GCLOUD_IMAGE}\" \n\
+ " >> ~/.bashrc
+
Review comment:
I think it wil work. In the worst case we can create "/usr/bin/gcloud" scripts. I had it done like that originally but aliases sound much cleaner.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle commented on pull request #9129: Replaces cloud-provider CLIs in CI image with scripts running containers
Posted by GitBox <gi...@apache.org>.
feluelle commented on pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#issuecomment-639414354
Yes, just the header `#!/bin/bash` is missing
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #9129: Replaces cloud-provider clis in CI image with docker image aliases
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#discussion_r435074867
##########
File path: Dockerfile.ci
##########
@@ -208,8 +178,8 @@ ENV AIRFLOW_SOURCES=${AIRFLOW_SOURCES}
WORKDIR ${AIRFLOW_SOURCES}
-RUN mkdir -pv ${AIRFLOW_HOME} \
- mkdir -pv ${AIRFLOW_HOME}/dags \
Review comment:
Yes. It worked it just created "mkdir" and "-pw" dirs additionally :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #9129: Replaces cloud-provider clis in CI image with docker image aliases
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#discussion_r435277393
##########
File path: Dockerfile.ci
##########
@@ -340,6 +308,23 @@ RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
pip install ${ADDITIONAL_PYTHON_DEPS}; \
fi
+RUN \
+ export AWSCLI_IMAGE="amazon/aws-cli:latest" && \
+ export AZURECLI_IMAGE="mcr.microsoft.com/azure-cli:latest" && \
+ export GCLOUD_IMAGE="gcr.io/google.com/cloudsdktool/cloud-sdk:latest" && \
+ echo -e "\
+ alias aws=\"docker run --rm -it -v /root/.aws:/root/.aws ${AWSCLI_IMAGE}\" \n\
Review comment:
> If you run `breeze --forward-credentials` it will mount your "${HOME}/.aws" to the breeze container's /root/.aws and from there it will be further mounted to the "aws" image. This means that you can use your host credentials and they will survive Breeze restart.
Is this docker-in-docker, or mounting /var/run/docker.sock from the host.
If it's the latter: this will not work. All volume mounts are resolved from the host:
```
airflow ❯ touch ~/.aws/example
airflow ❯ docker run -ti \
-v /var/run/docker.sock:/var/run/docker.sock \
-v $HOME/.aws:/root/.aws \
docker:latest sh
/ # echo container 1 -- "breeze"
container 1 -- breeze
/ # ls -altr /root/.aws/example
-rw-r--r-- 1 1000 1000 0 Jun 4 13:51 /root/.aws/example
/ # hostname
4b4a523958cb
/ # docker run -ti \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /root/.aws/.aws:/root/.aws \
docker:latest sh
/ # echo container 2 - cloud tools
container 2 - cloud tools
/ # hostname
dcae57c09ea4
/ # ls -altr /root/.aws
total 10
drwxr-xr-x 2 root root 2 Jun 4 13:57 .
drwx------ 3 root root 4 Jun 4 13:57 ..
/ # touch /root/.aws/hello
/ # ls /root/.aws/
example
/ # exit
/ # exit
airflow ❯ sudo ls /root/.aws
hello
```
(Added line breaks for a bit more clarity.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #9129: Replaces cloud-provider clis in CI image with docker image aliases
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#discussion_r435277393
##########
File path: Dockerfile.ci
##########
@@ -340,6 +308,23 @@ RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
pip install ${ADDITIONAL_PYTHON_DEPS}; \
fi
+RUN \
+ export AWSCLI_IMAGE="amazon/aws-cli:latest" && \
+ export AZURECLI_IMAGE="mcr.microsoft.com/azure-cli:latest" && \
+ export GCLOUD_IMAGE="gcr.io/google.com/cloudsdktool/cloud-sdk:latest" && \
+ echo -e "\
+ alias aws=\"docker run --rm -it -v /root/.aws:/root/.aws ${AWSCLI_IMAGE}\" \n\
Review comment:
> If you run `breeze --forward-credentials` it will mount your "${HOME}/.aws" to the breeze container's /root/.aws and from there it will be further mounted to the "aws" image. This means that you can use your host credentials and they will survive Breeze restart.
Is this docker-in-docker, or mounting /var/run/docker.sock from the host.
If it's the latter: this will not work. All volume mounts are resolved from the host:
```
airflow ❯ touch ~/.aws/example
airflow ❯ docker run -ti -v /var/run/docker.sock:/var/run/docker.sock -v $HOME/.aws:/root/.aws docker:latest sh
/ # echo container 1 -- "breeze"
container 1 -- breeze
/ # ls -altr /root/.aws/example
-rw-r--r-- 1 1000 1000 0 Jun 4 13:51 /root/.aws/example
/ # hostname
4b4a523958cb
/ # docker run -ti -v /var/run/docker.sock:/var/run/docker.sock -v /root/.aws/.aws:/root/.aws docker:latest sh
/ # echo container 2 - cloud tools
container 2 - cloud tools
/ # hostname
dcae57c09ea4
/ # ls -altr /root/.aws
total 18
drwxr-xr-x 2 root root 2 Jun 4 13:55 .
drwx------ 3 root root 4 Jun 4 13:55 ..
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #9129: Replaces cloud-provider CLIs in CI image with scripts running containers
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#issuecomment-638957204
I'm starting testing on Mac OS.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #9129: Replaces cloud-provider CLIs in CI image with scripts running containers
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #9129:
URL: https://github.com/apache/airflow/pull/9129
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #9129: Replaces cloud-provider clis in CI image with docker image aliases
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#discussion_r435242982
##########
File path: Dockerfile.ci
##########
@@ -340,6 +308,23 @@ RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
pip install ${ADDITIONAL_PYTHON_DEPS}; \
fi
+RUN \
+ export AWSCLI_IMAGE="amazon/aws-cli:latest" && \
+ export AZURECLI_IMAGE="mcr.microsoft.com/azure-cli:latest" && \
+ export GCLOUD_IMAGE="gcr.io/google.com/cloudsdktool/cloud-sdk:latest" && \
+ echo -e "\
+ alias aws=\"docker run --rm -it -v /root/.aws:/root/.aws ${AWSCLI_IMAGE}\" \n\
Review comment:
It will work OK. I tested it. There are two ways of running breeze ( I described it with TESTING.rst)
1) if you run it as normal you would mount the /root/.aws from the container which means that you have to authenticate separately and this authentication will not survive restart of Breeze.
2) if you run `breeze --forward-credentials` it will mount your "${HOME}/.aws" to the breeze container's /root/.aws and from there it will be further mounted to the "aws" image. Which means that you can use your host credentials and they will survive Breeze restart.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #9129: Replaces cloud-provider CLIs in CI image with scripts running containers
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#issuecomment-639481616
Yeah. Aliases are gone already the scripts are there but they might need shebang indeed. I saw they were working fine and I thought they were mainly there for manual use, but when we want to use it in python it will be indeed problematic without it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #9129: Replaces cloud-provider clis in CI image with docker image aliases
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#discussion_r435185248
##########
File path: Dockerfile.ci
##########
@@ -338,6 +308,24 @@ RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
pip install ${ADDITIONAL_PYTHON_DEPS}; \
fi
+ARG AWSCLI_IMAGE="amazon/aws-cli:latest"
+ENV AWSCLI_IMAGE=${AWSCLI_IMAGE}
+
+ARG AZURECLI_IMAGE="mcr.microsoft.com/azure-cli:latest"
+ENV AZURECLI_IMAGE=${AZURECLI_IMAGE}
+
+ARG GCLOUD_IMAGE="gcr.io/google.com/cloudsdktool/cloud-sdk:latest"
+ENV GCLOUD_IMAGE=${GCLOUD_IMAGE}
+
+RUN echo -e "\
+ alias aws=\"docker run --rm -it -v /root/.aws:/root/.aws ${AWSCLI_IMAGE}\" \n\
+ alias aws-update=\"docker pull ${AWSCLI_IMAGE}\" \n\
+ alias az=\"docker run --rm -it -v /root/.azure:/root/.azure ${AZURECLI_IMAGE} az\" \n\
+ alias az-update=\"docker pull ${AZURECLI_IMAGE}\" \n\
+ alias gcloud=\"docker run --rm -it -v /root/.config:/root/.config ${GCLOUD_IMAGE} gcloud\" \n\
+ alias gcloud-update=\"docker pull ${GCLOUD_IMAGE}\" \n\
+ " >> ~/.bashrc
+
Review comment:
We can work it out - if you want to do terraform similar way - the worst that we will be able to do is to build our own image with terraform, aws, az also embedded :). But I think we can work it out with some smart aliases and docker-in-docker as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org