You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/06/07 13:48:52 UTC
[GitHub] [airflow] potiuk opened a new pull request #9167: Improved cloud tool available in the trimmed down CI container
potiuk opened a new pull request #9167:
URL: https://github.com/apache/airflow/pull/9167
The tools now have shebangs which make them available for
python tools. Also /opt/airflow is now mounted from the
host Airflow sources which makes it possible for the tools to
copy files directly to/from the sources of Airflow.
It also contains one small change for Linux users - the files
created by docker gcloud are created with root user so in order to fix
that the directories mounted from the host are fixed when you exit
the tool - their ownership is changed to be owned by the host user
---
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Target Github ISSUE in description if exists
- [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #9167: Improved cloud tool available in the trimmed down CI container
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #9167:
URL: https://github.com/apache/airflow/pull/9167
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on pull request #9167: Improved cloud tool available in the trimmed down CI container
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on pull request #9167:
URL: https://github.com/apache/airflow/pull/9167#issuecomment-640256880
I still have problems with gcloud. First of all, I never use host credentials for authorization. When I need to authorize, I use service account keys that are shared by my organization in the `/files/gcp/keys/` directory.
When I try to do this, I get the following message.
```
root@8d3ff81941e3:/opt/airflow# gcloud auth activate-service-account --key-file=/files/gcp/keys/sa.json
ERROR: (gcloud.auth.activate-service-account) Unable to read file [/files/gcp/keys/sa.json]: [Errno 2] No such file or directory: '/files/gcp/keys/sa.json'
````
The same key directory is used to run system tests.
Do you have a solution that allows keys from the `/files/` directory to be used by gcloud?.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #9167: Improved cloud tool available in the trimmed down CI container
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #9167:
URL: https://github.com/apache/airflow/pull/9167#issuecomment-640370804
* The `/tmp` problem shuld be fixed now.
* I checked that anything looking for gcloud executable should be able to find it. Also I checked that the credential helper works:
```
git config --global credential.'https://source.developers.google.com'.helper gcloud
```
Note that there is bug in [documentation](https://cloud.google.com/source-repositories/docs/adding-repositories-as-remotes) and they incorrectly say that the credential helper is named `gcloud.sh` -> it should be just `gcloud`. I just sent feedback abou it it using their "send feedback' tool.
After I set it up - it works perfectly for me:
```
gcloud source repos list
REPO_NAME PROJECT_ID URL
airflow-breeze-config polidea-airflow https://source.developers.google.com/p/polidea-airflow/r/airflow-breeze-config
...
```
* I also added small improvement for MacOS users. Changing ownership of created files is only needed for Linux users of Breeze, so I added HOST_OS variable and only run the "ownership change" when your host system is Linux - that should eliminate unncessary overhead for Mac OS users.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle commented on a change in pull request #9167: Improved cloud tool available in the trimmed down CI container
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #9167:
URL: https://github.com/apache/airflow/pull/9167#discussion_r436593001
##########
File path: scripts/ci/prepare_tool_scripts.sh
##########
@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+set -euo pipefail
+
+function prepare_tool_script() {
+ IMAGE="${1}"
+ VOLUME="${2}"
Review comment:
The docker run command which I use for terraform is: `docker run --rm -it -v \${HOST_HOME}/.aws:/root/.aws -v \${HOST_HOME}/.azure:/root/.azure -v \${HOST_HOME}/.config:/root/.config -v \${HOST_AIRFLOW_SOURCES}:/opt/airflow -w /opt/airflow --env-file <(env | grep TF) ${TERRAFORM_IMAGE} \"\$@\"`
So I would also like to be able to set the working directory and maybe `--env-file` or if there is a different way of accessing the `TF` env vars, I am open to change it :/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle commented on a change in pull request #9167: Improved cloud tool available in the trimmed down CI container
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #9167:
URL: https://github.com/apache/airflow/pull/9167#discussion_r436610152
##########
File path: scripts/ci/prepare_tool_scripts.sh
##########
@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+set -euo pipefail
+
+function prepare_tool_script() {
+ IMAGE="${1}"
+ VOLUME="${2}"
Review comment:
Okay, I am gonna add it when I am adding terraform. :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle commented on a change in pull request #9167: Improved cloud tool available in the trimmed down CI container
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #9167:
URL: https://github.com/apache/airflow/pull/9167#discussion_r436595623
##########
File path: scripts/ci/prepare_tool_scripts.sh
##########
@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+set -euo pipefail
+
+function prepare_tool_script() {
+ IMAGE="${1}"
+ VOLUME="${2}"
Review comment:
@potiuk WDYT of having `function prepare_terraform_script` so we can keep the rest as it is.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle commented on a change in pull request #9167: Improved cloud tool available in the trimmed down CI container
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #9167:
URL: https://github.com/apache/airflow/pull/9167#discussion_r436593001
##########
File path: scripts/ci/prepare_tool_scripts.sh
##########
@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+set -euo pipefail
+
+function prepare_tool_script() {
+ IMAGE="${1}"
+ VOLUME="${2}"
Review comment:
The docker run command which I use for terraform is: `docker run --rm -it -v \${HOST_HOME}/.aws:/root/.aws -v \${HOST_HOME}/.azure:/root/.azure -v \${HOST_HOME}/.config:/root/.config -v \${HOST_AIRFLOW_SOURCES}:/opt/airflow -w /opt/airflow --env-file <(env | grep TF) ${TERRAFORM_IMAGE} \"\$@\"`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #9167: Improved cloud tool available in the trimmed down CI container
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #9167:
URL: https://github.com/apache/airflow/pull/9167#discussion_r436371833
##########
File path: scripts/ci/prepare_tool_scripts.sh
##########
@@ -0,0 +1,58 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+set -euo pipefail
+
+function prepare_tool_script() {
+ IMAGE="${1}"
+ VOLUME="${2}"
+ TOOL="${3}"
+ COMMAND="${4:-}"
+
+ TARGET_TOOL_PATH="/usr/bin/${TOOL}"
+ TARGET_TOOL_UPDATE_PATH="/usr/bin/${TOOL}-update"
+
+ cat >"${TARGET_TOOL_PATH}" <<EOF
+#!/usr/bin/env bash
Review comment:
It will not work for 'aws' for example - because the command is a bit different there. I think I prefer to generate separate scripts - also because in the future we might add other tools like that and they might have different commands to run. having separate script. I prefer to have different logic to generate those scripts.
For busybox it makes perfect sense because it is quite substantial binary and copying it over takes precious space. In our case it's not an issue and I prefer each tool is separate
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9167: Improved cloud tool available in the trimmed down CI container
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #9167:
URL: https://github.com/apache/airflow/pull/9167#discussion_r436366035
##########
File path: scripts/ci/prepare_tool_scripts.sh
##########
@@ -0,0 +1,58 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+set -euo pipefail
+
+function prepare_tool_script() {
+ IMAGE="${1}"
+ VOLUME="${2}"
+ TOOL="${3}"
+ COMMAND="${4:-}"
+
+ TARGET_TOOL_PATH="/usr/bin/${TOOL}"
+ TARGET_TOOL_UPDATE_PATH="/usr/bin/${TOOL}-update"
+
+ cat >"${TARGET_TOOL_PATH}" <<EOF
+#!/usr/bin/env bash
Review comment:
What do you think about creating one generic script and many symbolic links? This one script can select an image based on "basename "${BASH_SOURCE[0]}". If it detects suffix "-update" it will update the image. The script can be located in one file e.g. scripts/ci/in_container/run_cli_tools.sh.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #9167: Improved cloud tool available in the trimmed down CI container
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #9167:
URL: https://github.com/apache/airflow/pull/9167#issuecomment-640256880
I still have problems with gcloud. First of all, I never use host credentials for authorization. When I need to authorize, I use service account keys that are shared by my organization in the `/files/gcp/keys/` directory.
When I try to do this, I get the following message.
```
root@8d3ff81941e3:/opt/airflow# gcloud auth activate-service-account --key-file=/files/gcp/keys/sa.json
ERROR: (gcloud.auth.activate-service-account) Unable to read file [/files/gcp/keys/sa.json]: [Errno 2] No such file or directory: '/files/gcp/keys/sa.json'
````
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #9167: Improved cloud tool available in the trimmed down CI container
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #9167:
URL: https://github.com/apache/airflow/pull/9167#discussion_r436598357
##########
File path: scripts/ci/prepare_tool_scripts.sh
##########
@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+set -euo pipefail
+
+function prepare_tool_script() {
+ IMAGE="${1}"
+ VOLUME="${2}"
Review comment:
Sure 'prepare_terraform_script" sounds even better :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on pull request #9167: Improved cloud tool available in the trimmed down CI container
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on pull request #9167:
URL: https://github.com/apache/airflow/pull/9167#issuecomment-640256880
I still have problems with gcloud. First of all, I never use host credentials for authorization. When I need to authorize, I use service account keys that are shared by my organization in the `/files/gcp/keys/` directory.
When I try to do this, I get the following message.
```
root@8d3ff81941e3:/opt/airflow# gcloud auth activate-service-account --key-file=/files/gcp/keys/sa.json
ERROR: (gcloud.auth.activate-service-account) Unable to read file [/files/gcp/keys/sa.json]: [Errno 2] No such file or directory: '/files/gcp/keys/sa.json'
````
The same key directory is used to run system tests.
Do you have a solution that allows keys from the `/files/` directory to be used by gcloud?.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #9167: Improved cloud tool available in the trimmed down CI container
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #9167:
URL: https://github.com/apache/airflow/pull/9167#issuecomment-640362183
> I still have 2 problems
>
> First issue - Cloud Build system tests. This system tests use git and Cloud Source Repository. For authroization, git uses credentials helper According to the documentation, gcloud must be installed for this helper to work correctly.
>
> > The gcloud command-line tool must be in your $PATH for the credential helper scripts to work.
It should be in your path It's an executable now in '/usr/bin/gcloud". Can you please double-check it? The initial solution with alias could cause it but since we have scripts now, it should work.
> Second issue - Tests for Google Cloud Vision are sad. The corresponding files could not be found.
>
> ```
> [2020-06-07 21:49:51,189] {logging_command_executor.py:33} INFO - Executing: 'gsutil cp /tmp/tmp6v_shqlg.png gs://polidea-airflow-tests-37/imagetest-build-24790.png'
> [2020-06-07 21:49:54,540] {logging_command_executor.py:40} INFO - Stdout: CommandException: No URLs matched: /tmp/tmp6v_shqlg.png
>
> [2020-06-07 21:49:54,540] {logging_command_executor.py:41} INFO - Stderr:
> [2020-06-07 21:49:54,540] {logging_command_executor.py:43} ERROR - Error when executing gsutil cp /tmp/tmp6v_shqlg.png gs://polidea-airflow-tests-37/imagetest-build-24790.png
> ```
Easy to fix. We already pass tmp to inside the container from Airflow sources so that's another mount to be added. One moment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #9167: Improved cloud tool available in the trimmed down CI container
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #9167:
URL: https://github.com/apache/airflow/pull/9167#issuecomment-640424530
This shoudl be OK - just quarantined tests failed @mik-laj
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #9167: Improved cloud tool available in the trimmed down CI container
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #9167:
URL: https://github.com/apache/airflow/pull/9167#issuecomment-640285270
I still have 2 problems
First issue - Cloud Build system tests. This system tests use git and Cloud Source Repository. For authroization, git uses credentials helper According to the documentation, gcloud must be installed for this helper to work correctly.
> The gcloud command-line tool must be in your $PATH for the credential helper scripts to work.
Second issue - Tests for Google Cloud Vision are sad. The corresponding files could not be found.
```
[2020-06-07 21:49:51,189] {logging_command_executor.py:33} INFO - Executing: 'gsutil cp /tmp/tmp6v_shqlg.png gs://polidea-airflow-tests-37/imagetest-build-24790.png'
[2020-06-07 21:49:54,540] {logging_command_executor.py:40} INFO - Stdout: CommandException: No URLs matched: /tmp/tmp6v_shqlg.png
[2020-06-07 21:49:54,540] {logging_command_executor.py:41} INFO - Stderr:
[2020-06-07 21:49:54,540] {logging_command_executor.py:43} ERROR - Error when executing gsutil cp /tmp/tmp6v_shqlg.png gs://polidea-airflow-tests-37/imagetest-build-24790.png
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #9167: Improved cloud tool available in the trimmed down CI container
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #9167:
URL: https://github.com/apache/airflow/pull/9167#issuecomment-640281698
> I still have problems with gcloud. First of all, I never use host credentials for authorization. When I need to authorize, I use service account keys that are shared by my organization in the `/files/gcp/keys/` directory.
>
> When I try to do this, I get the following message.
>
> ```
> root@8d3ff81941e3:/opt/airflow# gcloud auth activate-service-account --key-file=/files/gcp/keys/sa.json
> ERROR: (gcloud.auth.activate-service-account) Unable to read file [/files/gcp/keys/sa.json]: [Errno 2] No such file or directory: '/files/gcp/keys/sa.json'
> ```
>
> The same key directory is used to run system tests.
>
> Do you have a solution that allows keys from the `/files/` directory to be used by gcloud?.
There it goes. Please check it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #9167: Improved cloud tool available in the trimmed down CI container
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #9167:
URL: https://github.com/apache/airflow/pull/9167#issuecomment-640280986
Ah yeah. We should add files too then.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle commented on a change in pull request #9167: Improved cloud tool available in the trimmed down CI container
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #9167:
URL: https://github.com/apache/airflow/pull/9167#discussion_r436589197
##########
File path: scripts/ci/prepare_tool_scripts.sh
##########
@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+set -euo pipefail
+
+function prepare_tool_script() {
+ IMAGE="${1}"
+ VOLUME="${2}"
Review comment:
So this `prepare_tool_script "hashicorp/terraform:latest" ".aws" ".azure" ".config/gcloud" terraform` would not work.
So `VOLUMES` varies and also `COMMAND`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle commented on a change in pull request #9167: Improved cloud tool available in the trimmed down CI container
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #9167:
URL: https://github.com/apache/airflow/pull/9167#discussion_r436595623
##########
File path: scripts/ci/prepare_tool_scripts.sh
##########
@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+set -euo pipefail
+
+function prepare_tool_script() {
+ IMAGE="${1}"
+ VOLUME="${2}"
Review comment:
@potiuk WDYT of having `function prepare_terraform_script` ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #9167: Improved cloud tool available in the trimmed down CI container
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #9167:
URL: https://github.com/apache/airflow/pull/9167#discussion_r436596340
##########
File path: scripts/ci/prepare_tool_scripts.sh
##########
@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+set -euo pipefail
+
+function prepare_tool_script() {
+ IMAGE="${1}"
+ VOLUME="${2}"
Review comment:
I think it woudl be easy enough (as a follow-up change :)) to turn VOLUME into list of volumes. And adding "env" or -w is also quite OK. I'a m happy to make it as a follow-up pr afterwards.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle commented on a change in pull request #9167: Improved cloud tool available in the trimmed down CI container
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #9167:
URL: https://github.com/apache/airflow/pull/9167#discussion_r436586382
##########
File path: scripts/ci/prepare_tool_scripts.sh
##########
@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+set -euo pipefail
+
+function prepare_tool_script() {
+ IMAGE="${1}"
+ VOLUME="${2}"
Review comment:
I have a problem with this. I want to add terraform but it needs more volumes to be added i.e. auth script folder for every provider you want to use with terraform.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9167: Improved cloud tool available in the trimmed down CI container
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #9167:
URL: https://github.com/apache/airflow/pull/9167#discussion_r436366633
##########
File path: scripts/ci/prepare_tool_scripts.sh
##########
@@ -0,0 +1,58 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+set -euo pipefail
+
+function prepare_tool_script() {
+ IMAGE="${1}"
+ VOLUME="${2}"
+ TOOL="${3}"
+ COMMAND="${4:-}"
+
+ TARGET_TOOL_PATH="/usr/bin/${TOOL}"
+ TARGET_TOOL_UPDATE_PATH="/usr/bin/${TOOL}-update"
+
+ cat >"${TARGET_TOOL_PATH}" <<EOF
+#!/usr/bin/env bash
Review comment:
Busbox works in a similar way.
>
> BusyBox is a multi-call binary. A multi-call binary is an executable program that performs the same job as more than one utility program. That means there is just a single BusyBox binary, but that single binary acts like a large number of utilities. This allows BusyBox to be smaller since all the built-in utility programs (we call them applets) can share code for many common operations.
>
> You can also invoke BusyBox by issuing a command as an argument on the command line. For example, entering
>
> /bin/busybox ls
> will also cause BusyBox to behave as 'ls'.
>
> Of course, adding '/bin/busybox' into every command would be painful. So most people will invoke BusyBox using links to the BusyBox binary.
>
> For example, entering
>
> ln -s /bin/busybox ls
> ./ls
> will cause BusyBox to behave as 'ls' (if the 'ls' command has been compiled into BusyBox). Generally speaking, you should never need to make all these links yourself, as the BusyBox build system will do this for you when you run the 'make install' command.
>
> If you invoke BusyBox with no arguments, it will provide you with a list of the applets that have been compiled into your BusyBox binary.
https://busybox.net/downloads/BusyBox.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org