You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/10/25 14:02:53 UTC
[GitHub] [arrow] pitrou opened a new pull request, #14501: ARROW-18157: [Dev][Archery] "archery docker run" sets env var to None when inherited
pitrou opened a new pull request, #14501:
URL: https://github.com/apache/arrow/pull/14501
This would occur when:
* an environment variable is set to inherited in `docker-compose.yml`
* the variable isn't set in the calling environment
* Archery would use docker instead of docker-compose (for example because it's a GPU job)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #14501: ARROW-18157: [Dev][Archery] "archery docker run" sets env var to None when inherited
Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #14501:
URL: https://github.com/apache/arrow/pull/14501#discussion_r1005512712
##########
dev/archery/archery/docker/core.py:
##########
@@ -342,7 +342,8 @@ def run(self, service_name, command=None, *, env=None, volumes=None,
# append env variables from the compose conf
for k, v in service.get('environment', {}).items():
- args.extend(['-e', '{}={}'.format(k, v)])
+ if v is not None:
+ args.extend(['-e', '{}={}'.format(k, v)])
Review Comment:
I tested again locally and it works (using `SETUPTOOLS_SCM_PRETEND_VERSION=1.2.3 archery --debug docker run ubuntu-cuda-python bash`). Do you get a different result?
```console
$ SETUPTOOLS_SCM_PRETEND_VERSION=1.2.3 archery --debug docker run ubuntu-cuda-python bash
DEBUG:archery:Executing `['git', '-C', '/home/antoine/arrow/dev/cpp', 'rev-parse', '--show-toplevel']`
DEBUG:archery:Executing `['docker-compose', '--file', '/home/antoine/arrow/dev/docker-compose.yml', 'config']`
DEBUG:archery:Executing `['docker-compose', '--file', '/home/antoine/arrow/dev/docker-compose.yml', 'pull', '--ignore-pull-failures', 'ubuntu-cuda-cpp']`
Pulling ubuntu-cuda-cpp ... done
DEBUG:archery:Executing `['docker-compose', '--file', '/home/antoine/arrow/dev/docker-compose.yml', 'pull', '--ignore-pull-failures', 'ubuntu-cuda-python']`
Pulling ubuntu-cuda-python ... done
DEBUG:archery:Executing `['docker-compose', '--file', '/home/antoine/arrow/dev/docker-compose.yml', 'build', '--build-arg', 'BUILDKIT_INLINE_CACHE=1', 'ubuntu-cuda-cpp']`
...
DEBUG:archery:Executing `['docker-compose', '--file', '/home/antoine/arrow/dev/docker-compose.yml', 'build', '--build-arg', 'BUILDKIT_INLINE_CACHE=1', 'ubuntu-cuda-python']`
...
DEBUG:archery:Executing `['docker', 'run', '--rm', '--gpus', 'all', '--shm-size', '2G', '-e', 'ARROW_BUILD_UTILITIES=OFF', '-e', 'ARROW_COMPUTE=ON', '-e', 'ARROW_CSV=ON', '-e', 'ARROW_CUDA=ON', '-e', 'ARROW_DATASET=ON', '-e', 'ARROW_ENABLE_TIMING_TESTS=OFF', '-e', 'ARROW_FILESYSTEM=ON', '-e', 'ARROW_GANDIVA=OFF', '-e', 'ARROW_GCS=OFF', '-e', 'ARROW_HDFS=ON', '-e', 'ARROW_JEMALLOC=ON', '-e', 'ARROW_JSON=ON', '-e', 'ARROW_ORC=OFF', '-e', 'ARROW_PARQUET=ON', '-e', 'ARROW_PLASMA=OFF', '-e', 'ARROW_S3=OFF', '-e', 'ARROW_SUBSTRAIT=OFF', '-e', 'ARROW_WITH_OPENTELEMETRY=OFF', '-e', 'CCACHE_COMPILERCHECK=content', '-e', 'CCACHE_COMPRESS=1', '-e', 'CCACHE_COMPRESSLEVEL=6', '-e', 'CCACHE_DIR=/ccache', '-e', 'CCACHE_MAXSIZE=1G', '-e', 'SCCACHE_S3_KEY_PREFIX=sccache', '-e', 'SETUPTOOLS_SCM_PRETEND_VERSION=1.2.3', '-v', '/home/antoine/arrow/dev:/arrow:delegated', '-v', 'ubuntu-ccache:/ccache:delegated', '-it', 'apache/arrow-dev:amd64-ubuntu-20.04-cuda-11.0.3-python-3', 'bash']`
root@6e458d15abf9:/# echo $SETUPTOOLS_SCM_PRETEND_VERSION
1.2.3
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #14501: ARROW-18157: [Dev][Archery] "archery docker run" sets env var to None when inherited
Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #14501:
URL: https://github.com/apache/arrow/pull/14501#discussion_r1005306569
##########
dev/archery/archery/docker/core.py:
##########
@@ -342,7 +342,8 @@ def run(self, service_name, command=None, *, env=None, volumes=None,
# append env variables from the compose conf
for k, v in service.get('environment', {}).items():
- args.extend(['-e', '{}={}'.format(k, v)])
+ if v is not None:
+ args.extend(['-e', '{}={}'.format(k, v)])
Review Comment:
Are you sure? It didn't seem needed when I tried locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a diff in pull request #14501: ARROW-18157: [Dev][Archery] "archery docker run" sets env var to None when inherited
Posted by GitBox <gi...@apache.org>.
kou commented on code in PR #14501:
URL: https://github.com/apache/arrow/pull/14501#discussion_r1004979472
##########
dev/archery/archery/docker/core.py:
##########
@@ -342,7 +342,8 @@ def run(self, service_name, command=None, *, env=None, volumes=None,
# append env variables from the compose conf
for k, v in service.get('environment', {}).items():
- args.extend(['-e', '{}={}'.format(k, v)])
+ if v is not None:
+ args.extend(['-e', '{}={}'.format(k, v)])
Review Comment:
It seems that we need to get an environment variable value when `v` is `None` to emulate the `docker-compose` behavior:
```suggestion
if v is None:
v = os.environ.get(k)
if v is not None:
args.extend(['-e', '{}={}'.format(k, v)])
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #14501: ARROW-18157: [Dev][Archery] "archery docker run" sets env var to None when inherited
Posted by GitBox <gi...@apache.org>.
pitrou commented on PR #14501:
URL: https://github.com/apache/arrow/pull/14501#issuecomment-1290613210
cc @assignUser @raulcd
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a diff in pull request #14501: ARROW-18157: [Dev][Archery] "archery docker run" sets env var to None when inherited
Posted by GitBox <gi...@apache.org>.
kou commented on code in PR #14501:
URL: https://github.com/apache/arrow/pull/14501#discussion_r1006454938
##########
dev/archery/archery/docker/core.py:
##########
@@ -342,7 +342,8 @@ def run(self, service_name, command=None, *, env=None, volumes=None,
# append env variables from the compose conf
for k, v in service.get('environment', {}).items():
- args.extend(['-e', '{}={}'.format(k, v)])
+ if v is not None:
+ args.extend(['-e', '{}={}'.format(k, v)])
Review Comment:
Ah, sorry. I just saw only code around this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #14501: ARROW-18157: [Dev][Archery] "archery docker run" sets env var to None when inherited
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #14501:
URL: https://github.com/apache/arrow/pull/14501#issuecomment-1294922636
Benchmark runs are scheduled for baseline = 7f6c5aeb5388936709642e48aed6419d1e2144a6 and contender = ede0b594468b997212f599455620649925eab8fd. ede0b594468b997212f599455620649925eab8fd is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/f0ee8f64054f4dd59c06ef768a1d56b6...95b2f0d82df6491f98799c7034c19a90/)
[Failed :arrow_down:0.0% :arrow_up:0.0%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/a29ff626199b4bbc8cbb0a73c902193a...efc7ec36e2f84cf495e26c7d8fba271a/)
[Finished :arrow_down:0.27% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/13856c91307a44f69eb47c5bffff5e3e...6e557d94d5884802a941f7f779bd0b1b/)
[Finished :arrow_down:0.21% :arrow_up:0.04%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/5aae21170bc04d2f849305376c245b69...141a089661e84d32940c13d85c1a46fe/)
Buildkite builds:
[Finished] [`ede0b594` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1765)
[Failed] [`ede0b594` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1786)
[Finished] [`ede0b594` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1752)
[Finished] [`ede0b594` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1778)
[Finished] [`7f6c5aeb` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1764)
[Failed] [`7f6c5aeb` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1785)
[Finished] [`7f6c5aeb` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1751)
[Finished] [`7f6c5aeb` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1777)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on a diff in pull request #14501: ARROW-18157: [Dev][Archery] "archery docker run" sets env var to None when inherited
Posted by GitBox <gi...@apache.org>.
assignUser commented on code in PR #14501:
URL: https://github.com/apache/arrow/pull/14501#discussion_r1005572477
##########
dev/archery/archery/docker/core.py:
##########
@@ -342,7 +342,8 @@ def run(self, service_name, command=None, *, env=None, volumes=None,
# append env variables from the compose conf
for k, v in service.get('environment', {}).items():
- args.extend(['-e', '{}={}'.format(k, v)])
+ if v is not None:
+ args.extend(['-e', '{}={}'.format(k, v)])
Review Comment:
archery already gets the enclosing env vars:
https://github.com/apache/arrow/blob/master/dev/archery/archery/docker/core.py#L81
so no need to do it manually here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #14501: ARROW-18157: [Dev][Archery] "archery docker run" sets env var to None when inherited
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #14501:
URL: https://github.com/apache/arrow/pull/14501#issuecomment-1290646128
No, the behavior of setting it to `None` is contra to how docker-compose handles things. Having a key with an empty value means "inherit this from the calling env, if unset then ignore it". That should happen with archery in the same way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou merged pull request #14501: ARROW-18157: [Dev][Archery] "archery docker run" sets env var to None when inherited
Posted by GitBox <gi...@apache.org>.
pitrou merged PR #14501:
URL: https://github.com/apache/arrow/pull/14501
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd commented on pull request #14501: ARROW-18157: [Dev][Archery] "archery docker run" sets env var to None when inherited
Posted by GitBox <gi...@apache.org>.
raulcd commented on PR #14501:
URL: https://github.com/apache/arrow/pull/14501#issuecomment-1290639887
Shouldn't we just fail if the environment variable was not found? It feels like a problem on the configuration as if the env var is not needed why was there on the first place?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #14501: ARROW-18157: [Dev][Archery] "archery docker run" sets env var to None when inherited
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #14501:
URL: https://github.com/apache/arrow/pull/14501#issuecomment-1290885091
:warning: Ticket **has not been started in JIRA**, please click 'Start Progress'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #14501: ARROW-18157: [Dev][Archery] "archery docker run" sets env var to None when inherited
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #14501:
URL: https://github.com/apache/arrow/pull/14501#issuecomment-1290885052
https://issues.apache.org/jira/browse/ARROW-18157
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org