You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/08/19 08:41:57 UTC
[GitHub] [arrow] kou opened a new pull request, #13920: ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
kou opened a new pull request, #13920:
URL: https://github.com/apache/arrow/pull/13920
Because our official .jar packages are built in manylinux2014 image.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #13920: ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
Posted by GitBox <gi...@apache.org>.
pitrou commented on PR #13920:
URL: https://github.com/apache/arrow/pull/13920#issuecomment-1220806947
@kou Can you also display ccache statistics just like is done in `cpp_build.sh`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #13920: ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
Posted by GitBox <gi...@apache.org>.
kou commented on PR #13920:
URL: https://github.com/apache/arrow/pull/13920#issuecomment-1220699642
@pitrou manylinux2014 based image isn't slower than Debian 9 based image:
* manylinux2014: 25min: https://github.com/apache/arrow/runs/7918227379?check_suite_focus=true
* Debian 9: 44min: https://github.com/apache/arrow/runs/7917839841?check_suite_focus=true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #13920: ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #13920:
URL: https://github.com/apache/arrow/pull/13920#discussion_r950410826
##########
ci/scripts/java_jni_macos_build.sh:
##########
@@ -41,9 +41,15 @@ install_dir=${build_dir}/cpp-install
: ${ARROW_PLASMA:=ON}
: ${ARROW_PYTHON:=OFF}
: ${ARROW_S3:=ON}
+: ${ARROW_USE_CCACHE:=OFF}
Review Comment:
Why not, but you should perhaps also pass it to CMake below?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #13920: ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
Posted by GitBox <gi...@apache.org>.
kou commented on PR #13920:
URL: https://github.com/apache/arrow/pull/13920#issuecomment-1221246086
ccache statistics:
https://github.com/apache/arrow/runs/7929238651?check_suite_focus=true#step:6:2308a
```text
=== ccache statistics before build ===
cache directory /ccache
primary config /ccache/ccache.conf
secondary config (readonly) /usr/local/etc/ccache.conf
cache hit (direct) 0
cache hit (preprocessed) 0
cache miss 0
cache hit rate 0.00 %
cleanups performed 0
files in cache 0
cache size 0.0 kB
max cache size 1.0 GB
```
https://github.com/apache/arrow/runs/7929238651?check_suite_focus=true#step:6:3472
```text
=== ccache statistics after build ===
cache directory /ccache
primary config /ccache/ccache.conf
secondary config (readonly) /usr/local/etc/ccache.conf
stats updated Sat Aug 20 05:03:33 2022
cache hit (direct) 0
cache hit (preprocessed) 0
cache miss 98
cache hit rate 0.00 %
cleanups performed 0
files in cache 190
cache size 12.2 MB
max cache size 1.0 GB
```
Cache will be used after we merge this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13920: ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13920:
URL: https://github.com/apache/arrow/pull/13920#issuecomment-1220410603
https://issues.apache.org/jira/browse/ARROW-17451
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #13920: ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
Posted by GitBox <gi...@apache.org>.
kou commented on PR #13920:
URL: https://github.com/apache/arrow/pull/13920#issuecomment-1220745366
@github-actions crossbow submit java-jars
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #13920: ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #13920:
URL: https://github.com/apache/arrow/pull/13920#discussion_r950227111
##########
docker-compose.yml:
##########
@@ -1541,37 +1544,6 @@ services:
/arrow/ci/scripts/java_build.sh /arrow /build &&
/arrow/ci/scripts/java_test.sh /arrow /build"
- debian-java-jni:
- # Includes plasma test, jni for gandiva and orc, and C data interface.
- # Usage:
- # docker-compose build debian-java
- # docker-compose build debian-java-jni
- # docker-compose run debian-java-jni
- image: ${REPO}:${ARCH}-debian-9-java-${JDK}-maven-${MAVEN}-jni
- build:
- context: .
- dockerfile: ci/docker/linux-apt-jni.dockerfile
- cache_from:
- - ${REPO}:${ARCH}-debian-9-java-${JDK}-maven-${MAVEN}-jni
- args:
- llvm: ${LLVM}
- shm_size: *shm-size
- environment:
- <<: *ccache
- ARROW_BUILD_TESTS: "OFF"
- ARROW_S3: "OFF"
- ARROW_SUBSTRAIT: "OFF"
Review Comment:
Can you reuse these environment variables in the java-manylinux build?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #13920: ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
Posted by GitBox <gi...@apache.org>.
kou commented on PR #13920:
URL: https://github.com/apache/arrow/pull/13920#issuecomment-1221231690
@github-actions crossbow submit java-jars
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #13920: ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
Posted by GitBox <gi...@apache.org>.
lidavidm commented on PR #13920:
URL: https://github.com/apache/arrow/pull/13920#issuecomment-1220780183
I think it's fine to drop Debian if we have manylinux instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a diff in pull request #13920: ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
Posted by GitBox <gi...@apache.org>.
kou commented on code in PR #13920:
URL: https://github.com/apache/arrow/pull/13920#discussion_r950249787
##########
docker-compose.yml:
##########
@@ -1541,37 +1544,6 @@ services:
/arrow/ci/scripts/java_build.sh /arrow /build &&
/arrow/ci/scripts/java_test.sh /arrow /build"
- debian-java-jni:
- # Includes plasma test, jni for gandiva and orc, and C data interface.
- # Usage:
- # docker-compose build debian-java
- # docker-compose build debian-java-jni
- # docker-compose run debian-java-jni
- image: ${REPO}:${ARCH}-debian-9-java-${JDK}-maven-${MAVEN}-jni
- build:
- context: .
- dockerfile: ci/docker/linux-apt-jni.dockerfile
- cache_from:
- - ${REPO}:${ARCH}-debian-9-java-${JDK}-maven-${MAVEN}-jni
- args:
- llvm: ${LLVM}
- shm_size: *shm-size
- environment:
- <<: *ccache
- ARROW_BUILD_TESTS: "OFF"
- ARROW_S3: "OFF"
- ARROW_SUBSTRAIT: "OFF"
Review Comment:
We can use `ARROW_BUILD_TESTS=OFF` and `ARROW_SUBSTRAIT=OFF` and they are already used.
(But I want to enable `ARROW_BUILD_TESTS` in ARROW-17081 / #13911 to run a Dataset JNI test that uses `arrow_testing`.)
We can't use `ARROW_S3=OFF` because our JNI packages enable S3 recently: ARROW-16584 / #13157
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13920: ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13920:
URL: https://github.com/apache/arrow/pull/13920#issuecomment-1221245581
Revision: 64cf4c70fdeecc1eea56cab2105ae7da6d6d76dd
Submitted crossbow builds: [ursacomputing/crossbow @ actions-afee36cc35](https://github.com/ursacomputing/crossbow/branches/all?query=actions-afee36cc35)
|Task|Status|
|----|------|
|java-jars|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-afee36cc35-github-java-jars)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #13920: ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
Posted by GitBox <gi...@apache.org>.
kou commented on PR #13920:
URL: https://github.com/apache/arrow/pull/13920#issuecomment-1220423732
Let's confirm CI time of https://github.com/apache/arrow/runs/7914727219?check_suite_focus=true .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #13920: ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
Posted by GitBox <gi...@apache.org>.
kou commented on PR #13920:
URL: https://github.com/apache/arrow/pull/13920#issuecomment-1223349247
It seems that there is no objection.
I'll merge this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13920: ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13920:
URL: https://github.com/apache/arrow/pull/13920#issuecomment-1221231843
Revision: 3e7930da2259f7fe52a940e2fefaa8c6b3b422f4
Submitted crossbow builds: [ursacomputing/crossbow @ actions-14ab753fc3](https://github.com/ursacomputing/crossbow/branches/all?query=actions-14ab753fc3)
|Task|Status|
|----|------|
|java-jars|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-14ab753fc3-github-java-jars)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a diff in pull request #13920: ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
Posted by GitBox <gi...@apache.org>.
kou commented on code in PR #13920:
URL: https://github.com/apache/arrow/pull/13920#discussion_r950658921
##########
ci/scripts/java_jni_macos_build.sh:
##########
@@ -41,9 +41,15 @@ install_dir=${build_dir}/cpp-install
: ${ARROW_PLASMA:=ON}
: ${ARROW_PYTHON:=OFF}
: ${ARROW_S3:=ON}
+: ${ARROW_USE_CCACHE:=OFF}
Review Comment:
Ah, I changed wrong file... I should have changed `java_jni_manylinux_build.sh`...
I've passed it to CMake arguments. I've updated `java_jni_manylinux_build.sh` too.
I chose `ARROW_USE_CCACHE:=OFF` here because `cpp_build.sh` uses it. I set `ARROW_USE_CCACHE=ON` in caller like other jobs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #13920: ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
Posted by GitBox <gi...@apache.org>.
kou commented on PR #13920:
URL: https://github.com/apache/arrow/pull/13920#issuecomment-1221245475
@github-actions crossbow submit java-jars
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #13920: ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
Posted by GitBox <gi...@apache.org>.
pitrou commented on PR #13920:
URL: https://github.com/apache/arrow/pull/13920#issuecomment-1220726429
@lwhite1 @lidavidm What do you think about removing Debian-based CI for Java JNI?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a diff in pull request #13920: ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
Posted by GitBox <gi...@apache.org>.
kou commented on code in PR #13920:
URL: https://github.com/apache/arrow/pull/13920#discussion_r950249787
##########
docker-compose.yml:
##########
@@ -1541,37 +1544,6 @@ services:
/arrow/ci/scripts/java_build.sh /arrow /build &&
/arrow/ci/scripts/java_test.sh /arrow /build"
- debian-java-jni:
- # Includes plasma test, jni for gandiva and orc, and C data interface.
- # Usage:
- # docker-compose build debian-java
- # docker-compose build debian-java-jni
- # docker-compose run debian-java-jni
- image: ${REPO}:${ARCH}-debian-9-java-${JDK}-maven-${MAVEN}-jni
- build:
- context: .
- dockerfile: ci/docker/linux-apt-jni.dockerfile
- cache_from:
- - ${REPO}:${ARCH}-debian-9-java-${JDK}-maven-${MAVEN}-jni
- args:
- llvm: ${LLVM}
- shm_size: *shm-size
- environment:
- <<: *ccache
- ARROW_BUILD_TESTS: "OFF"
- ARROW_S3: "OFF"
- ARROW_SUBSTRAIT: "OFF"
Review Comment:
We can use `ARROW_BUILD_TEST=OFF` and `ARROW_SUBSTRAIT=OFF` and they are already used.
(But I want to enable `ARROW_BUILD_TEST` in ARROW-17081 / #13911 to run a Dataset JNI test that uses `arrow_testing`.)
We can't use `ARROW_S3=OFF` because our JNI packages enable S3 recently: ARROW-16584 / #13157
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13920: ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13920:
URL: https://github.com/apache/arrow/pull/13920#issuecomment-1220410627
:warning: Ticket **has not been started in JIRA**, please click 'Start Progress'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13920: ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13920:
URL: https://github.com/apache/arrow/pull/13920#issuecomment-1220746621
Revision: b319661cb587d3ae754f778211104d4e1382f96e
Submitted crossbow builds: [ursacomputing/crossbow @ actions-f9f22f6178](https://github.com/ursacomputing/crossbow/branches/all?query=actions-f9f22f6178)
|Task|Status|
|----|------|
|java-jars|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-f9f22f6178-github-java-jars)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou merged pull request #13920: ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
Posted by GitBox <gi...@apache.org>.
kou merged PR #13920:
URL: https://github.com/apache/arrow/pull/13920
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #13920: ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #13920:
URL: https://github.com/apache/arrow/pull/13920#issuecomment-1223785516
Benchmark runs are scheduled for baseline = 0600bb27d6b287d91c2073763e785232be67bb30 and contender = 9882df58e8b82640f1cde090b68d5da7cf37bdb4. 9882df58e8b82640f1cde090b68d5da7cf37bdb4 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/b55d1d15db0f4ec8a2a80c54592ef093...cebdb43fa7f545d78461eb05f993ea97/)
[Failed :arrow_down:0.27% :arrow_up:0.0%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/3016d71144734edc95206db2738f1a18...a250f4a02e694a5bb027ad9e8b19a54f/)
[Failed :arrow_down:0.55% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/62724af4f5214fdb965e43eba46a6d0d...9c39a4e883514e44b6f493e8111126d9/)
[Finished :arrow_down:0.28% :arrow_up:0.04%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/9f329e5ce14d42e49633c5f503165cbe...420b7e9e87874dc59e0cc1dae7f534c7/)
Buildkite builds:
[Finished] [`9882df58` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1352)
[Failed] [`9882df58` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1368)
[Failed] [`9882df58` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1350)
[Finished] [`9882df58` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1367)
[Finished] [`0600bb27` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1351)
[Failed] [`0600bb27` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1367)
[Failed] [`0600bb27` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1349)
[Finished] [`0600bb27` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1366)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org