You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/05/12 05:32:31 UTC
[GitHub] [arrow] anthonylouisbsb opened a new pull request #10300: ARROW-12699: [Java] Generate a jar compatible with Linux and MacOS for all Arrow components
anthonylouisbsb opened a new pull request #10300:
URL: https://github.com/apache/arrow/pull/10300
Change the build to generate the Arrow's libraries jar files containing the C++ shared libs both for Linux and macOS.
**Note**: It only generates the artifact jars for the components that depend on C++ libraries at the end of the build: gandiva, adapter/orc, and dataset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #10300: ARROW-12699: [CI][Packaging][Java] Generate a jar compatible with Linux and MacOS for all Arrow components
Posted by GitBox <gi...@apache.org>.
kou commented on pull request #10300:
URL: https://github.com/apache/arrow/pull/10300#issuecomment-841518230
Great!
Can we also generate `.jar` that doesn't need to bundle C++ libraries as the next step? Then we don't need to generate any `.jar` in our release process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] anthonylouisbsb commented on a change in pull request #10300: ARROW-12699: [CI][Packaging][Java] Generate a jar compatible with Linux and MacOS for all Arrow components
Posted by GitBox <gi...@apache.org>.
anthonylouisbsb commented on a change in pull request #10300:
URL: https://github.com/apache/arrow/pull/10300#discussion_r631054050
##########
File path: dev/tasks/jars/Dockerfile
##########
@@ -0,0 +1,124 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+FROM quay.io/pypa/manylinux2014_x86_64:latest
Review comment:
It sounds good, I think it is a good idea. I will apply the changes you suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #10300: ARROW-12699: [CI][Packaging][Java] Generate a jar compatible with Linux and MacOS for all Arrow components
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #10300:
URL: https://github.com/apache/arrow/pull/10300#discussion_r631821249
##########
File path: docker-compose.yml
##########
@@ -813,6 +814,24 @@ services:
source: .
target: "C:/arrow"
command: arrow\\ci\\scripts\\python_wheel_windows_test.bat
+
+ python-wheel-manylinux-2014-java:
Review comment:
We don't need to use the `python-wheel-manylinux-2014` prefix here despite that we depend on that image.
I think `java-<something>` would be more descriptive since this configuration is responsible for creating java jars with C++ libraries included.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #10300: ARROW-12699: [CI][Packaging][Java] Generate a jar compatible with Linux and MacOS for all Arrow components
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #10300:
URL: https://github.com/apache/arrow/pull/10300#discussion_r631042680
##########
File path: dev/tasks/jars/Dockerfile
##########
@@ -0,0 +1,124 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+FROM quay.io/pypa/manylinux2014_x86_64:latest
Review comment:
I'd suggest to create a child image which depends on the manylinux2014 image and install the additional packages required for gandiva there. This way we wouldn't need to check in and maintain that many scripts.
Something like the following should be enough:
#### ci/scripts/java_bundled_jar.sh
This file could be a customized version of `ci/scripts/python_wheel_manylinux_build.sh` or the [build-linux-cpp.sh]( https://github.com/apache/arrow/pull/10300/files#diff-eb09a993bfee5b11462aecb3e85df34161bbf5b82a792fceacf545adba5e6b8f) without multibuild.
#### ci/docker/java-bundled-jar.dockerfile:
```dockerfile
ARG base
FROM ${base}
RUN vcpkg install --clean-after-build <additional-dependencies>
ARG java=1.8.0
RUN yum install -y xz java-$java-openjdk-devel ccache flex wget && yum clean all
ENV JAVA_HOME=/usr/lib/jvm/java-$java-openjdk/
```
#### docker-compose.yml
```yml
x-hierarchy:
# ...
python-wheel-manylinux-2014:
- java-bundled-jar:
services:
# ...
java-bundled-jar:
image: ${REPO}:${ARCH}-java-bundled-jar-vcpkg-${VCPKG}
build:
args:
base: ${REPO}:${ARCH}-python-${PYTHON}-wheel-manylinux-2014-vcpkg-${VCPKG}
java: 1.8.0
context: .
dockerfile: ci/docker/java-bundled-jar.dockerfile
cache_from:
- ${REPO}:${ARCH}-java-bundled-jar-vcpkg-${VCPKG}
environment:
<<: *ccache
volumes:
- .:/arrow:delegated
- ${DOCKER_VOLUME_PREFIX}python-wheel-manylinux2014-ccache:/ccache:delegated
command: /arrow/ci/scripts/java_bundled_jar.sh
```
Then you can use archery to build and run this docker setup:
```console
archery docker build java-bundled-jar # just builds the images
archery docker run java-bundled-jar # builds and executes the image
```
How does it sound to you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] anthonylouisbsb commented on pull request #10300: ARROW-12699: [CI][Packaging][Java] Generate a jar compatible with Linux and MacOS for all Arrow components
Posted by GitBox <gi...@apache.org>.
anthonylouisbsb commented on pull request #10300:
URL: https://github.com/apache/arrow/pull/10300#issuecomment-841755233
@kou I opened a ticket https://issues.apache.org/jira/browse/ARROW-12801, and I will work on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #10300: ARROW-12699: [CI][Packaging][Java] Generate a jar compatible with Linux and MacOS for all Arrow components
Posted by GitBox <gi...@apache.org>.
kou commented on pull request #10300:
URL: https://github.com/apache/arrow/pull/10300#issuecomment-841873589
Thanks!
#10329 is related to it as you found.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #10300: ARROW-12699: [CI][Packaging][Java] Generate a jar compatible with Linux and MacOS for all Arrow components
Posted by GitBox <gi...@apache.org>.
kou commented on pull request #10300:
URL: https://github.com/apache/arrow/pull/10300#issuecomment-841873589
Thanks!
#10329 is related to it as you found.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] anthonylouisbsb commented on a change in pull request #10300: ARROW-12699: [CI][Packaging][Java] Generate a jar compatible with Linux and MacOS for all Arrow components
Posted by GitBox <gi...@apache.org>.
anthonylouisbsb commented on a change in pull request #10300:
URL: https://github.com/apache/arrow/pull/10300#discussion_r631893390
##########
File path: docker-compose.yml
##########
@@ -813,6 +814,24 @@ services:
source: .
target: "C:/arrow"
command: arrow\\ci\\scripts\\python_wheel_windows_test.bat
+
+ python-wheel-manylinux-2014-java:
Review comment:
I will change the name for `java-bundled-jars` as you suggested in the first review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] anthonylouisbsb commented on pull request #10300: ARROW-12699: [CI][Packaging][Java] Generate a jar compatible with Linux and MacOS for all Arrow components
Posted by GitBox <gi...@apache.org>.
anthonylouisbsb commented on pull request #10300:
URL: https://github.com/apache/arrow/pull/10300#issuecomment-841755233
@kou I opened a ticket https://issues.apache.org/jira/browse/ARROW-12801, and I will work on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #10300: ARROW-12699: [CI][Packaging][Java] Generate a jar compatible with Linux and MacOS for all Arrow components
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #10300:
URL: https://github.com/apache/arrow/pull/10300#issuecomment-841253009
@github-actions crossbow submit jars
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] anthonylouisbsb commented on a change in pull request #10300: ARROW-12699: [CI][Packaging][Java] Generate a jar compatible with Linux and MacOS for all Arrow components
Posted by GitBox <gi...@apache.org>.
anthonylouisbsb commented on a change in pull request #10300:
URL: https://github.com/apache/arrow/pull/10300#discussion_r632198627
##########
File path: dev/tasks/jars/build-java.sh
##########
@@ -23,12 +23,14 @@ CPP_BUILD_DIR=$GITHUB_WORKSPACE/arrow/dist/
pushd java
# build the entire project
- mvn clean install -q -DskipTests -P arrow-jni -Darrow.cpp.build.dir=$CPP_BUILD_DIR
- # test only gandiva
- mvn test -q -P arrow-jni -pl gandiva -Dgandiva.cpp.build.dir=$CPP_BUILD_DIR
+ mvn clean install -DskipTests -P arrow-jni -Darrow.cpp.build.dir=$CPP_BUILD_DIR
Review comment:
The script was moved to the folder
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] anthonylouisbsb commented on pull request #10300: ARROW-12699: [CI][Packaging][Java] Generate a jar compatible with Linux and MacOS for all Arrow components
Posted by GitBox <gi...@apache.org>.
anthonylouisbsb commented on pull request #10300:
URL: https://github.com/apache/arrow/pull/10300#issuecomment-840929681
@kszucs I applied all changes you suggested!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #10300: ARROW-12699: [CI][Packaging][Java] Generate a jar compatible with Linux and MacOS for all Arrow components
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #10300:
URL: https://github.com/apache/arrow/pull/10300#issuecomment-841354840
Thanks for the updates! I'm verifying it locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #10300: ARROW-12699: [Java] Generate a jar compatible with Linux and MacOS for all Arrow components
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #10300:
URL: https://github.com/apache/arrow/pull/10300#issuecomment-839457607
https://issues.apache.org/jira/browse/ARROW-12699
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #10300: ARROW-12699: [CI][Packaging][Java] Generate a jar compatible with Linux and MacOS for all Arrow components
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #10300:
URL: https://github.com/apache/arrow/pull/10300#discussion_r631814783
##########
File path: dev/tasks/jars/build-cpp-osx.sh
##########
@@ -29,21 +31,37 @@ pushd cpp
-DARROW_GANDIVA=ON \
-DARROW_GANDIVA_JAVA=ON \
-DARROW_GANDIVA_STATIC_LIBSTDCPP=ON \
+ -DARROW_ORC=ON \
+ -DARROW_JNI=ON \
+ -DARROW_PLASMA=ON \
+ -DARROW_PLASMA_JAVA_CLIENT=ON \
-DARROW_BUILD_TESTS=ON \
-DARROW_BUILD_UTILITIES=OFF \
- -DPARQUET_BUILD_ENCRYPTION=OFF \
- -DARROW_PARQUET=OFF \
- -DARROW_FILESYSTEM=OFF \
- -DARROW_DATASET=OFF \
+ -DPARQUET_REQUIRE_ENCRYPTION=OFF \
+ -DARROW_PARQUET=ON \
+ -DPARQUET_BUILD_EXAMPLES=OFF \
+ -DPARQUET_BUILD_EXECUTABLES=OFF \
+ -DARROW_FILESYSTEM=ON \
+ -DARROW_DATASET=ON \
-DARROW_BOOST_USE_SHARED=OFF \
-DARROW_PROTOBUF_USE_SHARED=OFF \
-DARROW_GFLAGS_USE_SHARED=OFF \
- -DARROW_OPENSSL_USE_SHARED=OFF"
+ -DARROW_OPENSSL_USE_SHARED=OFF \
+ -DARROW_BROTLI_USE_SHARED=OFF \
+ -DARROW_BZ2_USE_SHARED=OFF \
+ -DARROW_GRPC_USE_SHARED=OFF \
+ -DARROW_LZ4_USE_SHARED=OFF \
+ -DARROW_SNAPPY_USE_SHARED=OFF \
+ -DARROW_THRIFT_USE_SHARED=OFF \
+ -DARROW_UTF8PROC_USE_SHARED=OFF \
+ -DARROW_ZSTD_USE_SHARED=OFF"
cmake $CMAKE_FLAGS ..
make -j4
ctest
cp -L release/libgandiva_jni.dylib $GITHUB_WORKSPACE/arrow/dist
+ cp -L release/libarrow_dataset_jni.dylib $GITHUB_WORKSPACE/arrow/dist
Review comment:
Please factor out the `GITHUB_WORKSPACE` env variable and use arguments instead so we can more easily call this file locally, for example using arguments like in [cpp_build.sh](https://github.com/apache/arrow/blob/master/ci/scripts/cpp_build.sh#L22-L24)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #10300: ARROW-12699: [CI][Packaging][Java] Generate a jar compatible with Linux and MacOS for all Arrow components
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #10300:
URL: https://github.com/apache/arrow/pull/10300#discussion_r631042680
##########
File path: dev/tasks/jars/Dockerfile
##########
@@ -0,0 +1,124 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+FROM quay.io/pypa/manylinux2014_x86_64:latest
Review comment:
I'd suggest to create a child image which depends on the manylinux2014 image and install the additional packages required for gandiva there. This way we wouldn't need to check in and maintain that many scripts.
Something like the following should be enough:
#### ci/scripts/java_bundled_jar.sh
This file could be a customized version of `ci/scripts/python_wheel_manylinux_build.sh` or the [build-linux-cpp.sh]( https://github.com/apache/arrow/pull/10300/files#diff-eb09a993bfee5b11462aecb3e85df34161bbf5b82a792fceacf545adba5e6b8f) without multibuild.
#### ci/docker/java-bundled-jar.dockerfile:
```dockerfile
ARG base
FROM ${base}
RUN vcpkg install --clean-after-build <additional-dependencies>
ARG java=1.8.0
RUN yum install -y xz java-$java-openjdk-devel ccache flex wget && yum clean all
ENV JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk/
```
#### docker-compose.yml
```yml
x-hierarchy:
# ...
python-wheel-manylinux-2014:
- java-bundled-jar:
services:
# ...
java-bundled-jar:
image: ${REPO}:${ARCH}-java-bundled-jar-vcpkg-${VCPKG}
build:
args:
base: ${REPO}:${ARCH}-python-${PYTHON}-wheel-manylinux-2014-vcpkg-${VCPKG}
java: 1.8.0
context: .
dockerfile: ci/docker/java-bundled-jar.dockerfile
cache_from:
- ${REPO}:${ARCH}-java-bundled-jar-vcpkg-${VCPKG}
environment:
<<: *ccache
volumes:
- .:/arrow:delegated
- ${DOCKER_VOLUME_PREFIX}python-wheel-manylinux2014-ccache:/ccache:delegated
command: /arrow/ci/scripts/java_bundled_jar.sh
```
Then you can use archery to build and run this docker setup:
```console
archery docker build java-bundled-jar # just builds the images
archery docker run java-bundled-jar # builds and executes the image
```
How does it sound to you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #10300: ARROW-12699: [CI][Packaging][Java] Generate a jar compatible with Linux and MacOS for all Arrow components
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #10300:
URL: https://github.com/apache/arrow/pull/10300#discussion_r631813623
##########
File path: dev/tasks/jars/build-cpp-osx.sh
##########
@@ -17,9 +17,11 @@
# specific language governing permissions and limitations
# under the License.
-set -e
+set -ex
Review comment:
We maintain the OSX wheel scripts next to the manylinux ones in `ci/scripts`. Could you please move this file there as well for consistency?
Something like:
```
ci/scripts/java_<whatever>_manylinux_build.sh
ci/scripts/java_<whatever>_macos_build.sh
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #10300: ARROW-12699: [CI][Packaging][Java] Generate a jar compatible with Linux and MacOS for all Arrow components
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #10300:
URL: https://github.com/apache/arrow/pull/10300#issuecomment-841382779
Revision: f85f676eba72aff37182c3ec1f3d73368307bcfb
Submitted crossbow builds: [ursacomputing/crossbow @ actions-411](https://github.com/ursacomputing/crossbow/branches/all?query=actions-411)
|Task|Status|
|----|------|
|jars|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-411-github-jars)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #10300: ARROW-12699: [CI][Packaging][Java] Generate a jar compatible with Linux and MacOS for all Arrow components
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #10300:
URL: https://github.com/apache/arrow/pull/10300#discussion_r630921066
##########
File path: dev/tasks/jars/Dockerfile
##########
@@ -0,0 +1,124 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+FROM quay.io/pypa/manylinux2014_x86_64:latest
Review comment:
Thanks @anthonylouisbsb for working on this!
I wonder, could we reuse the [existing manylinux dockerfile](https://github.com/apache/arrow/blob/master/ci/docker/python-wheel-manylinux-201x.dockerfile) which uses vcpkg to build the dependencies?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #10300: ARROW-12699: [CI][Packaging][Java] Generate a jar compatible with Linux and MacOS for all Arrow components
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #10300:
URL: https://github.com/apache/arrow/pull/10300#discussion_r631820114
##########
File path: dev/tasks/jars/check-shared-dependencies.sh
##########
@@ -23,36 +23,51 @@ CPP_BUILD_DIR=$GITHUB_WORKSPACE/arrow/dist/
if [[ $OS_NAME == "linux" ]]; then
SO_DEP=ldd
+
GANDIVA_LIB="$CPP_BUILD_DIR"libgandiva_jni.so
+ DATASET_LIB="$CPP_BUILD_DIR"libarrow_dataset_jni.so
+ ORC_LIB="$CPP_BUILD_DIR"libarrow_orc_jni.so
+
WHITELIST=(linux-vdso libz librt libdl libpthread libstdc++ libm libgcc_s libc ld-linux-x86-64)
else
SO_DEP="otool -L"
+
GANDIVA_LIB="$CPP_BUILD_DIR"libgandiva_jni.dylib
- WHITELIST=(libgandiva_jni libz libncurses libSystem libc++)
+ DATASET_LIB="$CPP_BUILD_DIR"libarrow_dataset_jni.dylib
+ ORC_LIB="$CPP_BUILD_DIR"libarrow_orc_jni.dylib
+
+ WHITELIST=(libgandiva_jni libarrow_orc_jni libarrow_dataset_jni libz libncurses libSystem libc++)
fi
-# print the shared library dependencies
-$SO_DEP "$GANDIVA_LIB" | tee dependencies_temp_file.txt
+LIBRARIES=($GANDIVA_LIB $ORC_LIB $DATASET_LIB)
-if [[ $CHECK_SHARED_DEPENDENCIES ]] ; then
- # exit if any shared library not in whitelisted set is found
- echo "Checking shared dependencies"
+for library in "${LIBRARIES[@]}"
Review comment:
Seems like we could convert this to a bash function and call it from the macos and linux scripts:
```
check_dynamic_dependencies <queried-deps-using-ldd-or-otool> <allowed-ones>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs closed pull request #10300: ARROW-12699: [CI][Packaging][Java] Generate a jar compatible with Linux and MacOS for all Arrow components
Posted by GitBox <gi...@apache.org>.
kszucs closed pull request #10300:
URL: https://github.com/apache/arrow/pull/10300
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #10300: ARROW-12699: [CI][Packaging][Java] Generate a jar compatible with Linux and MacOS for all Arrow components
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #10300:
URL: https://github.com/apache/arrow/pull/10300#discussion_r631815612
##########
File path: dev/tasks/jars/build-java.sh
##########
@@ -23,12 +23,14 @@ CPP_BUILD_DIR=$GITHUB_WORKSPACE/arrow/dist/
pushd java
# build the entire project
- mvn clean install -q -DskipTests -P arrow-jni -Darrow.cpp.build.dir=$CPP_BUILD_DIR
- # test only gandiva
- mvn test -q -P arrow-jni -pl gandiva -Dgandiva.cpp.build.dir=$CPP_BUILD_DIR
+ mvn clean install -DskipTests -P arrow-jni -Darrow.cpp.build.dir=$CPP_BUILD_DIR
Review comment:
I think we can move this file to the `ci/scripts` directory as well using the same prefix as the cpp build scripts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] anthonylouisbsb commented on pull request #10300: ARROW-12699: [CI][Packaging][Java] Generate a jar compatible with Linux and MacOS for all Arrow components
Posted by GitBox <gi...@apache.org>.
anthonylouisbsb commented on pull request #10300:
URL: https://github.com/apache/arrow/pull/10300#issuecomment-840285843
@kszucs I applied the changes you suggested in the review. Thanks in advance for your help!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] anthonylouisbsb commented on a change in pull request #10300: ARROW-12699: [CI][Packaging][Java] Generate a jar compatible with Linux and MacOS for all Arrow components
Posted by GitBox <gi...@apache.org>.
anthonylouisbsb commented on a change in pull request #10300:
URL: https://github.com/apache/arrow/pull/10300#discussion_r631004663
##########
File path: dev/tasks/jars/Dockerfile
##########
@@ -0,0 +1,124 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+FROM quay.io/pypa/manylinux2014_x86_64:latest
Review comment:
@kszucs Thanks for your help! I did not notice that dockerfile.
I think I can use the [docker image from Arrow's Docker Hub repository](https://hub.docker.com/layers/apache/arrow-dev/amd64-python-3.7-wheel-manylinux-2014/images/sha256-e3691bcc875e7ce7ec633a0d2c9d8346c65f0f3bfabfd9f45ec1d2cbc32b3138?context=explore), but I need to update the Dockerfile that you sent in the message to import **LLVM** and **CLANG** using vcpkg and generate a new docker image in the hub.
Do you think is better to do it in this Pull Request or should I need to open a new one to apply that changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] anthonylouisbsb commented on a change in pull request #10300: ARROW-12699: [CI][Packaging][Java] Generate a jar compatible with Linux and MacOS for all Arrow components
Posted by GitBox <gi...@apache.org>.
anthonylouisbsb commented on a change in pull request #10300:
URL: https://github.com/apache/arrow/pull/10300#discussion_r632198549
##########
File path: dev/tasks/jars/check-shared-dependencies.sh
##########
@@ -23,36 +23,51 @@ CPP_BUILD_DIR=$GITHUB_WORKSPACE/arrow/dist/
if [[ $OS_NAME == "linux" ]]; then
SO_DEP=ldd
+
GANDIVA_LIB="$CPP_BUILD_DIR"libgandiva_jni.so
+ DATASET_LIB="$CPP_BUILD_DIR"libarrow_dataset_jni.so
+ ORC_LIB="$CPP_BUILD_DIR"libarrow_orc_jni.so
+
WHITELIST=(linux-vdso libz librt libdl libpthread libstdc++ libm libgcc_s libc ld-linux-x86-64)
else
SO_DEP="otool -L"
+
GANDIVA_LIB="$CPP_BUILD_DIR"libgandiva_jni.dylib
- WHITELIST=(libgandiva_jni libz libncurses libSystem libc++)
+ DATASET_LIB="$CPP_BUILD_DIR"libarrow_dataset_jni.dylib
+ ORC_LIB="$CPP_BUILD_DIR"libarrow_orc_jni.dylib
+
+ WHITELIST=(libgandiva_jni libarrow_orc_jni libarrow_dataset_jni libz libncurses libSystem libc++)
fi
-# print the shared library dependencies
-$SO_DEP "$GANDIVA_LIB" | tee dependencies_temp_file.txt
+LIBRARIES=($GANDIVA_LIB $ORC_LIB $DATASET_LIB)
-if [[ $CHECK_SHARED_DEPENDENCIES ]] ; then
- # exit if any shared library not in whitelisted set is found
- echo "Checking shared dependencies"
+for library in "${LIBRARIES[@]}"
Review comment:
The function was added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #10300: ARROW-12699: [CI][Packaging][Java] Generate a jar compatible with Linux and MacOS for all Arrow components
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #10300:
URL: https://github.com/apache/arrow/pull/10300#issuecomment-840572355
Thanks @anthonylouisbsb for the refactor, this seems pretty nice!
I only have organizational thoughts left to better align with the rest of the scripts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org