You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/10/23 21:15:12 UTC
[GitHub] [arrow] kou commented on a diff in pull request #14472: ARROW-18042: [Java] Distribute Apple M1 compatible JNI libraries via mavencentral
kou commented on code in PR #14472:
URL: https://github.com/apache/arrow/pull/14472#discussion_r1002771022
##########
dev/tasks/java-jars/github.yml:
##########
@@ -48,47 +50,65 @@ jobs:
{% endif %}
build-cpp-macos:
- name: Build C++ libraries macOS
- runs-on: macos-latest
+ {% set arch = '${{ matrix.platform.arch }}' %}
+ name: Build C++ libraries macOS {{ arch }}
+ runs-on: {{ '${{ matrix.platform.runs_on }}' }}
+ strategy:
+ fail-fast: false
+ matrix:
+ platform:
+ - { runs_on: 'macos-latest', arch: "x86_64"}
+ - { runs_on: ["self-hosted", "macOS", "arm64", "devops-managed"], arch: "aarch_64" }
env:
MACOSX_DEPLOYMENT_TARGET: "10.13"
steps:
{{ macros.github_checkout_arrow()|indent }}
- {{ macros.github_install_archery()|indent }}
+ - name: Install Archery
+ shell: bash
+ run: python3 -m pip install -e arrow/dev/archery[all]
- name: Install dependencies
run: |
- brew install --overwrite git
+ # brew numpy install fails on gh runner macos-11
+ # and is not required for this build
+ sed -i -e '/brew "numpy"/d' arrow/cpp/Brewfile
Review Comment:
We can remove `numpy` from `cpp/Brewfile` because we moved `cpp/src/arrow/python/` to `python/pyarrow/src/arrow/python/`.
##########
dev/tasks/java-jars/github.yml:
##########
@@ -117,44 +142,43 @@ jobs:
name: Build jar files
runs-on: macos-latest
needs:
- - build-cpp-ubuntu
- build-cpp-macos
+ - build-cpp-ubuntu
- build-cpp-windows
steps:
{{ macros.github_checkout_arrow(fetch_depth=0)|indent }}
- - name: Download Linux C++ Libraries
- uses: actions/download-artifact@v2
+ - name: Download Libraries
+ uses: actions/download-artifact@v3
with:
- name: ubuntu-shared-lib
- - name: Download macOS C++ libraries
- uses: actions/download-artifact@v2
- with:
- name: macos-shared-lib
- - name: Download Windows C++ libraries
- uses: actions/download-artifact@v2
- with:
- name: windows-shared-lib
- - name: Descompress artifacts
+ path: artifacts
+ - name: Decompress artifacts
run: |
+ mv artifacts/*/*.tar.gz .
tar -xvzf arrow-shared-libs-linux.tar.gz
- tar -xvzf arrow-shared-libs-macos.tar.gz
+ tar -xvzf arrow-shared-libs-macos-x86_64.tar.gz
+ tar -xvzf arrow-shared-libs-macos-aarch_64.tar.gz
tar -xvzf arrow-shared-libs-windows.tar.gz
- name: Test that shared libraries exist
run: |
set -x
- test -f arrow/java-dist/libarrow_cdata_jni.dylib
- test -f arrow/java-dist/libarrow_cdata_jni.so
+ test -f arrow/java-dist/x86_64/libarrow_orc_jni.so
+ test -f arrow/java-dist/x86_64/libarrow_cdata_jni.so
+ test -f arrow/java-dist/x86_64/libgandiva_jni.so
+ test -f arrow/java-dist/x86_64/libarrow_dataset_jni.so
+ test -f arrow/java-dist/x86_64/libplasma_java.so
+ test -f arrow/java-dist/x86_64/libarrow_cdata_jni.dylib
+ test -f arrow/java-dist/x86_64/libplasma_java.dylib
+ test -f arrow/java-dist/x86_64/libarrow_dataset_jni.dylib
+ test -f arrow/java-dist/x86_64/libgandiva_jni.dylib
+ test -f arrow/java-dist/x86_64/libarrow_orc_jni.dylib
+ test -f arrow/java-dist/aarch_64/libarrow_cdata_jni.dylib
+ test -f arrow/java-dist/aarch_64/libplasma_java.dylib
+ test -f arrow/java-dist/aarch_64/libarrow_dataset_jni.dylib
+ test -f arrow/java-dist/aarch_64/libgandiva_jni.dylib
+ test -f arrow/java-dist/aarch_64/libarrow_orc_jni.dylib
test -f arrow/java-dist/arrow_cdata_jni.dll
Review Comment:
Could you also move `.dll`s to `x86_64/`?
##########
dev/tasks/java-jars/github.yml:
##########
@@ -48,47 +50,65 @@ jobs:
{% endif %}
build-cpp-macos:
- name: Build C++ libraries macOS
- runs-on: macos-latest
+ {% set arch = '${{ matrix.platform.arch }}' %}
+ name: Build C++ libraries macOS {{ arch }}
+ runs-on: {{ '${{ matrix.platform.runs_on }}' }}
+ strategy:
+ fail-fast: false
+ matrix:
+ platform:
+ - { runs_on: 'macos-latest', arch: "x86_64"}
+ - { runs_on: ["self-hosted", "macOS", "arm64", "devops-managed"], arch: "aarch_64" }
env:
MACOSX_DEPLOYMENT_TARGET: "10.13"
steps:
{{ macros.github_checkout_arrow()|indent }}
- {{ macros.github_install_archery()|indent }}
+ - name: Install Archery
+ shell: bash
+ run: python3 -m pip install -e arrow/dev/archery[all]
- name: Install dependencies
run: |
- brew install --overwrite git
+ # brew numpy install fails on gh runner macos-11
+ # and is not required for this build
+ sed -i -e '/brew "numpy"/d' arrow/cpp/Brewfile
brew bundle --file=arrow/cpp/Brewfile
# We want to link aws-sdk-cpp statically but Homebrew's
# aws-sdk-cpp provides only shared library. If we have
# Homebrew's aws-sdk-cpp, our build mix Homebrew's
# aws-sdk-cpp and bundled aws-sdk-cpp. We uninstall Homebrew's
# aws-sdk-cpp to ensure using only bundled aws-sdk-cpp.
brew uninstall aws-sdk-cpp
- - name: Setup ccache
- run: |
- arrow/ci/scripts/ccache_setup.sh
+ brew install openjdk@11
+ brew install sccache
Review Comment:
How about adding `java/Brewfile` that includes `brew "openjdk@11"` and `brew "sccache"`?
##########
dev/tasks/java-jars/github.yml:
##########
@@ -48,47 +50,65 @@ jobs:
{% endif %}
build-cpp-macos:
- name: Build C++ libraries macOS
- runs-on: macos-latest
+ {% set arch = '${{ matrix.platform.arch }}' %}
+ name: Build C++ libraries macOS {{ arch }}
+ runs-on: {{ '${{ matrix.platform.runs_on }}' }}
+ strategy:
+ fail-fast: false
+ matrix:
+ platform:
+ - { runs_on: 'macos-latest', arch: "x86_64"}
+ - { runs_on: ["self-hosted", "macOS", "arm64", "devops-managed"], arch: "aarch_64" }
env:
MACOSX_DEPLOYMENT_TARGET: "10.13"
steps:
{{ macros.github_checkout_arrow()|indent }}
- {{ macros.github_install_archery()|indent }}
+ - name: Install Archery
+ shell: bash
+ run: python3 -m pip install -e arrow/dev/archery[all]
- name: Install dependencies
run: |
- brew install --overwrite git
+ # brew numpy install fails on gh runner macos-11
+ # and is not required for this build
+ sed -i -e '/brew "numpy"/d' arrow/cpp/Brewfile
brew bundle --file=arrow/cpp/Brewfile
# We want to link aws-sdk-cpp statically but Homebrew's
# aws-sdk-cpp provides only shared library. If we have
# Homebrew's aws-sdk-cpp, our build mix Homebrew's
# aws-sdk-cpp and bundled aws-sdk-cpp. We uninstall Homebrew's
# aws-sdk-cpp to ensure using only bundled aws-sdk-cpp.
brew uninstall aws-sdk-cpp
- - name: Setup ccache
- run: |
- arrow/ci/scripts/ccache_setup.sh
+ brew install openjdk@11
+ brew install sccache
- name: Build C++ libraries
+ env:
+ {{ macros.github_set_sccache_envvars()|indent(8) }}
run: |
set -e
+ # make brew java available to cmake
+ if [ {{ '"${{ matrix.platform.arch }}"' }} = "aarch_64" ]; then
+ export JAVA_HOME=/opt/homebrew/opt/openjdk@11/libexec/openjdk.jdk/Contents/Home
Review Comment:
```suggestion
export JAVA_HOME=$(brew --prefix openjdk@11)/libexec/openjdk.jdk/Contents/Home
```
##########
dev/tasks/java-jars/github.yml:
##########
@@ -164,6 +188,18 @@ jobs:
arrow/ci/scripts/java_full_build.sh \
$GITHUB_WORKSPACE/arrow \
$GITHUB_WORKSPACE/arrow/java-dist
+ arrow/ci/scripts/java_full_build.sh \
+ $GITHUB_WORKSPACE/arrow \
+ $GITHUB_WORKSPACE/arrow/java-dist/x86_64
+ arrow/ci/scripts/java_full_build.sh \
+ $GITHUB_WORKSPACE/arrow \
+ $GITHUB_WORKSPACE/arrow/java-dist/aarch_64
Review Comment:
Why do we need this?
I think that we need to bundle all `.so`s/`.dylib`s/.`dll`s to one `.jar`.
It seems that we don't need to do them because our pattern matches all `java-dist/**/*arrow_cdata_jni.*`: https://github.com/apache/arrow/blob/master/java/c/pom.xml#L70-L79
##########
docker-compose.yml:
##########
@@ -1008,10 +1008,10 @@ services:
- ${DOCKER_VOLUME_PREFIX}python-wheel-manylinux2014-ccache:/ccache:delegated
command:
["pip install -e /arrow/dev/archery && \
- /arrow/ci/scripts/java_jni_manylinux_build.sh /arrow /build /arrow/java-dist && \
+ /arrow/ci/scripts/java_jni_manylinux_build.sh /arrow /build /arrow/java-dist/x86_64 && \
source /opt/rh/rh-maven35/enable && \
- /arrow/ci/scripts/java_build.sh /arrow /build /arrow/java-dist && \
- /arrow/ci/scripts/java_test.sh /arrow /build /arrow/java-dist"]
+ /arrow/ci/scripts/java_build.sh /arrow /build /arrow/java-dist/x86_64 && \
+ /arrow/ci/scripts/java_test.sh /arrow /build /arrow/java-dist/x86_64"]
Review Comment:
I think that we don't need to append `/x86_64` here.
##########
dev/tasks/java-jars/github.yml:
##########
@@ -48,47 +50,65 @@ jobs:
{% endif %}
build-cpp-macos:
- name: Build C++ libraries macOS
- runs-on: macos-latest
+ {% set arch = '${{ matrix.platform.arch }}' %}
+ name: Build C++ libraries macOS {{ arch }}
+ runs-on: {{ '${{ matrix.platform.runs_on }}' }}
+ strategy:
+ fail-fast: false
+ matrix:
+ platform:
+ - { runs_on: 'macos-latest', arch: "x86_64"}
+ - { runs_on: ["self-hosted", "macOS", "arm64", "devops-managed"], arch: "aarch_64" }
env:
MACOSX_DEPLOYMENT_TARGET: "10.13"
steps:
{{ macros.github_checkout_arrow()|indent }}
- {{ macros.github_install_archery()|indent }}
+ - name: Install Archery
+ shell: bash
+ run: python3 -m pip install -e arrow/dev/archery[all]
- name: Install dependencies
run: |
- brew install --overwrite git
+ # brew numpy install fails on gh runner macos-11
+ # and is not required for this build
+ sed -i -e '/brew "numpy"/d' arrow/cpp/Brewfile
brew bundle --file=arrow/cpp/Brewfile
# We want to link aws-sdk-cpp statically but Homebrew's
# aws-sdk-cpp provides only shared library. If we have
# Homebrew's aws-sdk-cpp, our build mix Homebrew's
# aws-sdk-cpp and bundled aws-sdk-cpp. We uninstall Homebrew's
# aws-sdk-cpp to ensure using only bundled aws-sdk-cpp.
brew uninstall aws-sdk-cpp
- - name: Setup ccache
- run: |
- arrow/ci/scripts/ccache_setup.sh
+ brew install openjdk@11
+ brew install sccache
- name: Build C++ libraries
+ env:
+ {{ macros.github_set_sccache_envvars()|indent(8) }}
run: |
set -e
+ # make brew java available to cmake
Review Comment:
```suggestion
# make brew Java available to CMake
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org