You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "fb64 (via GitHub)" <gi...@apache.org> on 2024/02/02 18:58:58 UTC
[I] [Java][Dataset] JNI Error when reading parquet file [arrow]
fb64 opened a new issue, #39919:
URL: https://github.com/apache/arrow/issues/39919
### Describe the bug, including details regarding any error messages, version, and platform.
An JNI error happened when I try to read a parquet file with arrow-dataset (15.0.0) in a java program
Runtime : OpenJDK 64-Bit Server VM (build 17.0.2+8-86, mixed mode, sharing)
It seems that's due to the usage of `NativeMemoryPool.createListenable(
DirectReservationListener.instance())` instead of `NativeMemoryPool.getDefault()`
By the way the error not happened with all parquet files
Full code for reproduce the error
https://gist.github.com/fb64/71880cde297bc5234b02b68b785670fd
Stack trace
````
/Users/runner/work/crossbow/crossbow/arrow/java/dataset/src/main/cpp/jni_util.cc:79: Failed to update reservation while freeing bytes: JNIEnv was not attached to current thread
0 jnilib-10391875251891153172.tmp 0x00000001424a99ac _ZN5arrow4util7CerrLogD2Ev + 204
1 jnilib-10391875251891153172.tmp 0x00000001424a98ce _ZN5arrow4util7CerrLogD0Ev + 14
2 jnilib-10391875251891153172.tmp 0x00000001424a1112 _ZN5arrow4util8ArrowLogD1Ev + 34
3 jnilib-10391875251891153172.tmp 0x00000001412608ed _ZN5arrow7dataset3jni31ReservationListenableMemoryPool4Impl4FreeEPhxx + 237
4 jnilib-10391875251891153172.tmp 0x00000001427b2acb _ZN5arrow10PoolBufferD2Ev + 59
5 jnilib-10391875251891153172.tmp 0x00000001427b268e _ZN5arrow10PoolBufferD0Ev + 14
6 jnilib-10391875251891153172.tmp 0x000000014140f508 _ZN7parquet14SerializedFileD2Ev + 168
7 jnilib-10391875251891153172.tmp 0x000000014140ebfe _ZN7parquet14SerializedFileD0Ev + 14
8 jnilib-10391875251891153172.tmp 0x000000014153a8ef _ZN7parquet5arrow12_GLOBAL__N_114FileReaderImplD0Ev + 159
9 jnilib-10391875251891153172.tmp 0x000000014155048a _ZN5arrow8internal6FnOnceIFvRKNS_10FutureImplEEE6FnImplINS_6FutureINS0_5EmptyEE20WrapResultOnComplete8CallbackINSA_14ThenOnCompleteIZN7parquet5arrow17RowGroupGenerator9FetchNextEvEUlvE_NSA_17PassthruOnFailureISH_EEEEEEED0Ev + 122
10 jnilib-10391875251891153172.tmp 0x00000001424a503d _ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE + 189
11 jnilib-10391875251891153172.tmp 0x00000001428471d1 _ZN5arrow6FutureINS_8internal5EmptyEE14DoMarkFinishedENS_6ResultIS2_EE + 129
12 jnilib-10391875251891153172.tmp 0x0000000142847055 _ZN5arrow6FutureINS_8internal5EmptyEE12MarkFinishedIS2_vEEvNS_6StatusE + 53
13 jnilib-10391875251891153172.tmp 0x000000014154f4a4 _ZZN5arrow8internal8Executor10DoTransferINS0_5EmptyENS_6FutureIS3_EENS_6StatusEEENS4_IT_EES8_bENUlRKS6_E_clESA_ + 100
14 jnilib-10391875251891153172.tmp 0x00000001424a5650 _ZN5arrow8internal6FnOnceIFvvEE6FnImplIZNS_18ConcreteFutureImpl21RunOrScheduleCallbackERKNSt3__110shared_ptrINS_10FutureImplEEEONS8_14CallbackRecordEbEUlvE_E6invokeEv + 32
15 jnilib-10391875251891153172.tmp 0x000000014248e055 _ZNSt3__1L14__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEZN5arrow8internal10ThreadPool21LaunchWorkersUnlockedEiE4$_10EEEEEPvSC_ + 709
16 libsystem_pthread.dylib 0x00007ff810c77202 _pthread_start + 99
17 libsystem_pthread.dylib 0x00007ff810c72bab thread_start + 15
````
### Component(s)
Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] JNI Error when reading parquet file [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou closed issue #39919: [Java][Dataset] JNI Error when reading parquet file
URL: https://github.com/apache/arrow/issues/39919
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] JNI Error when reading parquet file [arrow]
Posted by "isimanovich (via GitHub)" <gi...@apache.org>.
isimanovich commented on issue #39919:
URL: https://github.com/apache/arrow/issues/39919#issuecomment-1931287116
seeing similar issue as in the linked issue from `Kotlin/dataframe` after upgrade to `dataset 15.0.0`
```
Exception in thread "main" java.util.concurrent.ExecutionException: java.lang.UnsatisfiedLinkError: /opt/amazon/var/tmp/jnilib-17078712129987048802.tmp:
/opt/amazon/var/tmp/jnilib-17078712129987048802.tmp: undefined symbol: _ZTIN6google8protobuf7MessageE
at org.apache.arrow.dataset.jni.JniLoader.load(JniLoader.java:92)
at org.apache.arrow.dataset.jni.JniLoader.loadRemaining(JniLoader.java:75)
at org.apache.arrow.dataset.jni.JniLoader.ensureLoaded(JniLoader.java:61)
at org.apache.arrow.dataset.jni.NativeMemoryPool.getDefault(NativeMemoryPool.java:34)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] JNI Error when reading parquet file [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #39919:
URL: https://github.com/apache/arrow/issues/39919#issuecomment-1934702909
The only thing we could do is create a new patch release but I would really like this to be diagnosed and tested / validated if we decide to do a new release. @lidavidm any thoughts? Could 15.0.0 be somehow yanked? BTW we seem to have been able to reproduce the same issue on the java cookbooks:
https://github.com/apache/arrow-cookbook/pull/343#issuecomment-1932470360
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] JNI Error when reading parquet file [arrow]
Posted by "isimanovich (via GitHub)" <gi...@apache.org>.
isimanovich commented on issue #39919:
URL: https://github.com/apache/arrow/issues/39919#issuecomment-1934950427
seeing following error when running on `14.0.1` `NativeMemoryPool pool = NativeMemoryPool.createListenable(listener);`
```
/arrow/java/dataset/src/main/cpp/jni_util.cc:79: Failed to update reservation while freeing bytes: JNIEnv was not attached to current thread
/opt/amazon/var/tmp/jnilib-639979346264002289.tmp(+0x1623f88)[0x7f34b1feef88]
/opt/amazon/var/tmp/jnilib-639979346264002289.tmp(_ZN5arrow4util8ArrowLogD1Ev+0xed)[0x7f34b1fef33d]
/opt/amazon/var/tmp/jnilib-639979346264002289.tmp(_ZN5arrow7dataset3jni31ReservationListenableMemoryPool4FreeEPhll+0x47d)[0x7f34b1591d1d]
/opt/amazon/var/tmp/jnilib-639979346264002289.tmp(_ZNSt23_Sp_counted_ptr_inplaceIN5arrow9ArrayDataESaIS1_ELN9__gnu_cxx12_Lock_policyE2EE10_M_disposeEv+0x10e)[0x7f34b1796d3e]
/opt/amazon/var/tmp/jnilib-639979346264002289.tmp(_ZNSt23_Sp_counted_ptr_inplaceIN5arrow3ipc25RecordBatchFileReaderImpl28CachedRecordBatchReadContextESaIS3_ELN9__gnu_cxx12_Lock_policyE2EE10_M_disposeEv+0x9e)[0x7f34b19c739e]
/opt/amazon/var/tmp/jnilib-639979346264002289.tmp(+0xbbf2ca)[0x7f34b158a2ca]
/opt/amazon/var/tmp/jnilib-639979346264002289.tmp(_ZN5arrow8internal6FnOnceIFvRKNS_10FutureImplEEE6FnImplINS_6FutureINS0_5EmptyEE21WrapResultyOnComplete8CallbackINSA_14ThenOnCompleteIZZNS_3ipc25RecordBatchFileReaderImpl21ReadCachedRecordBatchEiNS8_ISt10shared_ptrINSE_7MessageEEEEENKUlRKSI_E0_clESL_EUlvE_NSA_17PassthruOnFailureISN_EEEEEEED0Ev+0x2e)[0x7f34b19c66ae]
/opt/amazon/var/tmp/jnilib-639979346264002289.tmp(_ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE+0x1a9)[0x7f34b20171e9]
/opt/amazon/var/tmp/jnilib-639979346264002289.tmp(_ZN5arrow6FutureINS_8internal5EmptyEE12MarkFinishedIS2_vEEvNS_6StatusE+0xaa)[0x7f34b15e212a]
/opt/amazon/var/tmp/jnilib-639979346264002289.tmp(+0x1647846)[0x7f34b2012846]
/opt/amazon/var/tmp/jnilib-639979346264002289.tmp(_ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE+0x195)[0x7f34b20171d5]
/opt/amazon/var/tmp/jnilib-639979346264002289.tmp(+0x1688865)[0x7f34b2053865]
/opt/amazon/var/tmp/jnilib-639979346264002289.tmp(_ZNO5arrow8internal6FnOnceIFvvEEclEv+0x165)[0x7f34b1fed1c5]
/opt/amazon/var/tmp/jnilib-639979346264002289.tmp(+0x161e848)[0x7f34b1fe9848]
/opt/amazon/var/tmp/jnilib-639979346264002289.tmp(+0x2a971a0)[0x7f34b34621a0]
/lib64/libc.so.6(+0x9f7f2)[0x7facc71477f2]
/lib64/libc.so.6(clone+0x44)[0x7facc70e7314]
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007facc70d0898, pid=1, tid=234
#
# JRE version: OpenJDK Runtime Environment 1.0.1618.0 (17.0.10+8) (build 17.0.10+8-LTS)
# Java VM: OpenJDK 64-Bit Server VM 1.0.1618.0 (17.0.10+8-LTS, mixed mode, tiered, compressed class ptrs, g1 gc, linux-amd64)
#
# Problematic frame:
# C [libc.so.6+0x28898] abort+0x178
#
# Core dump will be written. Default location: //core.1
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid1.log
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] JNI Error when reading parquet file [arrow]
Posted by "davisusanibar (via GitHub)" <gi...@apache.org>.
davisusanibar commented on issue #39919:
URL: https://github.com/apache/arrow/issues/39919#issuecomment-1934367934
Hi Team,
The following Java jars were downloaded from the v15 release and compared with the v14 release:
**V15**
- Jar: https://repo1.maven.org/maven2/org/apache/arrow/arrow-dataset/15.0.0/arrow-dataset-15.0.0.jar
````
$ unzip arrow-dataset-15.0.0.jar
$ shasum -a 256 aarch_64/libarrow_dataset_jni.so
fd4d8a470815522e69e174c34840295a9e8e2e021575d203697ae09e1c864cca aarch_64/libarrow_dataset_jni.so
$ objdump -p aarch_64/libarrow_dataset_jni.so | grep NEED
NEEDED librt.so.1
NEEDED libpthread.so.0
NEEDED libdl.so.2
NEEDED libstdc++.so.6
NEEDED libm.so.6
NEEDED libgcc_s.so.1
NEEDED libc.so.6
NEEDED ld-linux-aarch64.so.1
$ nm aarch_64/libarrow_dataset_jni.so | grep ZTIN6google8protobuf7Messag
U _ZTIN6google8protobuf7MessageE
````
**U: The symbol is undefined (not found in this object)**
**V14**:
- Jar: https://repo1.maven.org/maven2/org/apache/arrow/arrow-dataset/14.0.2/arrow-dataset-14.0.2.jar
````
$ unzip arrow-dataset-14.0.2.jar
$ shasum -a 256 aarch_64/libarrow_dataset_jni.so
c21cabcd8aaaea8c19197de1955ae0456830c386ea50bd9a298b0e047c4ce932 aarch_64/libarrow_dataset_jni.so
$ objdump -p aarch_64/libarrow_dataset_jni.so | grep NEED
NEEDED librt.so.1
NEEDED libpthread.so.0
NEEDED libdl.so.2
NEEDED libstdc++.so.6
NEEDED libm.so.6
NEEDED libgcc_s.so.1
NEEDED libc.so.6
NEEDED ld-linux-aarch64.so.1
$ nm aarch_64/libarrow_dataset_jni.so | grep ZTIN6google8protobuf7Messag
0000000002ecddb8 V _ZTIN6google8protobuf7MessageE
````
Glossary:
T: The symbol is in the text (code) section
U: The symbol is undefined (not found in this object)
W: The symbol is weak, which means it has lesser priority than a strong global symbol
V: The symbol is weak in the symbol table
Using https://github.com/apache/arrow/tree/a61f4af724cd06c3a9b4abd20491345997e532c0, I have built the libraries and can read dataset files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] JNI Error when reading parquet file [arrow]
Posted by "fb64 (via GitHub)" <gi...@apache.org>.
fb64 commented on issue #39919:
URL: https://github.com/apache/arrow/issues/39919#issuecomment-1931469014
> seeing similar issue as in the linked issue from `Kotlin/dataframe` after upgrade to `dataset 15.0.0`
>
> ```
> Exception in thread "main" java.util.concurrent.ExecutionException: java.lang.UnsatisfiedLinkError: /opt/amazon/var/tmp/jnilib-17078712129987048802.tmp:
> /opt/amazon/var/tmp/jnilib-17078712129987048802.tmp: undefined symbol: _ZTIN6google8protobuf7MessageE
> at org.apache.arrow.dataset.jni.JniLoader.load(JniLoader.java:92)
> at org.apache.arrow.dataset.jni.JniLoader.loadRemaining(JniLoader.java:75)
> at org.apache.arrow.dataset.jni.JniLoader.ensureLoaded(JniLoader.java:61)
> at org.apache.arrow.dataset.jni.NativeMemoryPool.getDefault(NativeMemoryPool.java:34)
> ```
It's not excatly the same issue, we also faced to it on Kotlin dataframe : https://github.com/Kotlin/dataframe/pull/577#issuecomment-1918952628
It seems that it only happened on Linux, I think it's native library compilation issue as the library seems using Protobof but is not statically linked to it. It's working with arrow 14 by the way ....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] JNI Error when reading parquet file [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #39919:
URL: https://github.com/apache/arrow/issues/39919#issuecomment-1935362621
Thanks for checking the result.
I'll merge #40015 tomorrow if nobody objects it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] JNI Error when reading parquet file [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #39919:
URL: https://github.com/apache/arrow/issues/39919#issuecomment-1936168919
I'm not sure about that error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] JNI Error when reading parquet file [arrow]
Posted by "davisusanibar (via GitHub)" <gi...@apache.org>.
davisusanibar commented on issue #39919:
URL: https://github.com/apache/arrow/issues/39919#issuecomment-1935139198
Next, we will take the following steps:
1. Set up Java-jars to validate Linux OS as well.
2. Find out when this error occurs or try to reproduce it locally.
@danepitkin / @lidavidm / @raulcd If you could give me some advice about next steps on this, I would really appreciate it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] JNI Error when reading parquet file [arrow]
Posted by "kymotz (via GitHub)" <gi...@apache.org>.
kymotz commented on issue #39919:
URL: https://github.com/apache/arrow/issues/39919#issuecomment-3284243684
lower arrow-dataset version(14.0.2) can solve JNI error.
ENV: JDK8、centos
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] JNI Error when reading parquet file [arrow]
Posted by "davisusanibar (via GitHub)" <gi...@apache.org>.
davisusanibar commented on issue #39919:
URL: https://github.com/apache/arrow/issues/39919#issuecomment-1935337091
> https://github.com/ursacomputing/crossbow/releases/tag/actions-f7a2fd829d-github-java-jars
The java jar version you provided worked as expected (actions-f7a2fd829d).
Steps to reproduce - Error:
````
- git clone https://github.com/davisusanibar/dataset-error-linux-env.git
- git checkout test-with-nightly-version
- mvn compile exec:java -Dexec.mainClass="dataset.TestMe" -Dmaven.compiler.target=8 -Dmaven.compiler.source=8
...
java.lang.UnsatisfiedLinkError: /tmp/jnilib-15652094898310480780.tmp: /tmp/jnilib-15652094898310480780.tmp: undefined symbol: _ZTIN6google8protobuf7MessageE
...
````
Steps to reproduce - Fix: https://github.com/ursacomputing/crossbow/releases/tag/actions-f7a2fd829d-github-java-jars
````
- git clone https://github.com/davisusanibar/dataset-error-linux-env.git
- git checkout test-with-nightly-version
- rm -rf ~/.m2/repository/org/apache/arrow/arrow-dataset/16.0.0-SNAPSHOT
- wget https://github.com/ursacomputing/crossbow/releases/download/actions-f7a2fd829d-github-java-jars/arrow-dataset-16.0.0-SNAPSHOT.pom
- wget https://github.com/ursacomputing/crossbow/releases/download/actions-f7a2fd829d-github-java-jars/arrow-dataset-16.0.0-SNAPSHOT.jar
- mvn install:install-file -Dfile="$(pwd)/arrow-dataset-16.0.0-SNAPSHOT.pom" -DgroupId=org.apache.arrow -DartifactId=arrow-dataset -Dversion=16.0.0-SNAPSHOT -Dpackaging=pom
- mvn install:install-file -Dfile="$(pwd)/arrow-dataset-16.0.0-SNAPSHOT.jar" -DgroupId=org.apache.arrow -DartifactId=arrow-dataset -Dversion=16.0.0-SNAPSHOT -Dpackaging=jar
- mvn compile exec:java -Dexec.mainClass="dataset.TestMe" -Dmaven.compiler.target=8 -Dmaven.compiler.source=8
n_nationkey n_name n_regionkey n_comment
0 ALGERIA 0 haggle. carefully final deposits detect slyly agai
1 ARGENTINA 1 al foxes promise slyly according to the regular accounts. bold requests alon
...
````
Also aarch_64/libarrow_dataset_jni.so - protobuf changed from U to V:
````
- wget https://github.com/ursacomputing/crossbow/releases/download/actions-f7a2fd829d-github-java-jars/arrow-dataset-16.0.0-SNAPSHOT.jar
- unzip arrow-dataset-16.0.0-SNAPSHOT.jar
- nm arrow_dataset_jni/aarch_64/libarrow_dataset_jni.so | grep ZTIN6google8protobuf7Messag
000000000302df98 V _ZTIN6google8protobuf7MessageE
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] JNI Error when reading parquet file [arrow]
Posted by "davisusanibar (via GitHub)" <gi...@apache.org>.
davisusanibar commented on issue #39919:
URL: https://github.com/apache/arrow/issues/39919#issuecomment-1934376236
v15 release branch also uploaded to Gihub all Java artifacts as assets (v15 release: https://github.com/ursacomputing/crossbow/releases/tag/release-15.0.0-rc1-0-github-java-jars)
@raulcd Are there some way to try to upload again release branch to Github as an assets to test it again (commit: a61f4af724cd06c3a9b4abd20491345997e532c0).
It seems like there was a problem linking protobuf static libraries in the build process or something like that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] JNI Error when reading parquet file [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #39919:
URL: https://github.com/apache/arrow/issues/39919#issuecomment-1936991386
Can we use #20379 for further discussion for this problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] JNI Error when reading parquet file [arrow]
Posted by "davisusanibar (via GitHub)" <gi...@apache.org>.
davisusanibar commented on issue #39919:
URL: https://github.com/apache/arrow/issues/39919#issuecomment-1935115394
> I'm not sure we can yank a maven package.
>
> Why isn't this caught in the java-jars pipeline?
Java-jars is running unders macOS environment, then all the validation finished without problem on macOS: https://github.com/ursacomputing/crossbow/actions/runs/7543210134/job/20535105142
````
Operating System
macOS
12.7.2
21G1974
Runner Image
Image: macos-12
Version: 202[3](https://github.com/ursacomputing/crossbow/actions/runs/7543210134/job/20535105142#step:1:3)121[6](https://github.com/ursacomputing/crossbow/actions/runs/7543210134/job/20535105142#step:1:7).1
Included Software: https://github.com/actions/runner-images/blob/macOS-12/20231216.1/images/macos/macos-12-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/macOS-12%2F20231216.1
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] JNI Error when reading parquet file [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #39919:
URL: https://github.com/apache/arrow/issues/39919#issuecomment-1935234699
https://github.com/apache/arrow/blob/a0dec7f39394e619c8bdfe0eacb6ecde73a9ec12/dev/archery/archery/linking.py#L65-L75 should also check undefined symbols.
It seems that https://github.com/apache/arrow/pull/39137 is the cause of this. Sorry...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] JNI Error when reading parquet file [arrow]
Posted by "fb64 (via GitHub)" <gi...@apache.org>.
fb64 commented on issue #39919:
URL: https://github.com/apache/arrow/issues/39919#issuecomment-1934819775
Any hint about the main error of this issue ?
`/Users/runner/work/crossbow/crossbow/arrow/java/dataset/src/main/cpp/jni_util.cc:79: Failed to update reservation while freeing bytes: JNIEnv was not attached to current thread`
It also happened with Arrow 14 BTW
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] JNI Error when reading parquet file [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #39919:
URL: https://github.com/apache/arrow/issues/39919#issuecomment-1935319349
Could you try `.jar` at https://github.com/ursacomputing/crossbow/releases/tag/actions-f7a2fd829d-github-java-jars ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] JNI Error when reading parquet file [arrow]
Posted by "davisusanibar (via GitHub)" <gi...@apache.org>.
davisusanibar commented on issue #39919:
URL: https://github.com/apache/arrow/issues/39919#issuecomment-1936152309
> > Any hint about the main error of this issue ? `/Users/runner/work/crossbow/crossbow/arrow/java/dataset/src/main/cpp/jni_util.cc:79: Failed to update reservation while freeing bytes: JNIEnv was not attached to current thread` It also happened with Arrow 14 BTW
>
> Hi @fb64, I'm sorry to discuss other errors related to this issue. Please let me review the error `...JNIEnv was not attached to the current thread...` to see what is going on.
Hi @lidavidm this issue is also related with https://github.com/apache/arrow/issues/20379
What would be the [updated answer to that error today](https://github.com/apache/arrow/issues/20379#issuecomment-1377720530)? Would you please be able to assist me with that information?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] JNI Error when reading parquet file [arrow]
Posted by "davisusanibar (via GitHub)" <gi...@apache.org>.
davisusanibar commented on issue #39919:
URL: https://github.com/apache/arrow/issues/39919#issuecomment-1935130012
> > > I'm not sure we can yank a maven package.
> > > Why isn't this caught in the java-jars pipeline?
> >
> >
> > Java-jars is running unders macOS environment, then all the validation finished without problem on macOS: https://github.com/ursacomputing/crossbow/actions/runs/7543210134/job/20535105142
> > ```
> >
> > Operating System
> > macOS
> > 12.7.2
> > 21G1974
> > Runner Image
> > Image: macos-12
> > Version: 202[3](https://github.com/ursacomputing/crossbow/actions/runs/7543210134/job/20535105142#step:1:3)121[6](https://github.com/ursacomputing/crossbow/actions/runs/7543210134/job/20535105142#step:1:7).1
> > Included Software: https://github.com/actions/runner-images/blob/macOS-12/20231216.1/images/macos/macos-12-Readme.md
> > Image Release: https://github.com/actions/runner-images/releases/tag/macOS-12%2F20231216.1
> > ```
>
> Error related to `undefined symbol: _ZTIN6google8protobuf7MessageE` also appear with nightly versions:
>
> ```
> Error with Dataset 15.0.0:
> - git clone https://github.com/davisusanibar/dataset-error-linux-env.git
> - mvn compile exec:java -Dexec.mainClass="dataset.TestMe" -Dmaven.compiler.target=8 -Dmaven.compiler.source=8
>
>
> Error with Dataset 16.0.0-SNAPSHOT:
> - git clone https://github.com/davisusanibar/dataset-error-linux-env.git
> - git checkout test-with-nightly-version
> - mvn compile exec:java -Dexec.mainClass="dataset.TestMe" -Dmaven.compiler.target=8 -Dmaven.compiler.source=8
> ```
Reviewing crossbow http://crossbow.voltrondata.com/ JNI libraries uploaded to Github Assets, we found that 2023-12-09 was the first time we saw U=Undefined on the libarrow_dataset_jni.so library.
<img width="912" alt="image" src="https://github.com/apache/arrow/assets/4554485/af57f143-b020-4fcb-b21b-ac8c909d11f1">
````
2023-12-08: OK - https://github.com/ursacomputing/crossbow/actions/runs/7139118592
nm libarrow_dataset_jni.so | grep ZTIN6google8protobuf7Messag
0000000002f5dd78 V _ZTIN6google8protobuf7MessageE
2023-12-09: NOK --> First undefined appear U - https://github.com/ursacomputing/crossbow/actions/runs/7150103719
nm libarrow_dataset_jni.so | grep ZTIN6google8protobuf7Messag
U _ZTIN6google8protobuf7MessageE
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] JNI Error when reading parquet file [arrow]
Posted by "davisusanibar (via GitHub)" <gi...@apache.org>.
davisusanibar commented on issue #39919:
URL: https://github.com/apache/arrow/issues/39919#issuecomment-1935122929
> > I'm not sure we can yank a maven package.
> > Why isn't this caught in the java-jars pipeline?
>
> Java-jars is running unders macOS environment, then all the validation finished without problem on macOS: https://github.com/ursacomputing/crossbow/actions/runs/7543210134/job/20535105142
>
> ```
>
> Operating System
> macOS
> 12.7.2
> 21G1974
> Runner Image
> Image: macos-12
> Version: 202[3](https://github.com/ursacomputing/crossbow/actions/runs/7543210134/job/20535105142#step:1:3)121[6](https://github.com/ursacomputing/crossbow/actions/runs/7543210134/job/20535105142#step:1:7).1
> Included Software: https://github.com/actions/runner-images/blob/macOS-12/20231216.1/images/macos/macos-12-Readme.md
> Image Release: https://github.com/actions/runner-images/releases/tag/macOS-12%2F20231216.1
> ```
Error related to `undefined symbol: _ZTIN6google8protobuf7MessageE` also appear with nightly versions:
````
Error with Dataset 15.0.0:
- git clone https://github.com/davisusanibar/dataset-error-linux-env.git
- mvn compile exec:java -Dexec.mainClass="dataset.TestMe" -Dmaven.compiler.target=8 -Dmaven.compiler.source=8
Error with Dataset 16.0.0-SNAPSHOT:
- git clone https://github.com/davisusanibar/dataset-error-linux-env.git
- git checkout test-with-nightly-version
- mvn compile exec:java -Dexec.mainClass="dataset.TestMe" -Dmaven.compiler.target=8 -Dmaven.compiler.source=8
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] JNI Error when reading parquet file [arrow]
Posted by "davisusanibar (via GitHub)" <gi...@apache.org>.
davisusanibar commented on issue #39919:
URL: https://github.com/apache/arrow/issues/39919#issuecomment-1935140775
> Any hint about the main error of this issue ? `/Users/runner/work/crossbow/crossbow/arrow/java/dataset/src/main/cpp/jni_util.cc:79: Failed to update reservation while freeing bytes: JNIEnv was not attached to current thread` It also happened with Arrow 14 BTW
Hi @fb64, I'm sorry to discuss other errors related to this issue. Please let me review the error `...JNIEnv was not attached to the current thread...` to see what is going on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] JNI Error when reading parquet file [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on issue #39919:
URL: https://github.com/apache/arrow/issues/39919#issuecomment-1935152520
I took a look at the list of undefined symbols in the `.so` and they seem to be the symbols that the full `libprotobuf` defines but are not present in `libprotobuf-lite`. So this could be caused by one of:
1) statically linking to `libprotobuf-lite.a` instead of `libprotobuf.a`, or
2) a recent addition of code generated by `protoc` from a `.proto` file that doesn't contain `option optimize_for = LITE_RUNTIME;`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] JNI Error when reading parquet file [arrow]
Posted by "fb64 (via GitHub)" <gi...@apache.org>.
fb64 commented on issue #39919:
URL: https://github.com/apache/arrow/issues/39919#issuecomment-1936379545
Indeed, it seems to be the same as https://github.com/apache/arrow/issues/20379 , I didn't find it before creating this one ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] JNI Error when reading parquet file [arrow]
Posted by "isimanovich (via GitHub)" <gi...@apache.org>.
isimanovich commented on issue #39919:
URL: https://github.com/apache/arrow/issues/39919#issuecomment-1932373641
yes, i did see in your thread on https://github.com/Kotlin/dataframe/pull/577#issuecomment-1919048632 that downgrade to `14.0.2` helped, so I will be doing that for now. I wanted to point out that in my case it is also issue on Linux and it works fine on Mac M1 Pro.
While I understand that this is not exactly the same issue as current ticket, I was pointing out that it was similar to https://github.com/Kotlin/dataframe/pull/577#issuecomment-1917643952
> It seems that's due to the usage of NativeMemoryPool.createListenable( DirectReservationListener.instance()) instead of NativeMemoryPool.getDefault()
in my case, im seeing this behavior with both `NativeMemoryPool.createListenable( DirectReservationListener.instance())` and `NativeMemoryPool.getDefault()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] JNI Error when reading parquet file [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #39919:
URL: https://github.com/apache/arrow/issues/39919#issuecomment-1934716083
I'm not sure we can yank a maven package.
Why isn't this caught in the java-jars pipeline?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org