You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "myeyre (via GitHub)" <gi...@apache.org> on 2026/04/17 09:19:21 UTC
[I] Coredump on Kunpeng 920 CPU [gluten]
myeyre opened a new issue, #11954:
URL: https://github.com/apache/gluten/issues/11954
### Backend
VL (Velox)
### Bug description
I'm running Gluten 1.6.0 and 1.7.0 snapshot inside Ubuntu 22.04 container on Kunpeng 920 CPU, with JDK
to OpenJDK17. The JVM crashed if using gluten in spark-sql.
### Gluten version
main branch
### Spark version
Spark-3.5.x
### Spark configurations
bin/spark-sql \
--master local \
--executor-cores 4 \
--executor-memory 4g \
--num-executors 1 \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=12g \
--conf spark.plugins=org.apache.gluten.GlutenPlugin \
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
--jars /mnt/nfs-workdir/spark-deps/gluten-jdk17-20260416/gluten-velox-bundle-spark3.5_2.12-linux_aarch64-1.6.0-SNAPSHOT.jar \
--conf spark.driver.extraClassPath=/mnt/nfs-workdir/spark-deps/gluten-jdk17-20260416/gluten-velox-bundle-spark3.5_2.12-linux_aarch64-1.6.0-SNAPSHOT.jar \
--conf spark.executor.extraClassPath=/mnt/nfs-workdir/spark-deps/gluten-jdk17-20260416/gluten-velox-bundle-spark3.5_2.12-linux_aarch64-1.6.0-SNAPSHOT.jar \
### System information
Kernel:
Linux BClinux 5.10.0-200.0.0.181.111.bclinux.21.10U4.aarch64 #1 SMP Wed Sep 24 17:45:37 CST 2025 aarch64 aarch64 aarch64 GNU/Linux
HostOS:
BigCloud Enterprise Linux 21.10U4 LTS
ContainerOS:
Ubuntu 22.04.5 LTS
# lscpu
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 192
On-line CPU(s) list: 0-191
Vendor ID: HiSilicon
Model name: Kunpeng-920
Model: 0
Thread(s) per core: 1
Core(s) per cluster: 48
Socket(s): -
Cluster(s): 4
Stepping: 0x1
Frequency boost: disabled
CPU max MHz: 2600.0000
CPU min MHz: 200.0000
BogoMIPS: 200.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma dcpop asimddp asimdfhm ssbs
### Relevant logs
```bash
spark-sql (default)> select count(*) from passwdx;
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGILL (0x4) at pc=0x0000fffdb72d6f04, pid=2769, tid=3007
#
# JRE version: OpenJDK Runtime Environment (17.0.18+8) (build 17.0.18+8-Ubuntu-122.04.1)
# Java VM: OpenJDK 64-Bit Server VM (17.0.18+8-Ubuntu-122.04.1, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-aarch64)
# Problematic frame:
# C [libgluten.so+0x6a6f04] arrow::BaseMemoryPoolImpl<arrow::(anonymous namespace)::SystemAllocator>::Allocate(long, long, unsigned char**)+0x104
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" (or dumping to /opt/spark/core.2769)
#
# An error report file with more information is saved as:
# /opt/spark/hs_err_pid2769.log
#
# If you would like to submit a bug report, please visit:
# https://bugs.launchpad.net/ubuntu/+source/openjdk-17
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Aborted (core dumped)
-------------------
[New LWP 226]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
--Type <RET> for more, q to quit, c to continue without paging--
Core was generated by `/usr/lib/jvm/java-17-openjdk-arm64/bin/java -cp /mnt/nfs-workdir/spark-deps/glu'.
Program terminated with signal SIGABRT, Aborted.
#0 __pthread_kill_implementation (threadid=281458197336352, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
44 ./nptl/pthread_kill.c: No such file or directory.
[Current thread is 1 (Thread 0xfffc17df1120 (LWP 521))]
(gdb)
(gdb)
(gdb)
(gdb) bt
#0 __pthread_kill_implementation (threadid=281458197336352, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
#1 0x0000ffff9b612054 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2 0x0000ffff9b5ca83c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3 0x0000ffff9b5b7134 in __GI_abort () at ./stdlib/abort.c:79
#4 0x0000ffff9ae04fb8 in os::abort(bool, void*, void const*) () from /usr/lib/jvm/java-17-openjdk-arm64/lib/server/libjvm.so
#5 0x0000ffff9b0bcb6c in VMError::report_and_die(int, char const*, char const*, std::__va_list, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long) ()
from /usr/lib/jvm/java-17-openjdk-arm64/lib/server/libjvm.so
#6 0x0000ffff9b0bd51c in VMError::report_and_die(Thread*, unsigned int, unsigned char*, void*, void*, char const*, ...) () from /usr/lib/jvm/java-17-openjdk-arm64/lib/server/libjvm.so
#7 0x0000ffff9af6035c in JVM_handle_linux_signal () from /usr/lib/jvm/java-17-openjdk-arm64/lib/server/libjvm.so
#8 <signal handler called>
#9 0x0000fffd976e6f04 in arrow::BaseMemoryPoolImpl<arrow::(anonymous namespace)::SystemAllocator>::Allocate(long, long, unsigned char**) ()
from /tmp/gluten-1e27a60e-c7cd-4692-9bed-92a76da5d670/jni/969a2f58-bfc0-4b60-9d46-bad3521af894/gluten-3661091660157195524/linux/aarch64/libgluten.so
#10 0x0000fffd976eb428 in arrow::PoolBuffer::Reserve(long) ()
from /tmp/gluten-1e27a60e-c7cd-4692-9bed-92a76da5d670/jni/969a2f58-bfc0-4b60-9d46-bad3521af894/gluten-3661091660157195524/linux/aarch64/libgluten.so
#11 0x0000fffd976eb4e4 in arrow::PoolBuffer::Resize(long, bool) ()
from /tmp/gluten-1e27a60e-c7cd-4692-9bed-92a76da5d670/jni/969a2f58-bfc0-4b60-9d46-bad3521af894/gluten-3661091660157195524/linux/aarch64/libgluten.so
#12 0x0000fffd976ea408 in arrow::AllocateResizableBuffer(long, long, arrow::MemoryPool*) ()
from /tmp/gluten-1e27a60e-c7cd-4692-9bed-92a76da5d670/jni/969a2f58-bfc0-4b60-9d46-bad3521af894/gluten-3661091660157195524/linux/aarch64/libgluten.so
#13 0x0000fffd976ea93c in arrow::AllocateResizableBuffer(long, arrow::MemoryPool*) ()
from /tmp/gluten-1e27a60e-c7cd-4692-9bed-92a76da5d670/jni/969a2f58-bfc0-4b60-9d46-bad3521af894/gluten-3661091660157195524/linux/aarch64/libgluten.so
#14 0x0000fffd97758704 in arrow::io::BufferedBase::ResetBuffer() ()
from /tmp/gluten-1e27a60e-c7cd-4692-9bed-92a76da5d670/jni/969a2f58-bfc0-4b60-9d46-bad3521af894/gluten-3661091660157195524/linux/aarch64/libgluten.so
#15 0x0000fffd9775694c in arrow::io::BufferedOutputStream::Create(long, arrow::MemoryPool*, std::shared_ptr<arrow::io::OutputStream>) ()
from /tmp/gluten-1e27a60e-c7cd-4692-9bed-92a76da5d670/jni/969a2f58-bfc0-4b60-9d46-bad3521af894/gluten-3661091660157195524/linux/aarch64/libgluten.so
#16 0x0000fffd975986dc in gluten::(anonymous namespace)::openFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long) ()
from /tmp/gluten-1e27a60e-c7cd-4692-9bed-92a76da5d670/jni/969a2f58-bfc0-4b60-9d46-bad3521af894/gluten-3661091660157195524/linux/aarch64/libgluten.so
#17 0x0000fffd9759b30c in gluten::LocalPartitionWriter::stop(gluten::ShuffleWriterMetrics*, long&) ()
from /tmp/gluten-1e27a60e-c7cd-4692-9bed-92a76da5d670/jni/969a2f58-bfc0-4b60-9d46-bad3521af894/gluten-3661091660157195524/linux/aarch64/libgluten.so
#18 0x0000fffd90be922c in gluten::VeloxHashShuffleWriter::stop() ()
from /tmp/gluten-1e27a60e-c7cd-4692-9bed-92a76da5d670/jni/969a2f58-bfc0-4b60-9d46-bad3521af894/gluten-3661091660157195524/linux/aarch64/libvelox.so
#19 0x0000fffd9758719c in Java_org_apache_gluten_vectorized_ShuffleWriterJniWrapper_stop ()
from /tmp/gluten-1e27a60e-c7cd-4692-9bed-92a76da5d670/jni/969a2f58-bfc0-4b60-9d46-bad3521af894/gluten-3661091660157195524/linux/aarch64/libgluten.so
#20 0x0000ffff848c2af8 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Coredump on Kunpeng 920 CPU [gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on issue #11954:
URL: https://github.com/apache/gluten/issues/11954#issuecomment-4294712604
@myeyre The issue may due to some arrow ABI mismatch: the nightly release is compiled on gvt2(Arm Neoverse N1) using `march=arm-v8`
could you please have a try by make a local package on Kunpeng 920? here's the example flow to build a package
https://github.com/apache/gluten/actions/runs/24541068996/workflow#L184
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Coredump on Kunpeng 920 CPU [gluten]
Posted by "myeyre (via GitHub)" <gi...@apache.org>.
myeyre commented on issue #11954:
URL: https://github.com/apache/gluten/issues/11954#issuecomment-4362994221
It seems that it is the Arrow binary causing the core.
According to the action logs, it seems that the docker images apache/gluten:vcpkg-centos-* were built on a neoverse-n2, which seems to be an ARMv9.0-A CPU: https://en.wikipedia.org/wiki/ARM_Neoverse . It will surely crash on any CPU pre ARMv9.0-A.
Should we have it built with same flag as Gluten/Velox? I think -march=armv8 should run on both Neoverse N1 and Kunpeng920.
Having it run on the same instance type as Gluten/Velox would NOT do the job because the the script will genearte a -mcpu=neoverse-n1 flag, which will build a native binary for neoverse-n1. I think it differs slightly with Kunpeng920.
https://github.com/apache/gluten/blob/main/dev/build-helper-functions.sh
https://github.com/IBM/velox/blob/main/scripts/setup-helper-functions.sh
Gluten/Velox not using above script but Arrow did.
https://github.com/apache/gluten/blob/main/dev/builddeps-veloxbe.sh#L282
Trying to rebuild it on Kunpeng. Rebuilding should solve the problem for me, but looking forward for an official solution, and binaries😃
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org