You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "tomscut (via GitHub)" <gi...@apache.org> on 2025/05/09 07:26:03 UTC
[I] [VL] free(): invalid pointer [incubator-gluten]
tomscut opened a new issue, #9580:
URL: https://github.com/apache/incubator-gluten/issues/9580
### Backend
VL (Velox)
### Bug description
We have a hive table in `SequenceFile` format, and when we execute this sql `select count(1) from test.test_table where dt='20250401';
`, it will report a `free(): invalid pointer`.
After debugging the code, I realized that by changing one line of code in `/incubator-gluten/cpp/velox/compute/VeloxBackend.cc`, the problem was solved. We suspect that the same pointer of `configs_` has been freed repeatedly.
before the change:
```
if (backendConf_->get<bool>(kDebugModeEnabled, false)) {
LOG(INFO) << "VeloxBackend config:" << printConfig(backendConf_->rawConfigs());
}
```
after the change:
```
LOG(INFO) << "VeloxBackend config:" << printConfig(backendConf_->rawConfigs());
```
### Gluten version
Gluten-1.3
### Spark version
Spark-3.4.x
### Spark configurations
${SPARK_HOME}/bin/spark-sql --master yarn \
--queue xxx \
--deploy-mode client \
--executor-cores 4 \
--executor-memory 8g \
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
--conf spark.plugins=org.apache.gluten.GlutenPlugin \
--conf spark.executorEnv.JAVA_HOME=/apps/svr/jdk-17.0.12 \
--conf spark.executor.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true" \
--conf spark.sql.autoBroadcastJoinThreshold=-1 \
--conf spark.sql.adaptive.autoBroadcastJoinThreshold=-1
### System information
Gluten Version: 1.3.0
Commit: c205d4c7a54abfc1b866c2041192320eccfaa380
CMake Version: 3.28.3
System: Linux-6.8.0-59-generic
Arch: x86_64
CPU Name: Model name: Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
C++ Compiler: /opt/rh/devtoolset-11/root/usr/bin/c++
C++ Compiler Version: 11.2.1
C Compiler: /opt/rh/devtoolset-11/root/usr/bin/cc
C Compiler Version: 11.2.1
CMake Prefix Path: /usr/local;/usr;/;/usr/local/lib64/python3.6/site-packages/cmake/data;/usr/local;/usr/X11R6;/usr/pkg;/opt
### Relevant logs
```bash
25/05/07 15:56:25 INFO [dispatcher-Executor] ExecutorPluginContainer: Initialized executor component for plugin org.apache.gluten.GlutenPlugin.
25/05/07 15:56:31 INFO [dispatcher-Executor] YarnCoarseGrainedExecutorBackend: Got assigned task 0
25/05/07 15:56:31 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] Executor: Running task 0.0 in stage 0.0 (TID 0)
25/05/07 15:56:32 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] TorrentBroadcast: Started reading broadcast variable 1 with 1 pieces (estimated total size 16.0 MiB)
25/05/07 15:56:32 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] TransportClientFactory: Successfully created connection to /10.241.x.xx:3881 after 2 ms (0 ms spent in bootstraps)
25/05/07 15:56:32 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 49.1 KiB, free 8.6 GiB)
25/05/07 15:56:32 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] TorrentBroadcast: Reading broadcast variable 1 took 78 ms
25/05/07 15:56:32 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] MemoryStore: Block broadcast_1 stored as values in memory (estimated size 183.1 KiB, free 8.6 GiB)
25/05/07 15:56:32 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] HadoopRDD: Input split: hdfs://xxx/ction_relation/dt=20250401/part-00000-29cadb76-3c64-45a2-b12a-804d4c64a8db-c000:0+16600316
25/05/07 15:56:32 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] TorrentBroadcast: Started reading broadcast variable 0 with 1 pieces (estimated total size 16.0 MiB)
25/05/07 15:56:32 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 33.7 KiB, free 8.6 GiB)
25/05/07 15:56:32 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] TorrentBroadcast: Reading broadcast variable 0 took 15 ms
25/05/07 15:56:32 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] MemoryStore: Block broadcast_0 stored as values in memory (estimated size 478.5 KiB, free 8.6 GiB)
25/05/07 15:56:33 WARN [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
25/05/07 15:56:33 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] CodecPool: Got brand-new decompressor [.snappy]
25/05/07 15:56:33 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] CodecPool: Got brand-new decompressor [.snappy]
25/05/07 15:56:33 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] CodecPool: Got brand-new decompressor [.snappy]
25/05/07 15:56:33 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] CodecPool: Got brand-new decompressor [.snappy]
25/05/07 15:56:33 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] CodeGenerator: Code generated in 172.209446 ms
25/05/07 15:56:33 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] BaseAllocator: Debug mode disabled. Enable with the VM option -Darrow.memory.debug.allocator=true.
25/05/07 15:56:33 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] DefaultAllocationManagerOption: allocation manager type not specified, using netty as the default type
25/05/07 15:56:33 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] CheckAllocator: Using DefaultAllocationManager at memory/DefaultAllocationManagerFactory.class
W20250507 15:56:34.111573 2366039 TimeZoneMap.cpp:87] Unable to load [America/Ciudad_Juarez] from local timezone database: America/Ciudad_Juarez not found in timezone database
free(): invalid pointer
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] `free(): invalid pointer` [incubator-gluten]
Posted by "NEUpanning (via GitHub)" <gi...@apache.org>.
NEUpanning commented on issue #9580:
URL: https://github.com/apache/incubator-gluten/issues/9580#issuecomment-3187541882
@tomscut Hello, could you share the backtrace of the crash? We may encounter the same issue. Here is ours:
```
*** Error in `/usr/local/java18/bin/java': free(): invalid pointer: 0x00007fa95f6ebc80 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81329)[0x7fa9df463329]
/lib64/libstdc++.so.6(_ZNSt6locale5_Impl16_M_install_facetEPKNS_2idEPKNS_5facetE+0x142)[0x7fa95f4551c2]
/lib64/libstdc++.so.6(_ZNSt6locale5_ImplC1Em+0x1e3)[0x7fa95f455613]
/lib64/libstdc++.so.6(+0x71585)[0x7fa95f456585]
/lib64/libpthread.so.0(+0x620b)[0x7fa9dfbd020b]
/lib64/libstdc++.so.6(+0x715d1)[0x7fa95f4565d1]
/lib64/libstdc++.so.6(_ZNSt6localeC2Ev+0x13)[0x7fa95f456613]
/lib64/libstdc++.so.6(_ZNSt8ios_base4InitC2Ev+0xbc)[0x7fa95f45346c]
/data5/hadoop/yarn/nm-local-dir/usercache/hadoop-ba-search/appcache/application_1737353245333_48035562/container_e57_1737353245333_48035562_01_000001/tmp/jnilib-4628348603672654941.tmp(+0x2a90)[0x7fa985a8ca90]
/lib64/ld-linux-x86-64.so.2(+0xf9c3)[0x7fa9dfdf59c3]
/lib64/ld-linux-x86-64.so.2(+0x1459e)[0x7fa9dfdfa59e]
/lib64/ld-linux-x86-64.so.2(+0xf7d4)[0x7fa9dfdf57d4]
/lib64/ld-linux-x86-64.so.2(+0x13b8b)[0x7fa9dfdf9b8b]
/lib64/libdl.so.2(+0xfab)[0x7fa9df7b0fab]
/lib64/ld-linux-x86-64.so.2(+0xf7d4)[0x7fa9dfdf57d4]
/lib64/libdl.so.2(+0x15ad)[0x7fa9df7b15ad]
/lib64/libdl.so.2(dlopen+0x31)[0x7fa9df7b1041]
/usr/local/jdk1.8.0_112/jre/lib/amd64/server/libjvm.so(+0x926f0e)[0x7fa9ded16f0e]
/usr/local/jdk1.8.0_112/jre/lib/amd64/server/libjvm.so(JVM_LoadLibrary+0x14c)[0x7fa9deb0f74c]
/usr/local/jdk1.8.0_112/jre/lib/amd64/libjava.so(Java_java_lang_ClassLoader_00024NativeLibrary_load+0x138)[0x7fa9ddabadf8]
[0x7fa9c9017a34]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org