You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/08/22 08:58:26 UTC
[GitHub] [spark] LuciferYang commented on a diff in pull request #37610: [SPARK-38888][BUILD][CORE][YARN] Add RocksDB support for shuffle state store
LuciferYang commented on code in PR #37610:
URL: https://github.com/apache/spark/pull/37610#discussion_r951185256
##########
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:
##########
@@ -802,6 +802,7 @@ public void close() {
if (db != null) {
try {
db.close();
+ db = null;
Review Comment:
@zhouyejoe should we set `db = null`?
I found that `Finalized merged shuffle are written into DB and cleaned up after application stopped` in `YarnShuffleServiceWithRocksDBBackendSuite` will crash when use RocksDB.
The native stack as follows:
```
Stack: [0x0000000288410000,0x0000000288813000], sp=0x0000000288812150, free space=4104k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [librocksdbjni6224075543374471149.jnilib+0xd4ec8] _ZN7rocksdb15CompactedDBImpl3GetERKNS_11ReadOptionsEPNS_18ColumnFamilyHandleERKNS_5SliceEPNS_13PinnableSliceEPNSt3__112basic_stringIcNSB_11char_traitsIcEENSB_9allocatorIcEEEE+0xc6c
C [librocksdbjni6224075543374471149.jnilib+0x1406a0] _ZN7rocksdb6DBImpl6DeleteERKNS_12WriteOptionsEPNS_18ColumnFamilyHandleERKNS_5SliceE+0x38
C [librocksdbjni6224075543374471149.jnilib+0x1e38c] _Z21rocksdb_delete_helperP7JNIEnv_PN7rocksdb2DBERKNS1_12WriteOptionsEPNS1_18ColumnFamilyHandleEP11_jbyteArrayii+0xe0
j org.rocksdb.RocksDB.delete(J[BII)V+0
j org.rocksdb.RocksDB.delete([B)V+9
j org.apache.spark.network.shuffledb.RocksDB.delete([B)V+5
j org.apache.spark.network.shuffle.RemoteBlockPushResolver.removeAppShufflePartitionInfoFromDB(Lorg/apache/spark/network/shuffle/RemoteBlockPushResolver$AppAttemptShuffleMergeId;)V+16
j org.apache.spark.network.shuffle.RemoteBlockPushResolver.lambda$removeAppShuffleInfoFromDB$8(Lorg/apache/spark/network/shuffle/RemoteBlockPushResolver$AppShuffleInfo;Ljava/lang/Integer;Lorg/apache/spark/network/shuffle/RemoteBlockPushResolver$AppShuffleMergePartitionsInfo;)V+24
j org.apache.spark.network.shuffle.RemoteBlockPushResolver$$Lambda$321.accept(Ljava/lang/Object;Ljava/lang/Object;)V+16
j java.util.concurrent.ConcurrentHashMap.forEach(Ljava/util/function/BiConsumer;)V+56
j org.apache.spark.network.shuffle.RemoteBlockPushResolver.removeAppShuffleInfoFromDB(Lorg/apache/spark/network/shuffle/RemoteBlockPushResolver$AppShuffleInfo;)V+18
j org.apache.spark.network.shuffle.RemoteBlockPushResolver.closeAndDeletePartitionsIfNeeded(Lorg/apache/spark/network/shuffle/RemoteBlockPushResolver$AppShuffleInfo;Z)V+25
j org.apache.spark.network.shuffle.RemoteBlockPushResolver.lambda$applicationRemoved$5(Lorg/apache/spark/network/shuffle/RemoteBlockPushResolver$AppShuffleInfo;Z)V+3
j org.apache.spark.network.shuffle.RemoteBlockPushResolver$$Lambda$319.run()V+12
j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j io.netty.util.concurrent.FastThreadLocalRunnable.run()V+4
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
V [libjvm.dylib+0x289d70]
V [libjvm.dylib+0x2887fc]
V [libjvm.dylib+0x288a50]
V [libjvm.dylib+0x3118d0]
V [libjvm.dylib+0x524408]
V [libjvm.dylib+0x5242a4]
V [libjvm.dylib+0x456874]
C [libsystem_pthread.dylib+0x7878] _pthread_start+0x140
C [libsystem_pthread.dylib+0x25e0] thread_start+0x8
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j org.rocksdb.RocksDB.delete(J[BII)V+0
j org.rocksdb.RocksDB.delete([B)V+9
j org.apache.spark.network.shuffledb.RocksDB.delete([B)V+5
j org.apache.spark.network.shuffle.RemoteBlockPushResolver.removeAppShufflePartitionInfoFromDB(Lorg/apache/spark/network/shuffle/RemoteBlockPushResolver$AppAttemptShuffleMergeId;)V+16
j org.apache.spark.network.shuffle.RemoteBlockPushResolver.lambda$removeAppShuffleInfoFromDB$8(Lorg/apache/spark/network/shuffle/RemoteBlockPushResolver$AppShuffleInfo;Ljava/lang/Integer;Lorg/apache/spark/network/shuffle/RemoteBlockPushResolver$AppShuffleMergePartitionsInfo;)V+24
j org.apache.spark.network.shuffle.RemoteBlockPushResolver$$Lambda$321.accept(Ljava/lang/Object;Ljava/lang/Object;)V+16
j java.util.concurrent.ConcurrentHashMap.forEach(Ljava/util/function/BiConsumer;)V+56
j org.apache.spark.network.shuffle.RemoteBlockPushResolver.removeAppShuffleInfoFromDB(Lorg/apache/spark/network/shuffle/RemoteBlockPushResolver$AppShuffleInfo;)V+18
j org.apache.spark.network.shuffle.RemoteBlockPushResolver.closeAndDeletePartitionsIfNeeded(Lorg/apache/spark/network/shuffle/RemoteBlockPushResolver$AppShuffleInfo;Z)V+25
j org.apache.spark.network.shuffle.RemoteBlockPushResolver.lambda$applicationRemoved$5(Lorg/apache/spark/network/shuffle/RemoteBlockPushResolver$AppShuffleInfo;Z)V+3
j org.apache.spark.network.shuffle.RemoteBlockPushResolver$$Lambda$319.run()V+12
j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j io.netty.util.concurrent.FastThreadLocalRunnable.run()V+4
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
```
[hs_err_pid12358.log](https://github.com/apache/spark/files/9392790/hs_err_pid12358.log)
We may have operated a closed DB due to `close` and 'removeappshufflepartitioninfofromdb' are on different threads,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org