Re: [PR] [GLUTEN-9163][VL] Use stream de/compressor in sort-based shuffle [incubator-gluten]

["wangyum (via GitHub)" <gi...@apache.org>]

wangyum commented on PR #9278:
URL: https://github.com/apache/incubator-gluten/pull/9278#issuecomment-2901970336

   It seems this patch makes core dump. Error message:
   ```
   /bin/bash: line 1: 320713 Aborted                 (core dumped) LD_LIBRARY_PATH="/apache/hadoop/lib/native/Linux-amd64-64/lib:/apache/hadoop/lib/native:" /apache/java17/bin/java -server -Xmx46080m -Xms46080m '-Djava.net.preferIPv6Addresses=false' '-XX:+IgnoreUnrecognizedVMOptions' '--add-opens=java.base/java.lang=ALL-UNNAMED' '--add-opens=java.base/java.lang.invoke=ALL-UNNAMED' '--add-opens=java.base/java.lang.reflect=ALL-UNNAMED' '--add-opens=java.base/java.io=ALL-UNNAMED' '--add-opens=java.base/java.net=ALL-UNNAMED' '--add-opens=java.base/java.nio=ALL-UNNAMED' '--add-opens=java.base/java.util=ALL-UNNAMED' '--add-opens=java.base/java.util.concurrent=ALL-UNNAMED' '--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED' '--add-opens=java.base/sun.nio.ch=ALL-UNNAMED' '--add-opens=java.base/sun.nio.cs=ALL-UNNAMED' '--add-opens=java.base/sun.security.action=ALL-UNNAMED' '--add-opens=java.base/sun.util.calendar=ALL-UNNAMED' '--add-opens=java.security.jgss/sun.security.krb5=ALL-U
 NNAMED' '-Djdk.reflect.useDirectMethodHandle=false' '-Dio.netty.tryReflectionSetAccessible=true' '-Xss8M' '-XX:MaxPermSize=512m' '-XX:+IgnoreUnrecognizedVMOptions' '-XX:+UseG1GC' '-XX:+UnlockExperimentalVMOptions' '-XX:MaxGCPauseMillis=500' '-XX:ParallelGCThreads=10' '-XX:ConcGCThreads=4' '-XX:G1MixedGCLiveThresholdPercent=55' '-XX:G1ReservePercent=15' '-XX:+PrintGC' '-XX:+PrintGCDetails' '-XX:+PrintGCDateStamps' '-XX:+PrintGCApplicationStoppedTime' '-XX:+PrintGCApplicationConcurrentTime' '-XX:+PrintClassHistogramAfterFullGC' '-XX:MaxJavaStackTraceDepth=50000' '-XX:ReservedCodeCacheSize=512M' '-XX:PerMethodRecompilationCutoff=10000' '-XX:PerBytecodeRecompilationCutoff=10000' '-Djava.net.preferIPv4Stack=true' '-Dsun.net.inetaddr.ttl=120' '-Dio.netty.eventLoop.maxPendingTasks=20000' '-Dio.netty.maxDirectMemory=15032385536' '-Dio.netty.tryReflectionSetAccessible=true' -Djava.io.tmpdir=/hadoop/4/yarn/local/usercache/appcache/application_1736396393732_145750/container_e2322_1736396393732
 _145750_01_000191/tmp '-Dspark.port.maxRetries=100' '-Dspark.driver.blockManager.port=30601' '-Dspark.blockManager.port=30501' '-Dspark.ui.port=30401' '-Dspark.viewpoint.ui.port=9090' '-Dspark.history.ui.port=9090' '-Dspark.network.timeout=300' '-Dspark.ssl.historyServer.port=8080' '-Dspark.driver.port=30201' '-Dspark.rpc.io.serverThreads=16' '-Dspark.shuffle.ess.service.port=7337' '-Dspark.shuffle.service.port=7338' -Dspark.yarn.app.container.log.dir=/hadoop/2/yarn/logs/application_1736396393732_145750/container_e2322_1736396393732_145750_01_000191 -XX:OnOutOfMemoryError='kill %p' org.apache.spark.executor.YarnCoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@10.211.174.46:30201 --executor-id 107  --cores 18 --app-id application_1736396393732_145750 --resourceProfileId 0 > /hadoop/2/yarn/logs/application_1736396393732_145750/container_e2322_1736396393732_145750_01_000191/stdout 2> /hadoop/2/yarn/logs/application_1736396393732_145750/container_e2322_1736396393
 732_145750_01_000191/stderr
   Last 4096 bytes of stderr :
   W20250522 09:56:36.712019 321053 MemoryArbitrator.cpp:84] Query memory capacity[10.50GB] is set for NOOP arbitrator which has no capacity enforcement
   java: malloc.c:4687: _int_free: Assertion `heap->ar_ptr == av' failed.
   ```
   Conf:
   ```
   spark.gluten.sql.columnar.shuffle.sort.partitions.threshol=0
   spark.gluten.sql.columnar.shuffle.sort.columns.threshold=0
   spark.gluten.storage.hdfsViewfs.enabled=true
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org