You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "zhztheplayer (via GitHub)" <gi...@apache.org> on 2024/10/11 08:36:56 UTC
[I] [VL] Q97 cross-task spilling hangs [incubator-gluten]
zhztheplayer opened a new issue, #7243:
URL: https://github.com/apache/incubator-gluten/issues/7243
Happens when Velox task A grows buffers created by task B, during which spill is requested on task A again.
Stack:
```
syscall 0x00007f53533e673d
folly::detail::nativeFutexWaitImpl Futex.cpp:126
folly::detail::futexWaitImpl Futex.cpp:254
folly::detail::futexWait<…> Futex-inl.h:94
folly::detail::MemoryIdler::futexWait<…> MemoryIdler.h:128
folly::fibers::Baton::waitThread Baton.cpp:70
folly::fibers::Baton::wait<…> Baton-inl.h:54
folly::fibers::Baton::wait Baton.cpp:46
folly::futures::detail::waitImpl<…> Future-inl.h:2230
folly::SemiFuture<folly::Unit>::wait() & Future-inl.h:2337
folly::SemiFuture<folly::Unit>::wait() && Future-inl.h:2343
facebook::velox::exec::Task::MemoryReclaimer::reclaimTask Task.cpp:2933
facebook::velox::exec::Task::MemoryReclaimer::reclaim Task.cpp:2907
facebook::velox::memory::MemoryReclaimer::reclaim MemoryArbitrator.cpp:266
gluten::ListenableArbitrator::shrinkCapacity VeloxMemoryManager.cc:122
gluten::WholeStageResultIterator::spillFixedSize WholeStageResultIterator.cc:239
gluten::ResultIterator::spillFixedSize ResultIterator.h:71
Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeSpill JniWrapper.cc:516
<unknown> 0x00007f533d018427
<unknown> 0x0000000000800000
<unknown> 0x00007f4e38003260
<unknown> 0x00000007c1420c28
<unknown> 0x00007f52d8dc7890
<unknown> 0x00007f4e38003000
<unknown> 0x00007f52d8dc7838
<unknown> 0x0000000000000000
```
Log:
https://github.com/apache/incubator-gluten/actions/runs/10856089295/job/30131530452?pr=7237
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Q97 cross-task spilling hangs [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #7243:
URL: https://github.com/apache/incubator-gluten/issues/7243#issuecomment-2406916997
Reopening. Observed in a local TPC benchmark where Velox tasks are usually broken into several in one Spark task
Error:
```
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007f044de29e3e in folly::detail::(anonymous namespace)::nativeFutexWaitImpl (waitMask=4294967295, absSteadyTime=0x0, absSystemTime=0x0, expected=4294967293, addr=0x7f05d4d037f8)
at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.05.20.00-935b4335b7.clean/folly/detail/Futex.cpp:126
#2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f05d4d037f8, expected=expected@entry=4294967293, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x0,
waitMask=waitMask@entry=4294967295) at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.05.20.00-935b4335b7.clean/folly/detail/Futex.cpp:254
#3 0x00007f044de71416 in folly::detail::futexWait<std::atomic<unsigned int> > (waitMask=4294967295, expected=4294967293, futex=0x7f05d4d037f8)
at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.05.20.00-935b4335b7.clean/folly/detail/Futex-inl.h:94
#4 folly::detail::MemoryIdler::futexWait<std::atomic<unsigned int>, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (expected=4294967293, waitMask=4294967295, stackToRetain=1024,
timeoutVariationFrac=0.5, idleTimeout=..., fut=std::atomic<unsigned int> = { 4294967293 })
at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.05.20.00-935b4335b7.clean/folly/detail/MemoryIdler.h:128
#5 folly::fibers::Baton::waitThread (this=0x7f05d4d037f8) at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.05.20.00-935b4335b7.clean/folly/fibers/Baton.cpp:70
#6 0x00007f044de716e0 in folly::fibers::Baton::wait<folly::fibers::Baton::wait()::<lambda()> > (mainContextFunc=..., this=0x7f05d4d037f8)
at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.05.20.00-935b4335b7.clean/folly/fibers/Baton-inl.h:54
#7 folly::fibers::Baton::wait (this=this@entry=0x7f05d4d037f8) at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.05.20.00-935b4335b7.clean/folly/fibers/Baton.cpp:46
#8 0x00007f044a513abe in folly::futures::detail::waitImpl<folly::SemiFuture<folly::Unit>, folly::Unit> (f=...)
at /opt/gluten/dev/vcpkg/vcpkg_installed/x64-linux-avx/include/folly/futures/Future-inl.h:2230
#9 0x00007f044a513f90 in folly::SemiFuture<folly::Unit>::wait() & (this=0x7f05d4d03928) at /opt/gluten/dev/vcpkg/vcpkg_installed/x64-linux-avx/include/folly/futures/Future-inl.h:2337
#10 0x00007f044b39aa48 in folly::SemiFuture<folly::Unit>::wait() && (this=0x7f05d4d03928) at /opt/gluten/dev/vcpkg/vcpkg_installed/x64-linux-avx/include/folly/futures/Future-inl.h:2343
#11 facebook::velox::exec::Task::MemoryReclaimer::reclaimTask (this=0x7f04807707b0, task=std::shared_ptr<facebook::velox::exec::Task> (use count 6, weak count 3) = {...}, targetBytes=18035507,
maxWaitMs=0, stats=...) at /opt/gluten/ep/build-velox/build/velox_ep/velox/exec/Task.cpp:2996
#12 0x00007f044b39b0d1 in facebook::velox::exec::Task::MemoryReclaimer::reclaim (this=0x7f04807707b0, pool=0x7f02a670f450, targetBytes=18035507, maxWaitMs=0, stats=...)
at /opt/gluten/ep/build-velox/build/velox_ep/velox/exec/Task.cpp:2970
#13 0x00007f044a70a4f9 in facebook::velox::memory::MemoryReclaimer::reclaim (this=<optimized out>, pool=<optimized out>, targetBytes=<optimized out>, maxWaitMs=0, stats=...)
at /opt/gluten/ep/build-velox/build/velox_ep/velox/common/memory/MemoryArbitrator.cpp:266
#14 0x00007f044a539dc5 in gluten::ListenableArbitrator::shrinkCapacity (this=0x7f04810b4010, targetBytes=18035507, allowSpill=<optimized out>, allowAbort=<optimized out>)
at /opt/gluten/cpp/velox/memory/VeloxMemoryManager.cc:122
#15 0x00007f044a51c285 in gluten::WholeStageResultIterator::spillFixedSize (this=0x7f04803627c0, size=<optimized out>) at /opt/gluten/cpp/velox/compute/WholeStageResultIterator.cc:250
#16 0x00007f05ce621c35 in gluten::ResultIterator::spillFixedSize (size=18035507, this=0x7f048044b380) at /opt/gluten/cpp/core/compute/ResultIterator.h:71
#17 Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeSpill (env=0x7f046860aa60, wrapper=<optimized out>, iterHandle=75136157876230, size=18035507)
at /opt/gluten/cpp/core/jni/JniWrapper.cc:516
#18 0x00007f06dd018427 in ?? ()
#19 0x0000000772211e98 in ?? ()
#20 0x4908b36b977f1f00 in ?? ()
#21 0x0000000600000001 in ?? ()
#22 0x00007f05cce302e8 in ?? ()
#23 0x00007f05d4d03fe8 in ?? ()
#24 0x00007f05d4d03f78 in ?? ()
#25 0x0000000000000000 in ?? ()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Q97 cross-task spilling hangs [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer closed issue #7243: [VL] Q97 cross-task spilling hangs
URL: https://github.com/apache/incubator-gluten/issues/7243
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Q97 cross-task spilling hangs [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer closed issue #7243: [VL] Q97 cross-task spilling hangs
URL: https://github.com/apache/incubator-gluten/issues/7243
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org