You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "zhztheplayer (via GitHub)" <gi...@apache.org> on 2024/10/11 08:36:56 UTC

[I] [VL] Q97 cross-task spilling hangs [incubator-gluten]

zhztheplayer opened a new issue, #7243:
URL: https://github.com/apache/incubator-gluten/issues/7243

   Happens when Velox task A grows buffers created by task B, during which spill is requested on task A again.
   
   Stack:
   ```
   syscall 0x00007f53533e673d
   folly::detail::nativeFutexWaitImpl Futex.cpp:126
   folly::detail::futexWaitImpl Futex.cpp:254
   folly::detail::futexWait<…> Futex-inl.h:94
   folly::detail::MemoryIdler::futexWait<…> MemoryIdler.h:128
   folly::fibers::Baton::waitThread Baton.cpp:70
   folly::fibers::Baton::wait<…> Baton-inl.h:54
   folly::fibers::Baton::wait Baton.cpp:46
   folly::futures::detail::waitImpl<…> Future-inl.h:2230
   folly::SemiFuture<folly::Unit>::wait() & Future-inl.h:2337
   folly::SemiFuture<folly::Unit>::wait() && Future-inl.h:2343
   facebook::velox::exec::Task::MemoryReclaimer::reclaimTask Task.cpp:2933
   facebook::velox::exec::Task::MemoryReclaimer::reclaim Task.cpp:2907
   facebook::velox::memory::MemoryReclaimer::reclaim MemoryArbitrator.cpp:266
   gluten::ListenableArbitrator::shrinkCapacity VeloxMemoryManager.cc:122
   gluten::WholeStageResultIterator::spillFixedSize WholeStageResultIterator.cc:239
   gluten::ResultIterator::spillFixedSize ResultIterator.h:71
   Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeSpill JniWrapper.cc:516
   <unknown> 0x00007f533d018427
   <unknown> 0x0000000000800000
   <unknown> 0x00007f4e38003260
   <unknown> 0x00000007c1420c28
   <unknown> 0x00007f52d8dc7890
   <unknown> 0x00007f4e38003000
   <unknown> 0x00007f52d8dc7838
   <unknown> 0x0000000000000000
   ```
   
   Log:
   
   https://github.com/apache/incubator-gluten/actions/runs/10856089295/job/30131530452?pr=7237


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [VL] Q97 cross-task spilling hangs [incubator-gluten]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #7243:
URL: https://github.com/apache/incubator-gluten/issues/7243#issuecomment-2406916997

   Reopening. Observed in a local TPC benchmark where Velox tasks are usually broken into several in one Spark task
   
   Error:
   ```
   #0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
   #1  0x00007f044de29e3e in folly::detail::(anonymous namespace)::nativeFutexWaitImpl (waitMask=4294967295, absSteadyTime=0x0, absSystemTime=0x0, expected=4294967293, addr=0x7f05d4d037f8)
       at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.05.20.00-935b4335b7.clean/folly/detail/Futex.cpp:126
   #2  folly::detail::futexWaitImpl (futex=futex@entry=0x7f05d4d037f8, expected=expected@entry=4294967293, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x0,
       waitMask=waitMask@entry=4294967295) at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.05.20.00-935b4335b7.clean/folly/detail/Futex.cpp:254
   #3  0x00007f044de71416 in folly::detail::futexWait<std::atomic<unsigned int> > (waitMask=4294967295, expected=4294967293, futex=0x7f05d4d037f8)
       at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.05.20.00-935b4335b7.clean/folly/detail/Futex-inl.h:94
   #4  folly::detail::MemoryIdler::futexWait<std::atomic<unsigned int>, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (expected=4294967293, waitMask=4294967295, stackToRetain=1024,
       timeoutVariationFrac=0.5, idleTimeout=..., fut=std::atomic<unsigned int> = { 4294967293 })
       at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.05.20.00-935b4335b7.clean/folly/detail/MemoryIdler.h:128
   #5  folly::fibers::Baton::waitThread (this=0x7f05d4d037f8) at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.05.20.00-935b4335b7.clean/folly/fibers/Baton.cpp:70
   #6  0x00007f044de716e0 in folly::fibers::Baton::wait<folly::fibers::Baton::wait()::<lambda()> > (mainContextFunc=..., this=0x7f05d4d037f8)
       at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.05.20.00-935b4335b7.clean/folly/fibers/Baton-inl.h:54
   #7  folly::fibers::Baton::wait (this=this@entry=0x7f05d4d037f8) at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.05.20.00-935b4335b7.clean/folly/fibers/Baton.cpp:46
   #8  0x00007f044a513abe in folly::futures::detail::waitImpl<folly::SemiFuture<folly::Unit>, folly::Unit> (f=...)
       at /opt/gluten/dev/vcpkg/vcpkg_installed/x64-linux-avx/include/folly/futures/Future-inl.h:2230
   #9  0x00007f044a513f90 in folly::SemiFuture<folly::Unit>::wait() & (this=0x7f05d4d03928) at /opt/gluten/dev/vcpkg/vcpkg_installed/x64-linux-avx/include/folly/futures/Future-inl.h:2337
   #10 0x00007f044b39aa48 in folly::SemiFuture<folly::Unit>::wait() && (this=0x7f05d4d03928) at /opt/gluten/dev/vcpkg/vcpkg_installed/x64-linux-avx/include/folly/futures/Future-inl.h:2343
   #11 facebook::velox::exec::Task::MemoryReclaimer::reclaimTask (this=0x7f04807707b0, task=std::shared_ptr<facebook::velox::exec::Task> (use count 6, weak count 3) = {...}, targetBytes=18035507,
       maxWaitMs=0, stats=...) at /opt/gluten/ep/build-velox/build/velox_ep/velox/exec/Task.cpp:2996
   #12 0x00007f044b39b0d1 in facebook::velox::exec::Task::MemoryReclaimer::reclaim (this=0x7f04807707b0, pool=0x7f02a670f450, targetBytes=18035507, maxWaitMs=0, stats=...)
       at /opt/gluten/ep/build-velox/build/velox_ep/velox/exec/Task.cpp:2970
   #13 0x00007f044a70a4f9 in facebook::velox::memory::MemoryReclaimer::reclaim (this=<optimized out>, pool=<optimized out>, targetBytes=<optimized out>, maxWaitMs=0, stats=...)
       at /opt/gluten/ep/build-velox/build/velox_ep/velox/common/memory/MemoryArbitrator.cpp:266
   #14 0x00007f044a539dc5 in gluten::ListenableArbitrator::shrinkCapacity (this=0x7f04810b4010, targetBytes=18035507, allowSpill=<optimized out>, allowAbort=<optimized out>)
       at /opt/gluten/cpp/velox/memory/VeloxMemoryManager.cc:122
   #15 0x00007f044a51c285 in gluten::WholeStageResultIterator::spillFixedSize (this=0x7f04803627c0, size=<optimized out>) at /opt/gluten/cpp/velox/compute/WholeStageResultIterator.cc:250
   #16 0x00007f05ce621c35 in gluten::ResultIterator::spillFixedSize (size=18035507, this=0x7f048044b380) at /opt/gluten/cpp/core/compute/ResultIterator.h:71
   #17 Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeSpill (env=0x7f046860aa60, wrapper=<optimized out>, iterHandle=75136157876230, size=18035507)
       at /opt/gluten/cpp/core/jni/JniWrapper.cc:516
   #18 0x00007f06dd018427 in ?? ()
   #19 0x0000000772211e98 in ?? ()
   #20 0x4908b36b977f1f00 in ?? ()
   #21 0x0000000600000001 in ?? ()
   #22 0x00007f05cce302e8 in ?? ()
   #23 0x00007f05d4d03fe8 in ?? ()
   #24 0x00007f05d4d03f78 in ?? ()
   #25 0x0000000000000000 in ?? ()
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [VL] Q97 cross-task spilling hangs [incubator-gluten]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer closed issue #7243: [VL] Q97 cross-task spilling hangs
URL: https://github.com/apache/incubator-gluten/issues/7243


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [VL] Q97 cross-task spilling hangs [incubator-gluten]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer closed issue #7243: [VL] Q97 cross-task spilling hangs
URL: https://github.com/apache/incubator-gluten/issues/7243


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org