You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "ayushi-agarwal (via GitHub)" <gi...@apache.org> on 2024/12/16 17:59:07 UTC

Re: [I] [VL] One task writes too many hive partitions causing OOM [incubator-gluten]

ayushi-agarwal commented on issue #7656:
URL: https://github.com/apache/incubator-gluten/issues/7656#issuecomment-2546290228

   The query is using velox writer to write parquet data and we are hitting this error.
   Executor Configs: 1 executor per node
   memoryOverhead: [amount: 1024]
   cores: [amount: 16]
   memory: [amount: 13312]
   offHeap: [amount: 80896]
   This is how the plan looks like for the stage. Many scan, filter project fallbacks to spark as from_json is not supported yet.
   There are 23000 tasks spawned for this stage. It is a non partitioned write.
   Need suggestions on how configurations can be tweaked to make this job work?
   
   ![image](https://github.com/user-attachments/assets/09da86a8-472d-4ba0-8a31-4132ceebe387)
   
   Executor logs:
   I20241216 17:07:15.881199 292073 WholeStageResultIterator.cc:234] Spill[root/root]:  successfully reclaimed total 0B with shrunken 0B and spilled 0B.
   I20241216 17:07:15.888181 292073 WholeStageResultIterator.cc:230] Spill[root/root]:  trying to request spill for 8.00MB.
   I20241216 17:07:15.888319 292073 WholeStageResultIterator.cc:234] Spill[root/root]:  successfully reclaimed total 0B with shrunken 0B and spilled 0B.
   I20241216 17:07:15.888373 292073 WholeStageResultIterator.cc:230] Spill[root/root]:  trying to request spill for 8.00MB.
   I20241216 17:07:15.888398 292073 WholeStageResultIterator.cc:234] Spill[root/root]:  successfully reclaimed total 0B with shrunken 0B and spilled 0B.
   I20241216 17:07:15.888561 292073 WholeStageResultIterator.cc:230] Spill[root/root]:  trying to request spill for 509.40MB.
   I20241216 17:07:15.888585 292073 WholeStageResultIterator.cc:234] Spill[root/root]:  successfully reclaimed total 0B with shrunken 0B and spilled 0B.
   I20241216 17:07:15.888605 292073 WholeStageResultIterator.cc:230] Spill[root/root]:  trying to request spill for 509.40MB.
   I20241216 17:07:15.888621 292073 WholeStageResultIterator.cc:234] Spill[root/root]:  successfully reclaimed total 0B with shrunken 0B and spilled 0B.
   24/12/16 17:07:15 ERROR ManagedReservationListener: Error reserving memory from target
   org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget$OutOfMemoryException: Not enough spark off-heap execution memory. Acquired: 8.0 MiB, granted: 2.0 MiB. Try tweaking config option spark.memory.offHeap.size to get larger space to run this application (if spark.gluten.memory.dynamic.offHeap.sizing.enabled is not enabled). 
   Current config settings: 
   	spark.gluten.memory.offHeap.size.in.bytes=75.0 GiB
   	spark.gluten.memory.task.offHeap.size.in.bytes=4.7 GiB
   	spark.gluten.memory.conservative.task.offHeap.size.in.bytes=2.3 GiB
   	spark.memory.offHeap.enabled=true
   	spark.gluten.memory.dynamic.offHeap.sizing.enabled=false
   Memory consumer stats: 
   	Task.28642:                                                 Current used bytes:    4.7 GiB, peak bytes:        N/A
   	\- Gluten.Tree.1251:                                        Current used bytes:    4.7 GiB, peak bytes:    4.7 GiB
   	   \- root.1251:                                            Current used bytes:    4.7 GiB, peak bytes:    4.7 GiB
   	      +- ArrowContextInstance.272:                          Current used bytes:    2.9 GiB, peak bytes:    4.3 GiB
   	      +- RowToColumnar.272:                                 Current used bytes: 1696.0 MiB, peak bytes: 1698.0 MiB
   	      |  \- single:                                         Current used bytes: 1696.0 MiB, peak bytes: 1696.0 MiB
   	      |     +- root:                                        Current used bytes: 1695.8 MiB, peak bytes: 1696.0 MiB
   	      |     |  \- default_leaf:                             Current used bytes: 1695.8 MiB, peak bytes: 1695.8 MiB
   	      |     \- gluten::MemoryAllocator:                     Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      +- VeloxWriter.272:                                   Current used bytes:  112.0 MiB, peak bytes:  184.0 MiB
   	      |  \- single:                                         Current used bytes:  112.0 MiB, peak bytes:  184.0 MiB
   	      |     +- root:                                        Current used bytes:  111.5 MiB, peak bytes:  184.0 MiB
   	      |     |  +- datasource.272:                           Current used bytes:  111.5 MiB, peak bytes:  184.0 MiB
   	      |     |  |  \- .general:                              Current used bytes:  111.5 MiB, peak bytes:  176.7 MiB
   	      |     |  \- default_leaf:                             Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     \- gluten::MemoryAllocator:                     Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      +- ColumnarToRow.393:                                 Current used bytes:   64.0 MiB, peak bytes:   64.0 MiB
   	      |  \- single:                                         Current used bytes:   64.0 MiB, peak bytes:   64.0 MiB
   	      |     +- root:                                        Current used bytes:   64.0 MiB, peak bytes:   64.0 MiB
   	      |     |  \- default_leaf:                             Current used bytes:   64.0 MiB, peak bytes:   64.0 MiB
   	      |     \- gluten::MemoryAllocator:                     Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      +- NativePlanEvaluator-1332.0:                        Current used bytes:    4.0 MiB, peak bytes:   16.0 MiB
   	      |  \- single:                                         Current used bytes:    4.0 MiB, peak bytes:   16.0 MiB
   	      |     +- root:                                        Current used bytes:  900.9 KiB, peak bytes:   15.0 MiB
   	      |     |  +- task.Gluten_Stage_26_TID_28642_VTID_1332: Current used bytes:  899.4 KiB, peak bytes:   14.0 MiB
   	      |     |  |  +- node.1:                                Current used bytes:  482.5 KiB, peak bytes:    2.0 MiB
   	      |     |  |  |  \- op.1.0.0.FilterProject:             Current used bytes:  482.5 KiB, peak bytes: 1443.3 KiB
   	      |     |  |  +- node.3:                                Current used bytes:  294.4 KiB, peak bytes:   11.0 MiB
   	      |     |  |  |  \- op.3.0.0.FilterProject:             Current used bytes:  294.4 KiB, peak bytes:   10.9 MiB
   	      |     |  |  +- node.2:                                Current used bytes:  122.5 KiB, peak bytes: 1024.0 KiB
   	      |     |  |  |  \- op.2.0.0.FilterProject:             Current used bytes:  122.5 KiB, peak bytes:  380.0 KiB
   	      |     |  |  \- node.0:                                Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     |  |     \- op.0.0.0.ValueStream:               Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     |  \- default_leaf:                             Current used bytes:   1536.0 B, peak bytes:   1664.0 B
   	      |     \- gluten::MemoryAllocator:                     Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      +- NativePlanEvaluator-1331.0:                        Current used bytes:    2.0 MiB, peak bytes:    8.0 MiB
   	      |  \- single:                                         Current used bytes:    2.0 MiB, peak bytes:    8.0 MiB
   	      |     +- root:                                        Current used bytes:  120.0 KiB, peak bytes:    2.0 MiB
   	      |     |  +- task.Gluten_Stage_26_TID_28642_VTID_1331: Current used bytes:  120.0 KiB, peak bytes:    2.0 MiB
   	      |     |  |  +- node.1:                                Current used bytes:   96.0 KiB, peak bytes: 1024.0 KiB
   	      |     |  |  |  \- op.1.0.0.Unnest:                    Current used bytes:   96.0 KiB, peak bytes:   96.0 KiB
   	      |     |  |  +- node.2:                                Current used bytes:   24.0 KiB, peak bytes: 1024.0 KiB
   	      |     |  |  |  \- op.2.0.0.FilterProject:             Current used bytes:   24.0 KiB, peak bytes:   24.0 KiB
   	      |     |  |  \- node.0:                                Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     |  |     \- op.0.0.0.ValueStream:               Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     |  \- default_leaf:                             Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     \- gluten::MemoryAllocator:                     Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      +- IteratorMetrics.1155.OverAcquire.0:                Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      +- VeloxWriter.272.OverAcquire.0:                     Current used bytes:      0.0 B, peak bytes:   55.2 MiB
   	      +- RowToColumnar.272.OverAcquire.0:                   Current used bytes:      0.0 B, peak bytes:  391.2 MiB
   	      +- NativePlanEvaluator-1331.0.OverAcquire.0:          Current used bytes:      0.0 B, peak bytes:    2.4 MiB
   	      +- IndicatorVectorBase#init.1251.OverAcquire.0:       Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      +- ColumnarToRow.393.OverAcquire.0:                   Current used bytes:      0.0 B, peak bytes:   19.2 MiB
   	      +- IteratorMetrics.1155:                              Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |  \- single:                                         Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     +- root:                                        Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     |  \- default_leaf:                             Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     \- gluten::MemoryAllocator:                     Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      +- NativePlanEvaluator-1332.0.OverAcquire.0:          Current used bytes:      0.0 B, peak bytes:    4.8 MiB
   	      \- IndicatorVectorBase#init.1251:                     Current used bytes:      0.0 B, peak bytes:      0.0 B
   	         \- single:                                         Current used bytes:      0.0 B, peak bytes:      0.0 B
   	            +- root:                                        Current used bytes:      0.0 B, peak bytes:      0.0 B
   	            |  \- default_leaf:                             Current used bytes:      0.0 B, peak bytes:      0.0 B
   	            \- gluten::MemoryAllocator:                     Current used bytes:      0.0 B, peak bytes:      0.0 B
   
   	at org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget.borrow(ThrowOnOomMemoryTarget.java:105) 
   	at org.apache.gluten.memory.listener.ManagedReservationListener.reserve(ManagedReservationListener.java:49) 
   	at org.apache.gluten.vectorized.NativeRowToColumnarJniWrapper.nativeConvertRowToColumnar(Native Method) 
   
   Task logs:
   org.apache.spark.SparkException: Task failed while writing rows.
   	at org.apache.spark.sql.errors.QueryExecutionErrors$.taskFailedWhileWritingRowsError(QueryExecutionErrors.scala:654)
   	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:448)
   	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$22(FileFormatWriter.scala:346)
   	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
   	at org.apache.spark.scheduler.Task.run(Task.scala:136)
   	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
   	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1505)
   	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
   	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   	at java.base/java.lang.Thread.run(Thread.java:829)
   Caused by: org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
   Error Source: RUNTIME
   Error Code: INVALID_STATE
   Reason: Operator::getOutput failed for [operator: ValueStream, plan node ID: 0]: Error during calling Java code from native code: org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget$OutOfMemoryException: Not enough spark off-heap execution memory. Acquired: 3.9 GiB, granted: 2.7 GiB. Try tweaking config option spark.memory.offHeap.size to get larger space to run this application (if spark.gluten.memory.dynamic.offHeap.sizing.enabled is not enabled). 
   Current config settings: 
   	spark.gluten.memory.offHeap.size.in.bytes=79.0 GiB
   	spark.gluten.memory.task.offHeap.size.in.bytes=4.9 GiB
   	spark.gluten.memory.conservative.task.offHeap.size.in.bytes=2.5 GiB
   	spark.memory.offHeap.enabled=true
   	spark.gluten.memory.dynamic.offHeap.sizing.enabled=false
   Memory consumer stats: 
   	Task.28426:                                                 Current used bytes:    2.3 GiB, peak bytes:        N/A
   	\- Gluten.Tree.1304:                                        Current used bytes:    2.3 GiB, peak bytes:    4.9 GiB
   	   \- root.1304:                                            Current used bytes:    2.3 GiB, peak bytes:    4.9 GiB
   	      +- ArrowContextInstance.276:                          Current used bytes: 2000.0 MiB, peak bytes:    4.6 GiB
   	      +- RowToColumnar.276:                                 Current used bytes:  152.0 MiB, peak bytes: 1952.0 MiB
   	      |  \- single:                                         Current used bytes:  152.0 MiB, peak bytes: 1952.0 MiB
   	      |     +- root:                                        Current used bytes:  151.6 MiB, peak bytes: 1952.0 MiB
   	      |     |  \- default_leaf:                             Current used bytes:  151.6 MiB, peak bytes: 1950.3 MiB
   	      |     \- gluten::MemoryAllocator:                     Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      +- VeloxWriter.276:                                   Current used bytes:   88.0 MiB, peak bytes:  176.0 MiB
   	      |  \- single:                                         Current used bytes:   88.0 MiB, peak bytes:  176.0 MiB
   	      |     +- root:                                        Current used bytes:   82.2 MiB, peak bytes:  176.0 MiB
   	      |     |  +- datasource.276:                           Current used bytes:   82.2 MiB, peak bytes:  176.0 MiB
   	      |     |  |  \- .general:                              Current used bytes:   82.2 MiB, peak bytes:  175.8 MiB
   	      |     |  \- default_leaf:                             Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     \- gluten::MemoryAllocator:                     Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      +- ColumnarToRow.413:                                 Current used bytes:   64.0 MiB, peak bytes:   64.0 MiB
   	      |  \- single:                                         Current used bytes:   64.0 MiB, peak bytes:   64.0 MiB
   	      |     +- root:                                        Current used bytes:   64.0 MiB, peak bytes:   64.0 MiB
   	      |     |  \- default_leaf:                             Current used bytes:   64.0 MiB, peak bytes:   64.0 MiB
   	      |     \- gluten::MemoryAllocator:                     Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      +- NativePlanEvaluator-1381.0:                        Current used bytes:    4.0 MiB, peak bytes:   16.0 MiB
   	      |  \- single:                                         Current used bytes:    4.0 MiB, peak bytes:   16.0 MiB
   	      |     +- root:                                        Current used bytes:  722.1 KiB, peak bytes:   16.0 MiB
   	      |     |  +- task.Gluten_Stage_26_TID_28426_VTID_1381: Current used bytes:  720.6 KiB, peak bytes:   15.0 MiB
   	      |     |  |  +- node.1:                                Current used bytes:  369.0 KiB, peak bytes:    2.0 MiB
   	      |     |  |  |  \- op.1.0.0.FilterProject:             Current used bytes:  369.0 KiB, peak bytes: 1523.0 KiB
   	      |     |  |  +- node.3:                                Current used bytes:  229.1 KiB, peak bytes:   12.0 MiB
   	      |     |  |  |  \- op.3.0.0.FilterProject:             Current used bytes:  229.1 KiB, peak bytes:   11.2 MiB
   	      |     |  |  +- node.2:                                Current used bytes:  122.5 KiB, peak bytes: 1024.0 KiB
   	      |     |  |  |  \- op.2.0.0.FilterProject:             Current used bytes:  122.5 KiB, peak bytes:  380.0 KiB
   	      |     |  |  \- node.0:                                Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     |  |     \- op.0.0.0.ValueStream:               Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     |  \- default_leaf:                             Current used bytes:   1536.0 B, peak bytes:   1664.0 B
   	      |     \- gluten::MemoryAllocator:                     Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      +- NativePlanEvaluator-1380.0:                        Current used bytes:    2.0 MiB, peak bytes:    8.0 MiB
   	      |  \- single:                                         Current used bytes:    2.0 MiB, peak bytes:    8.0 MiB
   	      |     +- root:                                        Current used bytes:  120.0 KiB, peak bytes:    2.0 MiB
   	      |     |  +- task.Gluten_Stage_26_TID_28426_VTID_1380: Current used bytes:  120.0 KiB, peak bytes:    2.0 MiB
   	      |     |  |  +- node.1:                                Current used bytes:   96.0 KiB, peak bytes: 1024.0 KiB
   	      |     |  |  |  \- op.1.0.0.Unnest:                    Current used bytes:   96.0 KiB, peak bytes:   96.0 KiB
   	      |     |  |  +- node.2:                                Current used bytes:   24.0 KiB, peak bytes: 1024.0 KiB
   	      |     |  |  |  \- op.2.0.0.FilterProject:             Current used bytes:   24.0 KiB, peak bytes:   24.0 KiB
   	      |     |  |  \- node.0:                                Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     |  |     \- op.0.0.0.ValueStream:               Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     |  \- default_leaf:                             Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     \- gluten::MemoryAllocator:                     Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      +- IndicatorVectorBase#init.1304:                     Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |  \- single:                                         Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     +- root:                                        Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     |  \- default_leaf:                             Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     \- gluten::MemoryAllocator:                     Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      +- VeloxWriter.276.OverAcquire.0:                     Current used bytes:      0.0 B, peak bytes:   52.8 MiB
   	      +- NativePlanEvaluator-1381.0.OverAcquire.0:          Current used bytes:      0.0 B, peak bytes:    4.8 MiB
   	      +- IteratorMetrics.1204.OverAcquire.0:                Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      +- ColumnarToRow.413.OverAcquire.0:                   Current used bytes:      0.0 B, peak bytes:   19.2 MiB
   	      +- RowToColumnar.276.OverAcquire.0:                   Current used bytes:      0.0 B, peak bytes:  585.6 MiB
   	      +- IteratorMetrics.1204:                              Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |  \- single:                                         Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     +- root:                                        Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     |  \- default_leaf:                             Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     \- gluten::MemoryAllocator:                     Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      +- IndicatorVectorBase#init.1304.OverAcquire.0:       Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      \- NativePlanEvaluator-1380.0.OverAcquire.0:          Current used bytes:      0.0 B, peak bytes:    2.4 MiB
   
   	at org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget.borrow(ThrowOnOomMemoryTarget.java:105)
   	at org.apache.gluten.memory.arrow.alloc.ManagedAllocationListener.onPreAllocation(ManagedAllocationListener.java:61)
   	at org.apache.gluten.shaded.org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:300)
   	at org.apache.gluten.shaded.org.apache.arrow.memory.RootAllocator.buffer(RootAllocator.java:29)
   	at org.apache.gluten.shaded.org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:280)
   	at org.apache.gluten.shaded.org.apache.arrow.memory.RootAllocator.buffer(RootAllocator.java:29)
   	at org.apache.gluten.execution.RowToVeloxColumnarExec$$anon$1.next(RowToVeloxColumnarExec.scala:200)
   	at org.apache.gluten.execution.RowToVeloxColumnarExec$$anon$1.next(RowToVeloxColumnarExec.scala:138)
   	at org.apache.gluten.iterator.IteratorsV1$InvocationFlowProtection.next(IteratorsV1.scala:178)
   	at org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.next(IteratorsV1.scala:79)
   	at org.apache.gluten.iterator.IteratorsV1$PayloadCloser.next(IteratorsV1.scala:41)
   	at scala.collection.convert.Wrappers$IteratorWrapper.next(Wrappers.scala:33)
   	at org.apache.gluten.vectorized.ColumnarBatchInIterator.next(ColumnarBatchInIterator.java:39)
   	at org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
   	at org.apache.gluten.vectorized.ColumnarBatchOutIterator.hasNext0(ColumnarBatchOutIterator.java:57)
   	at org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:39)
   	at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
   	at org.apache.gluten.iterator.IteratorsV1$InvocationFlowProtection.hasNext(IteratorsV1.scala:159)
   	at org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.hasNext(IteratorsV1.scala:71)
   	at org.apache.gluten.iterator.IteratorsV1$PayloadCloser.hasNext(IteratorsV1.scala:37)
   	at org.apache.gluten.iterator.IteratorsV1$LifeTimeAccumulator.hasNext(IteratorsV1.scala:100)
   	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
   	at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithIterator(FileFormatDataWriter.scala:95)
   	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$1(FileFormatWriter.scala:429)
   	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1539)
   	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:438)
   	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$22(FileFormatWriter.scala:346)
   	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
   	at org.apache.spark.scheduler.Task.run(Task.scala:136)
   	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
   	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1505)
   	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
   	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   	at java.base/java.lang.Thread.run(Thread.java:829)
   
   Retriable: False
   Function: operator()
   File: /home/abc/incubator-gluten/ep/build-velox/build/velox_ep/velox/exec/Driver.cpp
   Line: 601
   Stack trace:
   0  _ZN8facebook5velox7process10StackTraceC1Ei
   1  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
   2  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
   3  _ZZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEEENKUlvE3_clEv.cold
   4  _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
    5  _ZN8facebook5velox4exec6Driver4nextEPN5folly10SemiFutureINS3_4UnitEEE
    6  _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
    7  _ZN6gluten24WholeStageResultIterator4nextEv
    8  Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeHasNext
    9  0x00007f36695f9a7b
   
   	at org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:41)
   	at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
   	at org.apache.gluten.iterator.IteratorsV1$InvocationFlowProtection.hasNext(IteratorsV1.scala:159)
   	at org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.hasNext(IteratorsV1.scala:71)
   	at org.apache.gluten.iterator.IteratorsV1$PayloadCloser.hasNext(IteratorsV1.scala:37)
   	at org.apache.gluten.iterator.IteratorsV1$LifeTimeAccumulator.hasNext(IteratorsV1.scala:100)
   	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
   	at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithIterator(FileFormatDataWriter.scala:95)
   	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$1(FileFormatWriter.scala:429)
   	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1539)
   	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:438)
   	... 9 more
   Caused by: org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
   Error Source: RUNTIME
   Error Code: INVALID_STATE
   Reason: Operator::getOutput failed for [operator: ValueStream, plan node ID: 0]: Error during calling Java code from native code: org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget$OutOfMemoryException: Not enough spark off-heap execution memory. Acquired: 3.9 GiB, granted: 2.7 GiB. Try tweaking config option spark.memory.offHeap.size to get larger space to run this application (if spark.gluten.memory.dynamic.offHeap.sizing.enabled is not enabled). 
   Current config settings: 
   	spark.gluten.memory.offHeap.size.in.bytes=79.0 GiB
   	spark.gluten.memory.task.offHeap.size.in.bytes=4.9 GiB
   	spark.gluten.memory.conservative.task.offHeap.size.in.bytes=2.5 GiB
   	spark.memory.offHeap.enabled=true
   	spark.gluten.memory.dynamic.offHeap.sizing.enabled=false
   Memory consumer stats: 
   	Task.28426:                                                 Current used bytes:    2.3 GiB, peak bytes:        N/A
   	\- Gluten.Tree.1304:                                        Current used bytes:    2.3 GiB, peak bytes:    4.9 GiB
   	   \- root.1304:                                            Current used bytes:    2.3 GiB, peak bytes:    4.9 GiB
   	      +- ArrowContextInstance.276:                          Current used bytes: 2000.0 MiB, peak bytes:    4.6 GiB
   	      +- RowToColumnar.276:                                 Current used bytes:  152.0 MiB, peak bytes: 1952.0 MiB
   	      |  \- single:                                         Current used bytes:  152.0 MiB, peak bytes: 1952.0 MiB
   	      |     +- root:                                        Current used bytes:  151.6 MiB, peak bytes: 1952.0 MiB
   	      |     |  \- default_leaf:                             Current used bytes:  151.6 MiB, peak bytes: 1950.3 MiB
   	      |     \- gluten::MemoryAllocator:                     Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      +- VeloxWriter.276:                                   Current used bytes:   88.0 MiB, peak bytes:  176.0 MiB
   	      |  \- single:                                         Current used bytes:   88.0 MiB, peak bytes:  176.0 MiB
   	      |     +- root:                                        Current used bytes:   82.2 MiB, peak bytes:  176.0 MiB
   	      |     |  +- datasource.276:                           Current used bytes:   82.2 MiB, peak bytes:  176.0 MiB
   	      |     |  |  \- .general:                              Current used bytes:   82.2 MiB, peak bytes:  175.8 MiB
   	      |     |  \- default_leaf:                             Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     \- gluten::MemoryAllocator:                     Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      +- ColumnarToRow.413:                                 Current used bytes:   64.0 MiB, peak bytes:   64.0 MiB
   	      |  \- single:                                         Current used bytes:   64.0 MiB, peak bytes:   64.0 MiB
   	      |     +- root:                                        Current used bytes:   64.0 MiB, peak bytes:   64.0 MiB
   	      |     |  \- default_leaf:                             Current used bytes:   64.0 MiB, peak bytes:   64.0 MiB
   	      |     \- gluten::MemoryAllocator:                     Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      +- NativePlanEvaluator-1381.0:                        Current used bytes:    4.0 MiB, peak bytes:   16.0 MiB
   	      |  \- single:                                         Current used bytes:    4.0 MiB, peak bytes:   16.0 MiB
   	      |     +- root:                                        Current used bytes:  722.1 KiB, peak bytes:   16.0 MiB
   	      |     |  +- task.Gluten_Stage_26_TID_28426_VTID_1381: Current used bytes:  720.6 KiB, peak bytes:   15.0 MiB
   	      |     |  |  +- node.1:                                Current used bytes:  369.0 KiB, peak bytes:    2.0 MiB
   	      |     |  |  |  \- op.1.0.0.FilterProject:             Current used bytes:  369.0 KiB, peak bytes: 1523.0 KiB
   	      |     |  |  +- node.3:                                Current used bytes:  229.1 KiB, peak bytes:   12.0 MiB
   	      |     |  |  |  \- op.3.0.0.FilterProject:             Current used bytes:  229.1 KiB, peak bytes:   11.2 MiB
   	      |     |  |  +- node.2:                                Current used bytes:  122.5 KiB, peak bytes: 1024.0 KiB
   	      |     |  |  |  \- op.2.0.0.FilterProject:             Current used bytes:  122.5 KiB, peak bytes:  380.0 KiB
   	      |     |  |  \- node.0:                                Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     |  |     \- op.0.0.0.ValueStream:               Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     |  \- default_leaf:                             Current used bytes:   1536.0 B, peak bytes:   1664.0 B
   	      |     \- gluten::MemoryAllocator:                     Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      +- NativePlanEvaluator-1380.0:                        Current used bytes:    2.0 MiB, peak bytes:    8.0 MiB
   	      |  \- single:                                         Current used bytes:    2.0 MiB, peak bytes:    8.0 MiB
   	      |     +- root:                                        Current used bytes:  120.0 KiB, peak bytes:    2.0 MiB
   	      |     |  +- task.Gluten_Stage_26_TID_28426_VTID_1380: Current used bytes:  120.0 KiB, peak bytes:    2.0 MiB
   	      |     |  |  +- node.1:                                Current used bytes:   96.0 KiB, peak bytes: 1024.0 KiB
   	      |     |  |  |  \- op.1.0.0.Unnest:                    Current used bytes:   96.0 KiB, peak bytes:   96.0 KiB
   	      |     |  |  +- node.2:                                Current used bytes:   24.0 KiB, peak bytes: 1024.0 KiB
   	      |     |  |  |  \- op.2.0.0.FilterProject:             Current used bytes:   24.0 KiB, peak bytes:   24.0 KiB
   	      |     |  |  \- node.0:                                Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     |  |     \- op.0.0.0.ValueStream:               Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     |  \- default_leaf:                             Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     \- gluten::MemoryAllocator:                     Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      +- IndicatorVectorBase#init.1304:                     Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |  \- single:                                         Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     +- root:                                        Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     |  \- default_leaf:                             Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     \- gluten::MemoryAllocator:                     Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      +- VeloxWriter.276.OverAcquire.0:                     Current used bytes:      0.0 B, peak bytes:   52.8 MiB
   	      +- NativePlanEvaluator-1381.0.OverAcquire.0:          Current used bytes:      0.0 B, peak bytes:    4.8 MiB
   	      +- IteratorMetrics.1204.OverAcquire.0:                Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      +- ColumnarToRow.413.OverAcquire.0:                   Current used bytes:      0.0 B, peak bytes:   19.2 MiB
   	      +- RowToColumnar.276.OverAcquire.0:                   Current used bytes:      0.0 B, peak bytes:  585.6 MiB
   	      +- IteratorMetrics.1204:                              Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |  \- single:                                         Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     +- root:                                        Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     |  \- default_leaf:                             Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      |     \- gluten::MemoryAllocator:                     Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      +- IndicatorVectorBase#init.1304.OverAcquire.0:       Current used bytes:      0.0 B, peak bytes:      0.0 B
   	      \- NativePlanEvaluator-1380.0.OverAcquire.0:          Current used bytes:      0.0 B, peak bytes:    2.4 MiB
   
   	at org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget.borrow(ThrowOnOomMemoryTarget.java:105)
   	at org.apache.gluten.memory.arrow.alloc.ManagedAllocationListener.onPreAllocation(ManagedAllocationListener.java:61)
   	at org.apache.gluten.shaded.org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:300)
   	at org.apache.gluten.shaded.org.apache.arrow.memory.RootAllocator.buffer(RootAllocator.java:29)
   	at org.apache.gluten.shaded.org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:280)
   	at org.apache.gluten.shaded.org.apache.arrow.memory.RootAllocator.buffer(RootAllocator.java:29)
   	at org.apache.gluten.execution.RowToVeloxColumnarExec$$anon$1.next(RowToVeloxColumnarExec.scala:200)
   	at org.apache.gluten.execution.RowToVeloxColumnarExec$$anon$1.next(RowToVeloxColumnarExec.scala:138)
   	at org.apache.gluten.iterator.IteratorsV1$InvocationFlowProtection.next(IteratorsV1.scala:178)
   	at org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.next(IteratorsV1.scala:79)
   	at org.apache.gluten.iterator.IteratorsV1$PayloadCloser.next(IteratorsV1.scala:41)
   	at scala.collection.convert.Wrappers$IteratorWrapper.next(Wrappers.scala:33)
   	at org.apache.gluten.vectorized.ColumnarBatchInIterator.next(ColumnarBatchInIterator.java:39)
   	at org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
   	at org.apache.gluten.vectorized.ColumnarBatchOutIterator.hasNext0(ColumnarBatchOutIterator.java:57)
   	at org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:39)
   	at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
   	at org.apache.gluten.iterator.IteratorsV1$InvocationFlowProtection.hasNext(IteratorsV1.scala:159)
   	at org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.hasNext(IteratorsV1.scala:71)
   	at org.apache.gluten.iterator.IteratorsV1$PayloadCloser.hasNext(IteratorsV1.scala:37)
   	at org.apache.gluten.iterator.IteratorsV1$LifeTimeAccumulator.hasNext(IteratorsV1.scala:100)
   	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
   	at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithIterator(FileFormatDataWriter.scala:95)
   	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$1(FileFormatWriter.scala:429)
   	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1539)
   	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:438)
   	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$22(FileFormatWriter.scala:346)
   	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
   	at org.apache.spark.scheduler.Task.run(Task.scala:136)
   	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
   	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1505)
   	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
   	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   	at java.base/java.lang.Thread.run(Thread.java:829)
   
   Retriable: False
   Function: operator()
   File: /home/abc/incubator-gluten/ep/build-velox/build/velox_ep/velox/exec/Driver.cpp
   Line: 601
   Stack trace:
    0  _ZN8facebook5velox7process10StackTraceC1Ei
    1  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
    2  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
    3  _ZZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEEENKUlvE3_clEv.cold
    4  _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
    5  _ZN8facebook5velox4exec6Driver4nextEPN5folly10SemiFutureINS3_4UnitEEE
    6  _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
    7  _ZN6gluten24WholeStageResultIterator4nextEv
    8  Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeHasNext
    9  0x00007f36695f9a7b
   
   	at org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
   	at org.apache.gluten.vectorized.ColumnarBatchOutIterator.hasNext0(ColumnarBatchOutIterator.java:57)
   	at org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:39)
   	... 19 more


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org