You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "zhouyuan (via GitHub)" <gi...@apache.org> on 2025/08/12 11:04:03 UTC
[I] [VL] Mismatch on default value of kMaxSpillLevel [incubator-gluten]
zhouyuan opened a new issue, #10417:
URL: https://github.com/apache/incubator-gluten/issues/10417
### Backend
VL (Velox)
### Bug description
Gluten:
https://github.com/apache/incubator-gluten/blob/7a3b0e38b2736b99d0461f0aa8018b1cb512078b/cpp/velox/compute/WholeStageResultIterator.cc#L529
Velox:
https://github.com/facebookincubator/velox/blob/main/velox/exec/Spill.h#L254
### Gluten version
_No response_
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Mismatch on default value of kMaxSpillLevel [incubator-gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on issue #10417:
URL: https://github.com/apache/incubator-gluten/issues/10417#issuecomment-3181201873
I interpreted this from the stack of a failed query. will verify and try to make more inputs for this
```
(org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Spill level 4 exceeds max spill level 3
Retriable: False
Context: Operator: HashBuild[3] 1
Function: SpillPartitionId
File: /velox/velox/exec/Spill.cpp
Line: 447
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZN8facebook5velox4exec16SpillPartitionIdC1ES2_j
# 4 _ZN8facebook5velox4exec11SpillerBase11finishSpillERSt3mapINS1_16SpillPartitionIdESt10unique_ptrINS1_14SpillPartitionESt14default_deleteIS6_EESt4lessIS4_ESaISt4pairIKS4_S9_EEE
# 5 _ZN8facebook5velox4exec9HashBuild15finishHashBuildEv
# 6 _ZN8facebook5velox4exec9HashBuild19noMoreInputInternalEv
# 7 _ZN8facebook5velox4exec9HashBuild20postHashBuildProcessEv
# 8 _ZN8facebook5velox4exec9HashBuild9isBlockedEPN5folly10SemiFutureINS3_4UnitEEE
# 9 _ZZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEEENKUlvE0_clEv
# 10 _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
# 11 _ZN8facebook5velox4exec6Driver4nextEPN5folly10SemiFutureINS3_4UnitEEERPNS1_8OperatorERNS1_14BlockingReasonE
# 12 _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 13 _ZN6gluten24WholeStageResultIterator4nextEv
# 14 Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeHasNext
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Mismatch on default value of kMaxSpillLevel [incubator-gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on issue #10417:
URL: https://github.com/apache/incubator-gluten/issues/10417#issuecomment-3179964105
@PaulOberg1 Thanks! Sure, please help to make a patch to fix this. Also for your reference, you may simply comment "take" in the comments and the issue will be assigned to you automatically
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Mismatch on default value of kMaxSpillLevel [incubator-gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan closed issue #10417: [VL] Mismatch on default value of kMaxSpillLevel
URL: https://github.com/apache/incubator-gluten/issues/10417
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Mismatch on default value of kMaxSpillLevel [incubator-gluten]
Posted by "PaulOberg1 (via GitHub)" <gi...@apache.org>.
PaulOberg1 commented on issue #10417:
URL: https://github.com/apache/incubator-gluten/issues/10417#issuecomment-3179879160
Hi @zhouyuan , I'm interested in contributing to this :). Could I be assigned?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Mismatch on default value of kMaxSpillLevel [incubator-gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on issue #10417:
URL: https://github.com/apache/incubator-gluten/issues/10417#issuecomment-3181311001
Update:
The initial issue appears to have been a false alarm. The root cause is the hard-coded maximum spill level.
https://github.com/facebookincubator/velox/blob/main/velox/exec/Spill.h#L254
Which is a limitation of Velox for now
https://facebookincubator.github.io/velox/develop/spilling.html#hash-join
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Mismatch on default value of kMaxSpillLevel [incubator-gluten]
Posted by "PaulOberg1 (via GitHub)" <gi...@apache.org>.
PaulOberg1 commented on issue #10417:
URL: https://github.com/apache/incubator-gluten/issues/10417#issuecomment-3179965791
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Mismatch on default value of kMaxSpillLevel [incubator-gluten]
Posted by "PaulOberg1 (via GitHub)" <gi...@apache.org>.
PaulOberg1 commented on issue #10417:
URL: https://github.com/apache/incubator-gluten/issues/10417#issuecomment-3180151946
So to clarify you just want the default value on Gluten changed to 3?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org