You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "hpplayer (via GitHub)" <gi...@apache.org> on 2025/08/12 03:10:37 UTC
[I] No forward progress made [incubator-gluten]
hpplayer opened a new issue, #10412:
URL: https://github.com/apache/incubator-gluten/issues/10412
### Backend
VL (Velox)
### Bug description
Sometimes the job fails with following error, and sometimes a rerun of the same job just succeeds
```
Job Aborted:Error: Job aborted.Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 1018) (100.67.244.114 executor 50): org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Operator::getOutput failed for [operator: ValueStream, plan node ID: 0]: Error during calling Java code from native code: org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Codec: no forward progress made
at org.apache.gluten.iterator.ClosableIterator.next(ClosableIterator.java:53)
at org.apache.spark.shuffle.CelebornColumnarBatchSerializerInstance$TaskDeserializationStream.liftedTree1$1(VeloxCelebornColumnarBatchSerializer.scala:197)
at org.apache.spark.shuffle.CelebornColumnarBatchSerializerInstance$TaskDeserializationStream.readValue(VeloxCelebornColumnarBatchSerializer.scala:196)
at org.apache.spark.shuffle.CelebornColumnarBatchSerializerInstance$TaskDeserializationStream$$anon$1.getNext(VeloxCelebornColumnarBatchSerializer.scala:149)
at org.apache.spark.shuffle.CelebornColumnarBatchSerializerInstance$TaskDeserializationStream$$anon$1.hasNext(VeloxCelebornColumnarBatchSerializer.scala:160)
...
Retriable: False
Function: operator()
File: /gluten/ep/build-velox/build/velox_ep/velox/exec/Driver.cpp
Line: 601
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEEENKUlvE3_clEv.cold
# 4 _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
# 5 _ZN8facebook5velox4exec6Driver4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 6 _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 7 _ZN6gluten24WholeStageResultIterator4nextEv
# 8 Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeHasNext
# 9 0x000000003001f930
```
### Gluten version
Gluten-1.3
### Spark version
Spark-3.2.x
### Spark configurations
spark.sql.parquet.compression.codec: zstd
let me know if I need to provide other info
### System information
_No response_
### Relevant logs
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Codec: no forward progress made [incubator-gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on issue #10412:
URL: https://github.com/apache/incubator-gluten/issues/10412#issuecomment-3180142928
@FelixYBW I haven't seen this error before. Could you elaborate more about the "compression code conflict"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Codec: no forward progress made [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #10412:
URL: https://github.com/apache/incubator-gluten/issues/10412#issuecomment-3177554037
@zhztheplayer @marin-ma It's related to the compression code conflict in folly. I remember I hit this once as well. Do you have the impression?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Codec: no forward progress made [incubator-gluten]
Posted by "kerwin-zk (via GitHub)" <gi...@apache.org>.
kerwin-zk commented on issue #10412:
URL: https://github.com/apache/incubator-gluten/issues/10412#issuecomment-3235532628
@hpplayer Could you help check if the problematic stage triggered the shuffle read skew optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Codec: no forward progress made [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #10412:
URL: https://github.com/apache/incubator-gluten/issues/10412#issuecomment-3180247334
@FelixYBW We used to encounter some codec-missing errors (like [this](https://github.com/apache/incubator-gluten/issues/4263)) but this one looks like another case to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Codec: no forward progress made [gluten]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on issue #10412:
URL: https://github.com/apache/gluten/issues/10412#issuecomment-4922319131
We also encountered this issue in Gluten 1.3, it's caused by the velox bug, and fixed in https://github.com/facebookincubator/velox/pull/12784.
When spilled compressed data size is more than a single buffer, it get wrong `compressedSize` and cause this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org