You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "FelixYBW (via GitHub)" <gi...@apache.org> on 2024/08/18 07:15:08 UTC
[I] [VL] stop == StopReason::kBlock || stop == StopReason::kAtEnd || stop == StopReason::kAlreadyTerminated || stop == StopReason::kTerminate [incubator-gluten]
FelixYBW opened a new issue, #6908:
URL: https://github.com/apache/incubator-gluten/issues/6908
### Backend
VL (Velox)
### Bug description
The error raised again, it's usually caused by background thread exit but there is no coredump this time.
```
File: /home/binweiyang/gluten/ep/build-velox/build/velox_ep/velox/exec/Driver.cpp
Line: 370
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorENS1_22CompileTimeEmptyStringEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZN8facebook5velox4exec6Driver4nextERSt10shared_ptrINS1_13BlockingStateEE
# 4 _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 5 _ZN6gluten24WholeStageResultIterator4nextEv
# 6 Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeHasNext
# 7 0x00007f3f2c3bf427
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.hasNextInternal(ColumnarBatchOutIterator.java:61)
at org.apache.gluten.vectorized.GeneralOutIterator.hasNext(GeneralOutIterator.java:37)
... 31 more
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.hasNextInternal(ColumnarBatchOutIterator.java:61)
at org.apache.gluten.vectorized.GeneralOutIterator.hasNext(GeneralOutIterator.java:37)
... 18 more
```
@zhztheplayer
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] stop == StopReason::kBlock || stop == StopReason::kAtEnd || stop == StopReason::kAlreadyTerminated || stop == StopReason::kTerminate [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #6908:
URL: https://github.com/apache/incubator-gluten/issues/6908#issuecomment-2300457778
confirmed, #6934 fixed the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] stop == StopReason::kBlock || stop == StopReason::kAtEnd || stop == StopReason::kAlreadyTerminated || stop == StopReason::kTerminate [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW closed issue #6908: [VL] stop == StopReason::kBlock || stop == StopReason::kAtEnd || stop == StopReason::kAlreadyTerminated || stop == StopReason::kTerminate
URL: https://github.com/apache/incubator-gluten/issues/6908
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] stop == StopReason::kBlock || stop == StopReason::kAtEnd || stop == StopReason::kAlreadyTerminated || stop == StopReason::kTerminate [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #6908:
URL: https://github.com/apache/incubator-gluten/issues/6908#issuecomment-2300443010
> Interesting, setting iothreads hurts performance.
It's a cluster issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] stop == StopReason::kBlock || stop == StopReason::kAtEnd || stop == StopReason::kAlreadyTerminated || stop == StopReason::kTerminate [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #6908:
URL: https://github.com/apache/incubator-gluten/issues/6908#issuecomment-2295521966
Is newest Gluten used? Had seen similar issue in GHA CI weeks ago which disappeared for a while then. Was not able to reproduce locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] stop == StopReason::kBlock || stop == StopReason::kAtEnd || stop == StopReason::kAlreadyTerminated || stop == StopReason::kTerminate [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #6908:
URL: https://github.com/apache/incubator-gluten/issues/6908#issuecomment-2295164507
OK, bad news, it's caused by io threads in scan. The issue is gone once we set iothread=0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] stop == StopReason::kBlock || stop == StopReason::kAtEnd || stop == StopReason::kAlreadyTerminated || stop == StopReason::kTerminate [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #6908:
URL: https://github.com/apache/incubator-gluten/issues/6908#issuecomment-2295171641
Interesting, setting iothreads hurts performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org