You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "cxzl25 (via GitHub)" <gi...@apache.org> on 2024/07/31 14:48:24 UTC
[PR] [CELEBORN-1533] Log location when CelebornInputStream#fillBuffer fails [celeborn]
cxzl25 opened a new pull request, #2655:
URL: https://github.com/apache/celeborn/pull/2655
### What changes were proposed in this pull request?
This PR aims to log location when `CelebornInputStream#fillBuffer` fails.
### Why are the changes needed?
```java
24/07/30 22:29:04,181 [Executor task launch worker for task 353.0 in stage 10.0 (TID 50641)] ERROR Executor: Exception in task 353.0 in stage 10.0 (TID 50641)
com.github.luben.zstd.ZstdException: Data corruption detected
at com.github.luben.zstd.ZstdDecompressCtx.decompressByteArray(ZstdDecompressCtx.java:205)
at com.github.luben.zstd.Zstd.decompressByteArray(Zstd.java:439)
at org.apache.celeborn.client.compress.ZstdDecompressor.decompress(ZstdDecompressor.java:54)
at org.apache.celeborn.client.read.CelebornInputStream$CelebornInputStreamImpl.fillBuffer(CelebornInputStream.java:563)
at org.apache.celeborn.client.read.CelebornInputStream$CelebornInputStreamImpl.read(CelebornInputStream.java:464)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.sparkproject.guava.io.ByteStreams.read(ByteStreams.java:899)
at org.sparkproject.guava.io.ByteStreams.readFully(ByteStreams.java:733)
at org.apache.spark.sql.execution.UnsafeRowSerializerInstance$$anon$2$$anon$3.next(UnsafeRowSerializer.scala:127)
at org.apache.spark.sql.execution.UnsafeRowSerializerInstance$$anon$2$$anon$3.next(UnsafeRowSerializer.scala:110)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:496)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:29)
at org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:40)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage6.sort_addToSorter_0$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage6.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:759)
at org.apache.spark.sql.execution.RowIteratorFromScala.advanceNext(RowIterator.scala:82)
at org.apache.spark.sql.execution.joins.SortMergeJoinScanner.advancedStreamed(SortMergeJoinExec.scala:1065)
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
GA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1533] Log location when CelebornInputStream#fillBuffer fails [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #2655:
URL: https://github.com/apache/celeborn/pull/2655#discussion_r1699524886
##########
client/src/main/java/org/apache/celeborn/client/read/CelebornInputStream.java:
##########
@@ -665,6 +665,13 @@ private boolean fillBuffer() throws IOException {
return hasData;
} catch (IOException e) {
+ logger.error(
+ "Failed to read data from chunk. AppShuffleId {}, shuffleId {}, partitionId {}, location {}",
Review Comment:
```suggestion
"Failed to fill buffer from chunk. AppShuffleId {}, shuffleId {}, partitionId {}, location {}",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1533] Log location when CelebornInputStream#fillBuffer fails [celeborn]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on PR #2655:
URL: https://github.com/apache/celeborn/pull/2655#issuecomment-2264569985
> but It seems Celeborn should also support throw FetchFailedException when Data corruption detected
hi @cxzl25 would you like to follow up the comments for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1533] Log location when CelebornInputStream#fillBuffer fails [celeborn]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on PR #2655:
URL: https://github.com/apache/celeborn/pull/2655#issuecomment-2265109103
> It seems Celeborn should also support throw FetchFailedException when `Data corruption detected`
Because in celeborn we are using ZstdDecompressCtx, decompression fails and it throws ZstdException, which is not inherited from IOException.
But in Spark, ZstdInputStreamNoFinalizer is used, which throws ZstdIOException and inherits IOException.
So this exception is not caught in CelebornInputStreamImpl#fillBuffer, we can do this next.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1533] Log location when CelebornInputStream#fillBuffer fails [celeborn]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on PR #2655:
URL: https://github.com/apache/celeborn/pull/2655#issuecomment-2264562129
thanks, merged to main, branch-0.5, branch-0.4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1533] Log location when CelebornInputStream#fillBuffer fails [celeborn]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei closed pull request #2655: [CELEBORN-1533] Log location when CelebornInputStream#fillBuffer fails
URL: https://github.com/apache/celeborn/pull/2655
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1533] Log location when CelebornInputStream#fillBuffer fails [celeborn]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #2655:
URL: https://github.com/apache/celeborn/pull/2655#discussion_r1699175374
##########
client/src/main/java/org/apache/celeborn/client/read/CelebornInputStream.java:
##########
@@ -680,6 +681,9 @@ private boolean fillBuffer() throws IOException {
}
}
throw ioe;
Review Comment:
how about `logger.error("....", ioe)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1533] Log location when CelebornInputStream#fillBuffer fails [celeborn]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on code in PR #2655:
URL: https://github.com/apache/celeborn/pull/2655#discussion_r1699337215
##########
client/src/main/java/org/apache/celeborn/client/read/CelebornInputStream.java:
##########
@@ -680,6 +681,9 @@ private boolean fillBuffer() throws IOException {
}
}
throw ioe;
Review Comment:
Because there is a `reportShuffleFetchFailure` RPC call, it may fail, so the log is still output when an exception occurs at the beginning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1533] Log location when CelebornInputStream#fillBuffer fails [celeborn]
Posted by "xumanbu (via GitHub)" <gi...@apache.org>.
xumanbu commented on PR #2655:
URL: https://github.com/apache/celeborn/pull/2655#issuecomment-3544676670
@SteNicholas @cxzl25 In our production environment, decompression failure occasionally occurs. Do you know the root cause of decompression failure?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org