You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uniffle.apache.org by "leixm (via GitHub)" <gi...@apache.org> on 2023/06/05 03:29:20 UTC
[GitHub] [incubator-uniffle] leixm opened a new issue, #926: [Bug] ShuffleReadClientImpl occurs StackOverflowError.
leixm opened a new issue, #926:
URL: https://github.com/apache/incubator-uniffle/issues/926
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
The following exception occurred in the four attempts of the same taskļ¼the probability of this kind of problem recurring is relatively low. About thousands of apps will have this type of exception in one app, causing the app attempt to retry, and every time this exception occurs, the app is also different.
```
org.apache.spark.SparkException: Task failed while writing rows.
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:321)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:209)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:208)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:121)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1363)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.StackOverflowError
at io.grpc.internal.RetriableStream.cancel(RetriableStream.java:493)
at io.grpc.internal.ClientCallImpl.sendMessageInternal(ClientCallImpl.java:528)
at io.grpc.internal.ClientCallImpl.sendMessage(ClientCallImpl.java:506)
at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:317)
at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:227)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:154)
at org.apache.uniffle.proto.ShuffleServerGrpc$ShuffleServerBlockingStub.getMemoryShuffleData(ShuffleServerGrpc.java:768)
at org.apache.uniffle.client.impl.grpc.ShuffleServerGrpcClient.getInMemoryShuffleData(ShuffleServerGrpcClient.java:627)
at org.apache.uniffle.storage.handler.impl.MemoryClientReadHandler.readShuffleData(MemoryClientReadHandler.java:94)
at org.apache.uniffle.storage.handler.impl.ComposedClientReadHandler.readShuffleData(ComposedClientReadHandler.java:93)
at org.apache.uniffle.client.impl.ShuffleReadClientImpl.read(ShuffleReadClientImpl.java:224)
at org.apache.uniffle.client.impl.ShuffleReadClientImpl.readShuffleBlockData(ShuffleReadClientImpl.java:160)
at org.apache.uniffle.client.impl.ShuffleReadClientImpl.readShuffleBlockData(ShuffleReadClientImpl.java:214)
at org.apache.uniffle.client.impl.ShuffleReadClientImpl.readShuffleBlockData(ShuffleReadClientImpl.java:214)
at org.apache.uniffle.client.impl.ShuffleReadClientImpl.readShuffleBlockData(ShuffleReadClientImpl.java:214)
at org.apache.uniffle.client.impl.ShuffleReadClientImpl.readShuffleBlockData(ShuffleReadClientImpl.java:214)
at org.apache.uniffle.client.impl.ShuffleReadClientImpl.readShuffleBlockData(ShuffleReadClientImpl.java:214)
at org.apache.uniffle.client.impl.ShuffleReadClientImpl.readShuffleBlockData(ShuffleReadClientImpl.java:214)
at org.apache.uniffle.client.impl.ShuffleReadClientImpl.readShuffleBlockData(ShuffleReadClientImpl.java:214)
at org.apache.uniffle.client.impl.ShuffleReadClientImpl.readShuffleBlockData(ShuffleReadClientImpl.java:214)
at org.apache.uniffle.client.impl.ShuffleReadClientImpl.readShuffleBlockData(ShuffleReadClientImpl.java:214)
at org.apache.uniffle.client.impl.ShuffleReadClientImpl.readShuffleBlockData(ShuffleReadClientImpl.java:214)
at org.apache.uniffle.client.impl.ShuffleReadClientImpl.readShuffleBlockData(ShuffleReadClientImpl.java:214)
at org.apache.uniffle.client.impl.ShuffleReadClientImpl.readShuffleBlockData(ShuffleReadClientImpl.java:214)
at org.apache.uniffle.client.impl.ShuffleReadClientImpl.readShuffleBlockData(ShuffleReadClientImpl.java:214)```
### Affects Version(s)
0.6.0
### Uniffle Server Log Output
```logtalk
Corresponding ShuffleServer has been repeatedly printing the following log
`[INFO] 2023-06-04 20:42:40,023 Grpc-839 ShuffleServerGrpcService getMemoryShuffleData - Successfully getInMemoryShuffleData cost 0 ms with 318133 bytes shuffle data for appId[xxxxx], shuffleId[0], partitionId[12]`
```
### Uniffle Engine Log Output
```logtalk
Before the exception is thrown, the following logs have been repeatedly appearing, indicating that the client has been requesting the GetInMemoryShuffleData interface of ShuffleServer.
`23/06/04 20:42:43 INFO grpc.ShuffleServerGrpcClient: GetInMemoryShuffleData from xxxxx:19999 for appId[xxxxx], shuffleId[0], partitionId[12] cost 2 ms`
```
### Uniffle Server Configurations
```yaml
rss.rpc.server.port 19999
rss.jetty.http.port 19998
rss.storage.basePath /aa/bb/cc,/aa1/bb/cc
rss.storage.type MEMORY_LOCALFILE_HDFS
rss.coordinator.quorum xxxxxxx
rss.server.buffer.capacity 40gb
rss.server.read.buffer.capacity 20gb
rss.server.flush.thread.alive 50
rss.server.flush.threadPool.size 4
rss.server.disk.capacity 1t
rss.server.heartbeat.timeout 60000
rss.server.heartbeat.interval 10000
rss.rpc.message.max.size 1073741824
rss.server.preAllocation.expired 120000
rss.server.commit.timeout 600000
rss.server.app.expired.withoutHeartbeat 300000
rss.server.buffer.flush.enabled true
rss.server.tags high_priority
rss.server.health.check.enable true
rss.server.multistorage.fallback.strategy.class org.apache.uniffle.server.storage.LocalStorageManagerFallbackStrategy
rss.server.flush.cold.storage.threshold.size 50gb
rss.server.buffer.flush.threshold 64m
rss.server.max.concurrency.of.single.partition.writer 20
rss.server.huge-partition.size.threshold 50g
rss.server.huge-partition.memory.limit.ratio 0.2
```
```
### Uniffle Engine Configurations
```yaml
--conf spark.rss.client.shuffle.data.distribution.type LOCAL_ORDER
```
### Additional context
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] ShuffleReadClientImpl occurs StackOverflowError. [incubator-uniffle]
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #926:
URL: https://github.com/apache/incubator-uniffle/issues/926#issuecomment-1754592698
@jerqi @zuston What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Bug] ShuffleReadClientImpl occurs StackOverflowError. [incubator-uniffle]
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #926:
URL: https://github.com/apache/incubator-uniffle/issues/926#issuecomment-1756726504
Block deduplication on the ShuffleServer side may have a greater impact on performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Bug] ShuffleReadClientImpl occurs StackOverflowError. [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #926:
URL: https://github.com/apache/incubator-uniffle/issues/926#issuecomment-1756878551
Maybe we could add a new id like flushId to solve this problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #926: [Bug] ShuffleReadClientImpl occurs StackOverflowError.
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on issue #926:
URL: https://github.com/apache/incubator-uniffle/issues/926#issuecomment-1575991772
> @jerqi @zuston Has this problem occurred in your production environment?
I haven't seen this error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] ShuffleReadClientImpl occurs StackOverflowError. [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on issue #926:
URL: https://github.com/apache/incubator-uniffle/issues/926#issuecomment-1756898713
Using the composite block id? like (last-1_block_id, last_block_id) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Bug] ShuffleReadClientImpl occurs StackOverflowError. [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #926:
URL: https://github.com/apache/incubator-uniffle/issues/926#issuecomment-1754607424
I think you are right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Bug] ShuffleReadClientImpl occurs StackOverflowError. [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #926:
URL: https://github.com/apache/incubator-uniffle/issues/926#issuecomment-1758858185
> How about using skip list to store blocks in `ShuffleBuffer`?
Maybe we can't, we should guarantee that the order is immutable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #926: [Bug] ShuffleReadClientImpl occurs StackOverflowError.
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #926:
URL: https://github.com/apache/incubator-uniffle/issues/926#issuecomment-1575987216
@jerqi @zuston Has this problem occurred in your production environment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] ShuffleReadClientImpl occurs StackOverflowError. [incubator-uniffle]
Posted by "xianjingfeng (via GitHub)" <gi...@apache.org>.
xianjingfeng commented on issue #926:
URL: https://github.com/apache/incubator-uniffle/issues/926#issuecomment-1758814427
How about using skip list to store blocks in `ShuffleBuffer`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Bug] ShuffleReadClientImpl occurs StackOverflowError. [incubator-uniffle]
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #926:
URL: https://github.com/apache/incubator-uniffle/issues/926#issuecomment-1754586337
I think this exception will occur when duplicate blocks are sent. Assume that there are two blocks with equal blockIds in a ShuffleBuffer. Before flush occurs, getMemoryShuffleData will always be called and never end.
As you can see from the log below, the first read is 5050999 bytes, subsequent reads are all 4625271 bytes, and the reading never ends. There should be duplicate blockIds.
```
23/10/10 03:25:44 INFO grpc.ShuffleServerGrpcClient: GetInMemoryShuffleData from xxx:19999 for appId[application_1695193790407_9195629_1696874216324], shuffleId[9], partitionId[252] cost 1114 ms, get 5050999 bytes
23/10/10 03:25:44 INFO grpc.ShuffleServerGrpcClient: GetInMemoryShuffleData from xxx:19999 for appId[application_1695193790407_9195629_1696874216324], shuffleId[9], partitionId[252] cost 236 ms, get 4625271 bytes
23/10/10 03:25:45 INFO grpc.ShuffleServerGrpcClient: GetInMemoryShuffleData from xxx:19999 for appId[application_1695193790407_9195629_1696874216324], shuffleId[9], partitionId[252] cost 642 ms, get 4625271 bytes
23/10/10 03:25:46 INFO grpc.ShuffleServerGrpcClient: GetInMemoryShuffleData from xxx:19999 for appId[application_1695193790407_9195629_1696874216324], shuffleId[9], partitionId[252] cost 815 ms, get 4625271 bytes
23/10/10 03:25:47 INFO grpc.ShuffleServerGrpcClient: GetInMemoryShuffleData from xx:19999 for appId[application_1695193790407_9195629_1696874216324], shuffleId[9], partitionId[252] cost 868 ms, get 4625271 bytes
......
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Bug] ShuffleReadClientImpl occurs StackOverflowError. [incubator-uniffle]
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #926:
URL: https://github.com/apache/incubator-uniffle/issues/926#issuecomment-1754592327
There are two blocks with blockId=3 in the picture. The first time getMemoryShuffledata will read all blocks, and the second time it will read from blockId=4 to the last block. This reciprocation will never end. [ShuffleBuffer Image](https://ibb.co/XJDQcvh)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Bug] ShuffleReadClientImpl occurs StackOverflowError. [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on issue #926:
URL: https://github.com/apache/incubator-uniffle/issues/926#issuecomment-1754604538
Sounds right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Bug] ShuffleReadClientImpl occurs StackOverflowError. [incubator-uniffle]
Posted by "lifeSo (via GitHub)" <gi...@apache.org>.
lifeSo commented on issue #926:
URL: https://github.com/apache/incubator-uniffle/issues/926#issuecomment-1790297940
> @jerqi @zuston Has this problem occurred in your production environment?
We see this Exception before on prd.
Later on prd, never see the Exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Bug] ShuffleReadClientImpl occurs StackOverflowError. [incubator-uniffle]
Posted by "xianjingfeng (via GitHub)" <gi...@apache.org>.
xianjingfeng commented on issue #926:
URL: https://github.com/apache/incubator-uniffle/issues/926#issuecomment-1790232897
@leixm @zuston
I have discussed this issue with @jerqi offline. We think we can use skipList to store blocks in `ShuffleBuffer`, but the key of the skipList should be a self-increment id.
Another reason to use skipList is that if we assign a lot of memory(1TB) to the shuffle server, it will be slow while getting memory data with current implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Bug] ShuffleReadClientImpl occurs StackOverflowError. [incubator-uniffle]
Posted by "xianjingfeng (via GitHub)" <gi...@apache.org>.
xianjingfeng commented on issue #926:
URL: https://github.com/apache/incubator-uniffle/issues/926#issuecomment-1758869731
> > How about using skip list to store blocks in `ShuffleBuffer`?
>
> Maybe we can't, we should guarantee that the order is immutable.
Why we should guarantee that the order is immutable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Bug] ShuffleReadClientImpl occurs StackOverflowError. [incubator-uniffle]
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #926:
URL: https://github.com/apache/incubator-uniffle/issues/926#issuecomment-1756722081
Do you have any optimization suggestions for this situation? @jerqi @zuston
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #926: [Bug] ShuffleReadClientImpl occurs StackOverflowError.
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #926:
URL: https://github.com/apache/incubator-uniffle/issues/926#issuecomment-1576183479
I haven't seen this error, too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org