You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "zhengchenyu (via GitHub)" <gi...@apache.org> on 2023/08/02 11:42:07 UTC
[GitHub] [incubator-uniffle] zhengchenyu opened a new issue, #1070: [Bug] [tez] shuffle server may leak if not register remote storage.
zhengchenyu opened a new issue, #1070:
URL: https://github.com/apache/incubator-uniffle/issues/1070
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
In our cluster, we found require_buffer_failed increase rapidly. Then we found used_buffer_size keep very high which is greater than high watermark even though no application is running for long time.
Then we found error log as below:
```
[ERROR] 2023-08-02 12:09:00,329 ProcessEventThread DefaultFlushEventHandler processNextEvent - Exception happened when process event.
org.apache.uniffle.common.exception.RssException: Unexpected storage type!
at org.apache.uniffle.server.DefaultFlushEventHandler.processNextEvent(DefaultFlushEventHandler.java:122)
at org.apache.uniffle.server.DefaultFlushEventHandler.eventLoop(DefaultFlushEventHandler.java:109)
at java.base/java.lang.Thread.run(Thread.java:829)
```
By some debug, we found when some big shuffle data occur, will fall back to hadoop storage, but here hadoop storage is null, so many flush event fail.
Why hadoop storage is null? because registerRemoteStorage is never called.
Why registerRemoteStorage is never called? because the conf in TezRemoteShuffleManager is not merge the 'rss.storage.type' from dynamic configuration. And I did not set 'rss.storage.type' in client side.
### Affects Version(s)
master
### Uniffle Server Log Output
_No response_
### Uniffle Engine Log Output
_No response_
### Uniffle Server Configurations
_No response_
### Uniffle Engine Configurations
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi closed issue #1070: [Bug] [tez] shuffle server may leak if not register remote storage.
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi closed issue #1070: [Bug] [tez] shuffle server may leak if not register remote storage.
URL: https://github.com/apache/incubator-uniffle/issues/1070
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org