You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/07/11 03:31:49 UTC
[GitHub] [hudi] Zhangshunyu opened a new issue, #6075: [SUPPORT]
Zhangshunyu opened a new issue, #6075:
URL: https://github.com/apache/hudi/issues/6075
Hudi's current driver cache management has some problems:
1) The cache is only shared within the session. In different sessions, because the cache is not shared, the cache information of the same table is loaded repeatedly;
2) When a session is released, the corresponding cache of the session is not released, causing the cache to accumulate until oom
3) When the session is first built, the query table will create the relation of the table, and all file status information will be loaded during the process of building the relation
Combining the above three points leads to the following results:
1) Multiple session connections are connected to the same driver for concurrent execution, memory * N will eventually lead to driver oom
2 Each query needs to build a relation, which is equivalent to executing the first query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #6075: Hudi's current driver cache management has some problems
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #6075:
URL: https://github.com/apache/hudi/issues/6075#issuecomment-1186694753
@Zhangshunyu thanks for the analysis. i see you've already done some profiling and code analysis
> When a session is released, the corresponding cache of the session is not released, causing the cache to accumulate until oom
would you take a step further share the related code paths for this issue? or even maybe file a PR directly and we can discuss further from there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zhangshunyu commented on issue #6075: Hudi's current driver cache management has some problems
Posted by GitBox <gi...@apache.org>.
Zhangshunyu commented on issue #6075:
URL: https://github.com/apache/hudi/issues/6075#issuecomment-1186698754
@xushiyan Hi shiyan, thanks for you reply, i find this is a spark problem as spark didnt release the cache for 1 session when the session is closed. we have fixed this in spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan closed issue #6075: Hudi's current driver cache management has some problems
Posted by GitBox <gi...@apache.org>.
xushiyan closed issue #6075: Hudi's current driver cache management has some problems
URL: https://github.com/apache/hudi/issues/6075
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #6075: Hudi's current driver cache management has some problems
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #6075:
URL: https://github.com/apache/hudi/issues/6075#issuecomment-1186707533
@Zhangshunyu you meant fixed in spark source code? it would be interesting to see the upstream patch for spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org