You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Yuqi Du (Code Review)" <ge...@cloudera.org> on 2023/04/04 07:02:54 UTC
[kudu-CR] [java] KUDU-3455 Reduce space complexity of hash partition pruning for in-list predicate
Hello Mahesh Reddy, Alexey Serbin, Ashwani Raina, Yingchun Lai, Yifan Zhang, Kudu Jenkins, Wang Xixu,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19568
to look at the new patch set (#9).
Change subject: [java] KUDU-3455 Reduce space complexity of hash partition pruning for in-list predicate
......................................................................
[java] KUDU-3455 Reduce space complexity of hash partition pruning for in-list predicate
Logic of pruning hash partitions for in-list predicate in Kudu Java client
has a high space complexity, and it may cause java-client to go out of memory.
At the same time, there are many deep copies for 'PartialRow', which makes the
current algorithm slow.
This patch fixes the problems and provides a recursive algorithm, that
uses an approach like DFS-based algorithm to pick all combinations for
every in-list columns and try to release PartialRow objects ASAP.
At the same time, new algorithm has a good speedup by reducing lots of heavy
copies of 'PartialRow' objects. A performance test case show that new
algorithm has around 100x gain over older one when latter doesn't cause
OOM.
After Yifan Zhang's reminder, same problem was found to be present in
cpp-client per code-review. I'll study it later and fix it in another patch.
Change-Id: Icd6f213cb705e1b2a001562cc7cebe4164281723
---
M java/kudu-client/src/main/java/org/apache/kudu/client/PartitionPruner.java
M java/kudu-client/src/test/java/org/apache/kudu/client/TestPartitionPruner.java
2 files changed, 392 insertions(+), 21 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/68/19568/9
--
To view, visit http://gerrit.cloudera.org:8080/19568
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icd6f213cb705e1b2a001562cc7cebe4164281723
Gerrit-Change-Number: 19568
Gerrit-PatchSet: 9
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Yingchun Lai <la...@apache.org>
Gerrit-Reviewer: Yuqi Du <sh...@gmail.com>