You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Noemi Pap-Takacs (Code Review)" <ge...@cloudera.org> on 2025/01/10 10:20:16 UTC
[Impala-ASF-CR] IMPALA-13655: UPDATE redundantly accumulates memory in HDFS WRITER
Noemi Pap-Takacs has uploaded this change for review. ( http://gerrit.cloudera.org:8080/22325
Change subject: IMPALA-13655: UPDATE redundantly accumulates memory in HDFS WRITER
......................................................................
IMPALA-13655: UPDATE redundantly accumulates memory in HDFS WRITER
When IcebergUpdateImpl created the table sink it didn't set
'inputIsClustered' to true. Therefore HdfsTableSink expected
random input and kept the output writers open for every partition,
which resulted in high memory consumption and potentially an
OOM error when the number of partitions are high.
Since we actually sort the rows before the sink we can set
'inputIsClustered' to true, which means HdfsTableSink can write
files one by one, because whenever it gets a row that belongs
to a new partition it knows that it can close the current output
writer, and open a new one.
Testing:
- e2e regression test
Change-Id: I9bad335cc946364fc612e8aaf90858eaabd7c4af
---
M fe/src/main/java/org/apache/impala/analysis/IcebergUpdateImpl.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-update-partitions.test
2 files changed, 23 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/22325/1
--
To view, visit http://gerrit.cloudera.org:8080/22325
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I9bad335cc946364fc612e8aaf90858eaabd7c4af
Gerrit-Change-Number: 22325
Gerrit-PatchSet: 1
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
[Impala-ASF-CR] IMPALA-13655: UPDATE redundantly accumulates memory in HDFS WRITER
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22325 )
Change subject: IMPALA-13655: UPDATE redundantly accumulates memory in HDFS WRITER
......................................................................
Patch Set 2: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/22325
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bad335cc946364fc612e8aaf90858eaabd7c4af
Gerrit-Change-Number: 22325
Gerrit-PatchSet: 2
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 10 Jan 2025 16:02:52 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13655: UPDATE redundantly accumulates memory in HDFS WRITER
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22325 )
Change subject: IMPALA-13655: UPDATE redundantly accumulates memory in HDFS WRITER
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/17938/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22325
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bad335cc946364fc612e8aaf90858eaabd7c4af
Gerrit-Change-Number: 22325
Gerrit-PatchSet: 1
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 10 Jan 2025 10:47:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13655: UPDATE redundantly accumulates memory in HDFS WRITER
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22325 )
Change subject: IMPALA-13655: UPDATE redundantly accumulates memory in HDFS WRITER
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/22325
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bad335cc946364fc612e8aaf90858eaabd7c4af
Gerrit-Change-Number: 22325
Gerrit-PatchSet: 2
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 10 Jan 2025 10:32:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13655: UPDATE redundantly accumulates memory in HDFS WRITER
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/22325 )
Change subject: IMPALA-13655: UPDATE redundantly accumulates memory in HDFS WRITER
......................................................................
IMPALA-13655: UPDATE redundantly accumulates memory in HDFS WRITER
When IcebergUpdateImpl created the table sink it didn't set
'inputIsClustered' to true. Therefore HdfsTableSink expected
random input and kept the output writers open for every partition,
which resulted in high memory consumption and potentially an
OOM error when the number of partitions are high.
Since we actually sort the rows before the sink we can set
'inputIsClustered' to true, which means HdfsTableSink can write
files one by one, because whenever it gets a row that belongs
to a new partition it knows that it can close the current output
writer, and open a new one.
Testing:
- e2e regression test
Change-Id: I9bad335cc946364fc612e8aaf90858eaabd7c4af
Reviewed-on: http://gerrit.cloudera.org:8080/22325
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/analysis/IcebergUpdateImpl.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-update-partitions.test
2 files changed, 23 insertions(+), 1 deletion(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/22325
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I9bad335cc946364fc612e8aaf90858eaabd7c4af
Gerrit-Change-Number: 22325
Gerrit-PatchSet: 3
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-13655: UPDATE redundantly accumulates memory in HDFS WRITER
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/22325 )
Change subject: IMPALA-13655: UPDATE redundantly accumulates memory in HDFS WRITER
......................................................................
Patch Set 1: Code-Review+2
Thanks for fixing this!
--
To view, visit http://gerrit.cloudera.org:8080/22325
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bad335cc946364fc612e8aaf90858eaabd7c4af
Gerrit-Change-Number: 22325
Gerrit-PatchSet: 1
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 10 Jan 2025 10:31:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13655: UPDATE redundantly accumulates memory in HDFS WRITER
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22325 )
Change subject: IMPALA-13655: UPDATE redundantly accumulates memory in HDFS WRITER
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11302/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/22325
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bad335cc946364fc612e8aaf90858eaabd7c4af
Gerrit-Change-Number: 22325
Gerrit-PatchSet: 2
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 10 Jan 2025 10:32:05 +0000
Gerrit-HasComments: No