You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org> on 2025/10/14 09:14:27 UTC

[Impala-ASF-CR] IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23537


Change subject: IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation
......................................................................

IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Impala crashes when it needs to write multiple delete files per
partition in a single DELETE operation. It is because
IcebergBufferedDeleteSink has its own DmlExecState object, but
sometimes the methods in TableSinkBase use the RuntimeState's
DmlExecState object. I.e. it can happen that we add a partition
to the IcebergBufferedDeleteSink's DmlExecState, but later we
expect to find it in the RuntimeState's DmlExecState.

This patch adds new methods to TableSinkBase that are specific
for writing delete files, and they always take a DmlExecState
object as a parameter. They are now used by IcebergBufferedDeleteSink.

Testing
 * added e2e tests

Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
---
M be/src/exec/iceberg-buffered-delete-sink.cc
M be/src/exec/table-sink-base.cc
M be/src/exec/table-sink-base.h
A testdata/workloads/functional-query/queries/QueryTest/iceberg-multiple-delete-per-partition.test
M tests/query_test/test_iceberg.py
5 files changed, 142 insertions(+), 29 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/23537/1
-- 
To view, visit http://gerrit.cloudera.org:8080/23537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
Gerrit-Change-Number: 23537
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23537 )

Change subject: IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation
......................................................................


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12563/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/23537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
Gerrit-Change-Number: 23537
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Tue, 14 Oct 2025 09:22:01 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has posted comments on this change. ( http://gerrit.cloudera.org:8080/23537 )

Change subject: IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation
......................................................................


Patch Set 4: Code-Review+1

LGTM, tests prove the issue is gone.


-- 
To view, visit http://gerrit.cloudera.org:8080/23537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
Gerrit-Change-Number: 23537
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Oct 2025 16:17:05 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23537 )

Change subject: IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/20679/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/23537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
Gerrit-Change-Number: 23537
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 14 Oct 2025 16:19:43 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23537 )

Change subject: IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation
......................................................................


Patch Set 2: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/23537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
Gerrit-Change-Number: 23537
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Tue, 14 Oct 2025 14:05:28 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/23537 )

Change subject: IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation
......................................................................


Patch Set 3:

(1 comment)

Thanks for the comment!

http://gerrit.cloudera.org:8080/#/c/23537/2/testdata/workloads/functional-query/queries/QueryTest/iceberg-multiple-delete-per-partition.test
File testdata/workloads/functional-query/queries/QueryTest/iceberg-multiple-delete-per-partition.test:

http://gerrit.cloudera.org:8080/#/c/23537/2/testdata/workloads/functional-query/queries/QueryTest/iceberg-multiple-delete-per-partition.test@3
PS2, Line 3: # Regression test for IMPALA-14496 where a DELETE operation needs to write
> typo: "Regression"
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/23537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
Gerrit-Change-Number: 23537
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 14 Oct 2025 15:51:05 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/23537 )

Change subject: IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation
......................................................................


Patch Set 5:

Thanks everyone for the review!


-- 
To view, visit http://gerrit.cloudera.org:8080/23537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
Gerrit-Change-Number: 23537
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 16 Oct 2025 10:30:38 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23537 )

Change subject: IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/20675/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/23537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
Gerrit-Change-Number: 23537
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Tue, 14 Oct 2025 09:36:52 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Mihaly Szjatinya, Noemi Pap-Takacs, Michael Smith, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/23537

to look at the new patch set (#2).

Change subject: IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation
......................................................................

IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Impala crashes when it needs to write multiple delete files per
partition in a single DELETE operation. It is because
IcebergBufferedDeleteSink has its own DmlExecState object, but
sometimes the methods in TableSinkBase use the RuntimeState's
DmlExecState object. I.e. it can happen that we add a partition
to the IcebergBufferedDeleteSink's DmlExecState, but later we
expect to find it in the RuntimeState's DmlExecState.

This patch adds new methods to TableSinkBase that are specific
for writing delete files, and they always take a DmlExecState
object as a parameter. They are now used by IcebergBufferedDeleteSink.

Testing
 * added e2e tests

Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
---
M be/src/exec/iceberg-buffered-delete-sink.cc
M be/src/exec/table-sink-base.cc
M be/src/exec/table-sink-base.h
A testdata/workloads/functional-query/queries/QueryTest/iceberg-multiple-delete-per-partition.test
M tests/query_test/test_iceberg.py
5 files changed, 142 insertions(+), 29 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/23537/2
-- 
To view, visit http://gerrit.cloudera.org:8080/23537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
Gerrit-Change-Number: 23537
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>

[Impala-ASF-CR] IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23537 )

Change subject: IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/20676/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/23537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
Gerrit-Change-Number: 23537
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Tue, 14 Oct 2025 09:49:01 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23537 )

Change subject: IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation
......................................................................


Patch Set 2: Code-Review+1

(1 comment)

http://gerrit.cloudera.org:8080/#/c/23537/2/testdata/workloads/functional-query/queries/QueryTest/iceberg-multiple-delete-per-partition.test
File testdata/workloads/functional-query/queries/QueryTest/iceberg-multiple-delete-per-partition.test:

http://gerrit.cloudera.org:8080/#/c/23537/2/testdata/workloads/functional-query/queries/QueryTest/iceberg-multiple-delete-per-partition.test@3
PS2, Line 3: # Regresion test for IMPALA-14496 where a DELETE operation needs to write
typo: "Regression"



-- 
To view, visit http://gerrit.cloudera.org:8080/23537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
Gerrit-Change-Number: 23537
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Tue, 14 Oct 2025 15:43:39 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23537 )

Change subject: IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation
......................................................................


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12568/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/23537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
Gerrit-Change-Number: 23537
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Oct 2025 15:01:06 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23537 )

Change subject: IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation
......................................................................


Patch Set 4: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/23537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
Gerrit-Change-Number: 23537
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Oct 2025 16:44:44 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23537 )

Change subject: IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation
......................................................................


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/23537/3/testdata/workloads/functional-query/queries/QueryTest/iceberg-multiple-delete-per-partition.test
File testdata/workloads/functional-query/queries/QueryTest/iceberg-multiple-delete-per-partition.test:

http://gerrit.cloudera.org:8080/#/c/23537/3/testdata/workloads/functional-query/queries/QueryTest/iceberg-multiple-delete-per-partition.test@32
PS3, Line 32: SET parquet_file_size=8m;
This is a helpful find. Test runs quickly, and triggers expected failures in release and debug builds. This patch then fixes them.



-- 
To view, visit http://gerrit.cloudera.org:8080/23537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
Gerrit-Change-Number: 23537
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 14 Oct 2025 16:47:51 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Mihaly Szjatinya, Noemi Pap-Takacs, Michael Smith, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/23537

to look at the new patch set (#4).

Change subject: IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation
......................................................................

IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Impala crashes when it needs to write multiple delete files per
partition in a single DELETE operation. It is because
IcebergBufferedDeleteSink has its own DmlExecState object, but
sometimes the methods in TableSinkBase use the RuntimeState's
DmlExecState object. I.e. it can happen that we add a partition
to the IcebergBufferedDeleteSink's DmlExecState, but later we
expect to find it in the RuntimeState's DmlExecState.

This patch adds new methods to TableSinkBase that are specific
for writing delete files, and they always take a DmlExecState
object as a parameter. They are now used by IcebergBufferedDeleteSink.

Testing
 * added e2e tests

Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
---
M be/src/exec/iceberg-buffered-delete-sink.cc
M be/src/exec/table-sink-base.cc
M be/src/exec/table-sink-base.h
A testdata/workloads/functional-query/queries/QueryTest/iceberg-multiple-delete-per-partition.test
M tests/query_test/test_iceberg.py
5 files changed, 135 insertions(+), 20 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/23537/4
-- 
To view, visit http://gerrit.cloudera.org:8080/23537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
Gerrit-Change-Number: 23537
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/23537 )

Change subject: IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation
......................................................................

IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Impala crashes when it needs to write multiple delete files per
partition in a single DELETE operation. It is because
IcebergBufferedDeleteSink has its own DmlExecState object, but
sometimes the methods in TableSinkBase use the RuntimeState's
DmlExecState object. I.e. it can happen that we add a partition
to the IcebergBufferedDeleteSink's DmlExecState, but later we
expect to find it in the RuntimeState's DmlExecState.

This patch adds new methods to TableSinkBase that are specific
for writing delete files, and they always take a DmlExecState
object as a parameter. They are now used by IcebergBufferedDeleteSink.

Testing
 * added e2e tests

Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
Reviewed-on: http://gerrit.cloudera.org:8080/23537
Reviewed-by: Mihaly Szjatinya <ms...@pm.me>
Reviewed-by: Michael Smith <mi...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/iceberg-buffered-delete-sink.cc
M be/src/exec/table-sink-base.cc
M be/src/exec/table-sink-base.h
A testdata/workloads/functional-query/queries/QueryTest/iceberg-multiple-delete-per-partition.test
M tests/query_test/test_iceberg.py
5 files changed, 135 insertions(+), 20 deletions(-)

Approvals:
  Mihaly Szjatinya: Looks good to me, but someone else must approve
  Michael Smith: Looks good to me, approved
  Impala Public Jenkins: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/23537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
Gerrit-Change-Number: 23537
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Mihaly Szjatinya, Noemi Pap-Takacs, Michael Smith, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/23537

to look at the new patch set (#3).

Change subject: IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation
......................................................................

IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Impala crashes when it needs to write multiple delete files per
partition in a single DELETE operation. It is because
IcebergBufferedDeleteSink has its own DmlExecState object, but
sometimes the methods in TableSinkBase use the RuntimeState's
DmlExecState object. I.e. it can happen that we add a partition
to the IcebergBufferedDeleteSink's DmlExecState, but later we
expect to find it in the RuntimeState's DmlExecState.

This patch adds new methods to TableSinkBase that are specific
for writing delete files, and they always take a DmlExecState
object as a parameter. They are now used by IcebergBufferedDeleteSink.

Testing
 * added e2e tests

Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
---
M be/src/exec/iceberg-buffered-delete-sink.cc
M be/src/exec/table-sink-base.cc
M be/src/exec/table-sink-base.h
A testdata/workloads/functional-query/queries/QueryTest/iceberg-multiple-delete-per-partition.test
M tests/query_test/test_iceberg.py
5 files changed, 142 insertions(+), 29 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/23537/3
-- 
To view, visit http://gerrit.cloudera.org:8080/23537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
Gerrit-Change-Number: 23537
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>

[Impala-ASF-CR] IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/23537 )

Change subject: IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation
......................................................................


Patch Set 4:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/23537/3/be/src/exec/table-sink-base.cc
File be/src/exec/table-sink-base.cc:

http://gerrit.cloudera.org:8080/#/c/23537/3/be/src/exec/table-sink-base.cc@443
PS3, Line 443:       state, partition, /*is_delete=*/false, state->dml_exec_state());
> nit: we could combine most of these functions into a helper that takes stat
Introduced FinalizePartitionFileImpl()


http://gerrit.cloudera.org:8080/#/c/23537/3/testdata/workloads/functional-query/queries/QueryTest/iceberg-multiple-delete-per-partition.test
File testdata/workloads/functional-query/queries/QueryTest/iceberg-multiple-delete-per-partition.test:

http://gerrit.cloudera.org:8080/#/c/23537/3/testdata/workloads/functional-query/queries/QueryTest/iceberg-multiple-delete-per-partition.test@32
PS3, Line 32: SET parquet_file_size=8m;
> This is a helpful find. Test runs quickly, and triggers expected failures i
Thanks for verifying.



-- 
To view, visit http://gerrit.cloudera.org:8080/23537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
Gerrit-Change-Number: 23537
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Oct 2025 09:45:58 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23537 )

Change subject: IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation
......................................................................


Patch Set 3: Verified+1 Code-Review+1

(1 comment)

Keeping Verified after minor comment change.

http://gerrit.cloudera.org:8080/#/c/23537/2/testdata/workloads/functional-query/queries/QueryTest/iceberg-multiple-delete-per-partition.test
File testdata/workloads/functional-query/queries/QueryTest/iceberg-multiple-delete-per-partition.test:

http://gerrit.cloudera.org:8080/#/c/23537/2/testdata/workloads/functional-query/queries/QueryTest/iceberg-multiple-delete-per-partition.test@3
PS2, Line 3: # Regression test for IMPALA-14496 where a DELETE operation needs to write
> Done
Ack



-- 
To view, visit http://gerrit.cloudera.org:8080/23537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
Gerrit-Change-Number: 23537
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 14 Oct 2025 15:57:44 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23537 )

Change subject: IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation
......................................................................


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/23537/3/be/src/exec/table-sink-base.cc
File be/src/exec/table-sink-base.cc:

http://gerrit.cloudera.org:8080/#/c/23537/3/be/src/exec/table-sink-base.cc@443
PS3, Line 443:   if (partition->tmp_hdfs_file == nullptr && !is_overwrite()) return Status::OK();
nit: we could combine most of these functions into a helper that takes state->dml_exec_state() and is_delete as arguments.



-- 
To view, visit http://gerrit.cloudera.org:8080/23537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
Gerrit-Change-Number: 23537
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 14 Oct 2025 16:52:26 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23537 )

Change subject: IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation
......................................................................


Patch Set 4: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/23537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
Gerrit-Change-Number: 23537
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Oct 2025 19:44:44 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23537 )

Change subject: IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/20690/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/23537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
Gerrit-Change-Number: 23537
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Oct 2025 10:09:30 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23537 )

Change subject: IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation
......................................................................


Patch Set 4: Code-Review+1

(1 comment)

http://gerrit.cloudera.org:8080/#/c/23537/4/be/src/exec/table-sink-base.h
File be/src/exec/table-sink-base.h:

http://gerrit.cloudera.org:8080/#/c/23537/4/be/src/exec/table-sink-base.h@227
PS4, Line 227:   Status FinalizePartitionFileImpl(RuntimeState* state, OutputPartition* partition,
Just observing you added a comment for WriteRowsToFile but not this one.



-- 
To view, visit http://gerrit.cloudera.org:8080/23537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
Gerrit-Change-Number: 23537
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Oct 2025 15:31:03 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23537 )

Change subject: IMPALA-14496: Impala crashes when it writes multiple delete files per partition in a single DELETE operation
......................................................................


Patch Set 4:

Waiting on existing dry run to finish.


-- 
To view, visit http://gerrit.cloudera.org:8080/23537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I46266007a6356e9ff3b63369dd855aff1396bb72
Gerrit-Change-Number: 23537
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Oct 2025 16:45:11 +0000
Gerrit-HasComments: No