You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org> on 2024/02/05 13:10:14 UTC

[Impala-ASF-CR] IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic

Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20999


Change subject: IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic
......................................................................

IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic

If an UPDATE operation runs concurrently with a DELETE operation, and
the DELETE commits first, then the UPDATE can revive deleted rows. This
is because only RowDelta.validateNoConflictingDataFiles() is called, but
RowDelta.validateNoConflictingDeleteFiles() is not. Therefore, the
UPDATE operation ignores the concurrently written delete files.

This patch adds RowDelta.validateNoConflictingDeleteFiles() to UPDATE
operations.

Testing
 * added a stress test to validate concurrent DELETE and UPDATE
    operations

Change-Id: I9e581ea17fa8f6ccd9c87aaad1281bb694079f6e
---
M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
M tests/stress/test_update_stress.py
2 files changed, 119 insertions(+), 7 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/20999/1
-- 
To view, visit http://gerrit.cloudera.org:8080/20999
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I9e581ea17fa8f6ccd9c87aaad1281bb694079f6e
Gerrit-Change-Number: 20999
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic

Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/20999 )

Change subject: IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic
......................................................................


Patch Set 1: Code-Review+1

(1 comment)

Thanks for this fix Zoltan! LGTM!

http://gerrit.cloudera.org:8080/#/c/20999/1/fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/20999/1/fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java@372
PS1, Line 372:       //added in the meantime, they potentially contain records that should have been
nit: space



-- 
To view, visit http://gerrit.cloudera.org:8080/20999
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e581ea17fa8f6ccd9c87aaad1281bb694079f6e
Gerrit-Change-Number: 20999
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Mon, 05 Feb 2024 13:21:32 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/20999 )

Change subject: IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic
......................................................................


Patch Set 2: Code-Review+1

(1 comment)

Thanks for the comments! Carry +1

http://gerrit.cloudera.org:8080/#/c/20999/1/fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/20999/1/fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java@372
PS1, Line 372:       // added in the meantime, they potentially contain records that should have been
> nit: space
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/20999
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e581ea17fa8f6ccd9c87aaad1281bb694079f6e
Gerrit-Change-Number: 20999
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 05 Feb 2024 14:13:12 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/20999 )

Change subject: IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic
......................................................................


Patch Set 2:

(1 comment)

Thanks for the comment!

http://gerrit.cloudera.org:8080/#/c/20999/2/fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/20999/2/fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java@375
PS2, Line 375:       rowDelta.validateNoConflictingDataFiles();
> This is a complete nit but is it more iceberg-style to chain the calls? Her
Personally I like method chaining for Builder classes, in other cases I find it a bit strange.
In general, I think what should matter is the Impala coding style, not the coding style of the library we are using. Impala's Java coding style doesn't say too much about anything though.
I can switch to chaining if you like, I don't really have a strong opinion about this.



-- 
To view, visit http://gerrit.cloudera.org:8080/20999
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e581ea17fa8f6ccd9c87aaad1281bb694079f6e
Gerrit-Change-Number: 20999
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 06 Feb 2024 13:04:56 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20999 )

Change subject: IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10248/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/20999
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e581ea17fa8f6ccd9c87aaad1281bb694079f6e
Gerrit-Change-Number: 20999
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 06 Feb 2024 17:34:13 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20999 )

Change subject: IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic
......................................................................


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10238/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20999
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e581ea17fa8f6ccd9c87aaad1281bb694079f6e
Gerrit-Change-Number: 20999
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 05 Feb 2024 14:13:34 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20999 )

Change subject: IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic
......................................................................


Patch Set 3: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/20999
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e581ea17fa8f6ccd9c87aaad1281bb694079f6e
Gerrit-Change-Number: 20999
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 06 Feb 2024 22:07:05 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20999 )

Change subject: IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic
......................................................................


Patch Set 2: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10238/


-- 
To view, visit http://gerrit.cloudera.org:8080/20999
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e581ea17fa8f6ccd9c87aaad1281bb694079f6e
Gerrit-Change-Number: 20999
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 05 Feb 2024 18:43:15 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20999 )

Change subject: IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic
......................................................................


Patch Set 3: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/20999
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e581ea17fa8f6ccd9c87aaad1281bb694079f6e
Gerrit-Change-Number: 20999
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 06 Feb 2024 17:34:12 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic

Posted by "Gabor Kaszab (Code Review)" <ge...@cloudera.org>.
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/20999 )

Change subject: IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic
......................................................................


Patch Set 2: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/20999
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e581ea17fa8f6ccd9c87aaad1281bb694079f6e
Gerrit-Change-Number: 20999
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 06 Feb 2024 13:11:08 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic

Posted by "Andrew Sherman (Code Review)" <ge...@cloudera.org>.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/20999 )

Change subject: IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/20999/2/fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/20999/2/fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java@375
PS2, Line 375:       rowDelta.validateNoConflictingDataFiles();
> Personally I like method chaining for Builder classes, in other cases I fin
I don't like chaining so I am happy with keeping it the way it is



-- 
To view, visit http://gerrit.cloudera.org:8080/20999
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e581ea17fa8f6ccd9c87aaad1281bb694079f6e
Gerrit-Change-Number: 20999
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 06 Feb 2024 16:33:01 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20999 )

Change subject: IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/15169/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20999
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e581ea17fa8f6ccd9c87aaad1281bb694079f6e
Gerrit-Change-Number: 20999
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 05 Feb 2024 14:39:09 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20999 )

Change subject: IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic
......................................................................


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10245/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20999
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e581ea17fa8f6ccd9c87aaad1281bb694079f6e
Gerrit-Change-Number: 20999
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 06 Feb 2024 12:50:01 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20999 )

Change subject: IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic
......................................................................


Patch Set 2:

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10245/


-- 
To view, visit http://gerrit.cloudera.org:8080/20999
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e581ea17fa8f6ccd9c87aaad1281bb694079f6e
Gerrit-Change-Number: 20999
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 06 Feb 2024 17:28:12 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic

Posted by "Andrew Sherman (Code Review)" <ge...@cloudera.org>.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/20999 )

Change subject: IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic
......................................................................


Patch Set 2: Code-Review+1

(1 comment)

http://gerrit.cloudera.org:8080/#/c/20999/2/fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/20999/2/fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java@375
PS2, Line 375:       rowDelta.validateNoConflictingDataFiles();
This is a complete nit but is it more iceberg-style to chain the calls? Here it does not matter for correctness (I think) but is that a coding style we should pursue?



-- 
To view, visit http://gerrit.cloudera.org:8080/20999
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e581ea17fa8f6ccd9c87aaad1281bb694079f6e
Gerrit-Change-Number: 20999
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 05 Feb 2024 18:15:29 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20999 )

Change subject: IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/15168/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20999
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e581ea17fa8f6ccd9c87aaad1281bb694079f6e
Gerrit-Change-Number: 20999
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Mon, 05 Feb 2024 13:36:11 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Tamas Mate, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20999

to look at the new patch set (#2).

Change subject: IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic
......................................................................

IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic

If an UPDATE operation runs concurrently with a DELETE operation, and
the DELETE commits first, then the UPDATE can revive deleted rows. This
is because only RowDelta.validateNoConflictingDataFiles() is called, but
RowDelta.validateNoConflictingDeleteFiles() is not. Therefore, the
UPDATE operation ignores the concurrently written delete files.

This patch adds RowDelta.validateNoConflictingDeleteFiles() to UPDATE
operations.

Testing
 * added a stress test to validate concurrent DELETE and UPDATE
    operations

Change-Id: I9e581ea17fa8f6ccd9c87aaad1281bb694079f6e
---
M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
M tests/stress/test_update_stress.py
2 files changed, 120 insertions(+), 7 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/20999/2
-- 
To view, visit http://gerrit.cloudera.org:8080/20999
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9e581ea17fa8f6ccd9c87aaad1281bb694079f6e
Gerrit-Change-Number: 20999
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>

[Impala-ASF-CR] IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20999 )

Change subject: IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic
......................................................................

IMPALA-12787: Concurrent DELETE and UPDATE operations on Iceberg tables can be problematic

If an UPDATE operation runs concurrently with a DELETE operation, and
the DELETE commits first, then the UPDATE can revive deleted rows. This
is because only RowDelta.validateNoConflictingDataFiles() is called, but
RowDelta.validateNoConflictingDeleteFiles() is not. Therefore, the
UPDATE operation ignores the concurrently written delete files.

This patch adds RowDelta.validateNoConflictingDeleteFiles() to UPDATE
operations.

Testing
 * added a stress test to validate concurrent DELETE and UPDATE
    operations

Change-Id: I9e581ea17fa8f6ccd9c87aaad1281bb694079f6e
Reviewed-on: http://gerrit.cloudera.org:8080/20999
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
M tests/stress/test_update_stress.py
2 files changed, 120 insertions(+), 7 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/20999
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I9e581ea17fa8f6ccd9c87aaad1281bb694079f6e
Gerrit-Change-Number: 20999
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>