You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org> on 2025/02/19 12:33:46 UTC

[Impala-ASF-CR] WIP: IMPALA-13759: Fix Hive ACID INSERT OVERWRITE base detection

Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/22499


Change subject: WIP: IMPALA-13759: Fix Hive ACID INSERT OVERWRITE base detection
......................................................................

WIP: IMPALA-13759: Fix Hive ACID INSERT OVERWRITE base detection

Change-Id: I838eaf4f41bae148e558f64288a1370c0908efa4
---
M fe/src/main/java/org/apache/impala/util/AcidUtils.java
M tests/query_test/test_acid.py
2 files changed, 33 insertions(+), 2 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/22499/1
-- 
To view, visit http://gerrit.cloudera.org:8080/22499
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I838eaf4f41bae148e558f64288a1370c0908efa4
Gerrit-Change-Number: 22499
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>

[Impala-ASF-CR] WIP: IMPALA-13759: Fix Hive ACID INSERT OVERWRITE base detection

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22499 )

Change subject: WIP: IMPALA-13759: Fix Hive ACID INSERT OVERWRITE base detection
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/18303/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/22499
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I838eaf4f41bae148e558f64288a1370c0908efa4
Gerrit-Change-Number: 22499
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 19 Feb 2025 13:05:47 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13759: Fix Hive ACID INSERT OVERWRITE base detection

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22499 )

Change subject: IMPALA-13759: Fix Hive ACID INSERT OVERWRITE base detection
......................................................................


Patch Set 3: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/22499
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I838eaf4f41bae148e558f64288a1370c0908efa4
Gerrit-Change-Number: 22499
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 19 Feb 2025 16:59:16 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: IMPALA-13759: Fix Hive ACID INSERT OVERWRITE base detection

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22499 )

Change subject: WIP: IMPALA-13759: Fix Hive ACID INSERT OVERWRITE base detection
......................................................................


Patch Set 1:

(5 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/22499/1/tests/query_test/test_acid.py
File tests/query_test/test_acid.py:

http://gerrit.cloudera.org:8080/#/c/22499/1/tests/query_test/test_acid.py@371
PS1, Line 371: d
flake8: E303 too many blank lines (2)


http://gerrit.cloudera.org:8080/#/c/22499/1/tests/query_test/test_acid.py@375
PS1, Line 375:  
flake8: E222 multiple spaces after operator


http://gerrit.cloudera.org:8080/#/c/22499/1/tests/query_test/test_acid.py@379
PS1, Line 379: ,
flake8: E231 missing whitespace after ','


http://gerrit.cloudera.org:8080/#/c/22499/1/tests/query_test/test_acid.py@384
PS1, Line 384:  
flake8: E222 multiple spaces after operator


http://gerrit.cloudera.org:8080/#/c/22499/1/tests/query_test/test_acid.py@388
PS1, Line 388:  
flake8: E222 multiple spaces after operator



-- 
To view, visit http://gerrit.cloudera.org:8080/22499
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I838eaf4f41bae148e558f64288a1370c0908efa4
Gerrit-Change-Number: 22499
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 19 Feb 2025 12:35:45 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13759: Fix Hive ACID INSERT OVERWRITE base detection

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22499 )

Change subject: IMPALA-13759: Fix Hive ACID INSERT OVERWRITE base detection
......................................................................


Patch Set 3: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/22499
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I838eaf4f41bae148e558f64288a1370c0908efa4
Gerrit-Change-Number: 22499
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 19 Feb 2025 22:38:54 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13759: Fix Hive ACID INSERT OVERWRITE base detection

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/22499

to look at the new patch set (#2).

Change subject: IMPALA-13759: Fix Hive ACID INSERT OVERWRITE base detection
......................................................................

IMPALA-13759: Fix Hive ACID INSERT OVERWRITE base detection

Base directory created by INSERT OVERWRITE / TRUNCATE should be
treated differently than bases created by compaction because
IOW/TRUNCATE bases must be accepted even if there is an earlier
open writeId. This scenario can easily occur if there is
a pending write to a single partition, as this doesn't block
an IOW/TRUNCATE to another partition, while the global
minOpenWrite affects whether the base is accepted.

This change updates Impala logic to consider these bases
valid similarly to Hive.

Note that differentiating IOW/TRUNCATE from compaction is
different than in Hive, as metadata files are not considered
in Impala (IMPALA-13769). This can only cause problems when
interacting with earlier Hive versions that did not use
visibilityTxnId in the base path. I don't consider this
to be a significant regression that should block the current
critical fix.

Testing:
- added regression EE/FE tests

Change-Id: I838eaf4f41bae148e558f64288a1370c0908efa4
---
M fe/src/main/java/org/apache/impala/util/AcidUtils.java
M fe/src/test/java/org/apache/impala/util/AcidUtilsTest.java
M tests/query_test/test_acid.py
3 files changed, 61 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/22499/2
-- 
To view, visit http://gerrit.cloudera.org:8080/22499
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I838eaf4f41bae148e558f64288a1370c0908efa4
Gerrit-Change-Number: 22499
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-13759: Fix Hive ACID INSERT OVERWRITE base detection

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/22499 )

Change subject: IMPALA-13759: Fix Hive ACID INSERT OVERWRITE base detection
......................................................................


Patch Set 2: Code-Review+2

Thanks for fixing this, LGTM!


-- 
To view, visit http://gerrit.cloudera.org:8080/22499
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I838eaf4f41bae148e558f64288a1370c0908efa4
Gerrit-Change-Number: 22499
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 19 Feb 2025 16:45:16 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13759: Fix Hive ACID INSERT OVERWRITE base detection

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/22499 )

Change subject: IMPALA-13759: Fix Hive ACID INSERT OVERWRITE base detection
......................................................................

IMPALA-13759: Fix Hive ACID INSERT OVERWRITE base detection

Base directory created by INSERT OVERWRITE / TRUNCATE should be
treated differently than bases created by compaction because
IOW/TRUNCATE bases must be accepted even if there is an earlier
open writeId. This scenario can easily occur if there is
a pending write to a single partition, as this doesn't block
an IOW/TRUNCATE to another partition, while the global
minOpenWrite affects whether the base is accepted.

This change updates Impala logic to consider these bases
valid similarly to Hive.

Note that differentiating IOW/TRUNCATE from compaction is
different than in Hive, as metadata files are not considered
in Impala (IMPALA-13769). This can only cause problems when
interacting with earlier Hive versions that did not use
visibilityTxnId in the base path. I don't consider this
to be a significant regression that should block the current
critical fix.

Testing:
- added regression EE/FE tests

Change-Id: I838eaf4f41bae148e558f64288a1370c0908efa4
Reviewed-on: http://gerrit.cloudera.org:8080/22499
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/util/AcidUtils.java
M fe/src/test/java/org/apache/impala/util/AcidUtilsTest.java
M tests/query_test/test_acid.py
3 files changed, 61 insertions(+), 2 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/22499
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I838eaf4f41bae148e558f64288a1370c0908efa4
Gerrit-Change-Number: 22499
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-13759: Fix Hive ACID INSERT OVERWRITE base detection

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22499 )

Change subject: IMPALA-13759: Fix Hive ACID INSERT OVERWRITE base detection
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11464/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/22499
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I838eaf4f41bae148e558f64288a1370c0908efa4
Gerrit-Change-Number: 22499
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 19 Feb 2025 16:59:17 +0000
Gerrit-HasComments: No