You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@impala.apache.org by "Quanlong Huang (Code Review)" <ge...@cloudera.org> on 2021/12/03 03:10:42 UTC

[Impala-ASF-CR] IMPALA-10886: Fix lossing createEventId for INSERT created partitions

Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18066

Change subject: IMPALA-10886: Fix lossing createEventId for INSERT created partitions
......................................................................

IMPALA-10886: Fix lossing createEventId for INSERT created partitions

Coordinator calls updateCatalog RPC to catalogd to finalize the INSERT
statement. Catalogd will create any new partitions and reload metadata
of the updated partitions (including new partitions).

After IMPALA-10502, each new partition has a createEventId which is used
to detect how "fresh" it is. If the event processor receives a
DROP_PARTITION event, it will compare its event id with the
createEventId of the partition. Only DROP_PARTITION events happen after
the createEventId will be evaluated.

There is a bug in CatalogOpExecutor#updateCatalog() that we loss the
createEventId of the new partitions. It should be used in the final call
of loadTableMetadata(). This bug causes intermittent failures in
TestReusePartitionMetadata.test_reuse_partition_meta. The last two DMLs
of the test is dropping a partition and then creating it back by an
INSERT. If the DROP_PARTITION event is processed after the INSERT
finishes, the partition will be dropped incorrectly.

Tests
- Ran the test 100 times locally. Without the fix, it fails in 10 runs.

Change-Id: I2622c28a5ce6084fc77f6ea475d2633445c7f8dd
---
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_local_catalog.py
2 files changed, 10 insertions(+), 9 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/18066/1
--
To view, visit http://gerrit.cloudera.org:8080/18066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I2622c28a5ce6084fc77f6ea475d2633445c7f8dd
Gerrit-Change-Number: 18066
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-10886: Fix lossing createEventId for INSERT created partitions

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18066 )

Change subject: IMPALA-10886: Fix lossing createEventId for INSERT created partitions
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9867/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2622c28a5ce6084fc77f6ea475d2633445c7f8dd
Gerrit-Change-Number: 18066
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Dec 2021 03:33:53 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10886: Fix loss of createEventId for INSERT created partitions

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has uploaded a new patch set (#2) to the change originally created by Quanlong Huang. ( http://gerrit.cloudera.org:8080/18066 )

Change subject: IMPALA-10886: Fix loss of createEventId for INSERT created partitions
......................................................................

IMPALA-10886: Fix loss of createEventId for INSERT created partitions

Coordinator calls updateCatalog RPC to catalogd to finalize the INSERT
statement. Catalogd will create any new partitions and reload metadata
of the updated partitions (including new partitions).

Tests
- Ran the test 100 times locally. Without the fix, it fails in 10 runs.

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/18066/2
--
To view, visit http://gerrit.cloudera.org:8080/18066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2622c28a5ce6084fc77f6ea475d2633445c7f8dd
Gerrit-Change-Number: 18066
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10886: Fix loss of createEventId for INSERT created partitions

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18066 )

Change subject: IMPALA-10886: Fix loss of createEventId for INSERT created partitions
......................................................................

IMPALA-10886: Fix loss of createEventId for INSERT created partitions

Coordinator calls updateCatalog RPC to catalogd to finalize the INSERT
statement. Catalogd will create any new partitions and reload metadata
of the updated partitions (including new partitions).

Tests
- Ran the test 100 times locally. Without the fix, it fails in 10 runs.

Change-Id: I2622c28a5ce6084fc77f6ea475d2633445c7f8dd
Reviewed-on: http://gerrit.cloudera.org:8080/18066
Reviewed-by: Vihang Karajgaonkar <vi...@cloudera.com>
Tested-by: Vihang Karajgaonkar <vi...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_local_catalog.py
2 files changed, 10 insertions(+), 9 deletions(-)

Approvals:
Vihang Karajgaonkar: Looks good to me, approved; Verified

--
To view, visit http://gerrit.cloudera.org:8080/18066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I2622c28a5ce6084fc77f6ea475d2633445c7f8dd
Gerrit-Change-Number: 18066
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10886: Fix loss of createEventId for INSERT created partitions

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/18066 )

Change subject: IMPALA-10886: Fix loss of createEventId for INSERT created partitions
......................................................................


Patch Set 2: Code-Review+2

(1 comment)

Carrying +2 from earlier and merging.

http://gerrit.cloudera.org:8080/#/c/18066/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/18066/1//COMMIT_MSG@7
PS1, Line 7: loss of
> nit, may be better wording could be "loss of"
it was a trivial commit message change. I can do it myself since it may be a while when Quanlong comes online. I would like to merge this patch as soon as the gerrit job comes back green.



-- 
To view, visit http://gerrit.cloudera.org:8080/18066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2622c28a5ce6084fc77f6ea475d2633445c7f8dd
Gerrit-Change-Number: 18066
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Sat, 04 Dec 2021 04:15:30 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10886: Fix loss of createEventId for INSERT created partitions

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/18066 )

Change subject: IMPALA-10886: Fix loss of createEventId for INSERT created partitions
......................................................................


Patch Set 2: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/18066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2622c28a5ce6084fc77f6ea475d2633445c7f8dd
Gerrit-Change-Number: 18066
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Sat, 04 Dec 2021 04:15:34 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10886: Fix lossing createEventId for INSERT created partitions

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18066 )

Change subject: IMPALA-10886: Fix lossing createEventId for INSERT created partitions
......................................................................


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7691/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/18066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2622c28a5ce6084fc77f6ea475d2633445c7f8dd
Gerrit-Change-Number: 18066
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Dec 2021 21:25:44 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10886: Fix lossing createEventId for INSERT created partitions

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/18066 )

Change subject: IMPALA-10886: Fix lossing createEventId for INSERT created partitions
......................................................................


Patch Set 1: Code-Review+2

(1 comment)

Thanks for identifying and fixing this. The patch looks good to me. Feel free to merge it after addressing the nit.

http://gerrit.cloudera.org:8080/#/c/18066/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/18066/1//COMMIT_MSG@7
PS1, Line 7: lossing
nit, may be better wording could be "loss of"



-- 
To view, visit http://gerrit.cloudera.org:8080/18066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2622c28a5ce6084fc77f6ea475d2633445c7f8dd
Gerrit-Change-Number: 18066
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Dec 2021 20:25:55 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10886: Fix loss of createEventId for INSERT created partitions

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18066 )

Change subject: IMPALA-10886: Fix loss of createEventId for INSERT created partitions
......................................................................


Patch Set 1: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/18066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2622c28a5ce6084fc77f6ea475d2633445c7f8dd
Gerrit-Change-Number: 18066
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Sat, 04 Dec 2021 03:47:50 +0000
Gerrit-HasComments: No