You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@impala.apache.org by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org> on 2021/11/19 02:54:40 UTC

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Vihang Karajgaonkar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18038

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................

IMPALA-11028: Table loading can fail when events are cleaned up

IMPALA-10502 introduces a createEventId field of a table which
is updated when Impala creates a table. This is used by
the events processor to determine if the subsequent CREATE_TABLE
event which is received should be skipped or not.

When the table is loaded for the first time, in order to avoid
race conditions, TableLoader updates the createEventId to the
last CREATE_TABLE event id from the metastore. In order to
fetch the latest CREATE_TABLE event id, it fetches all the
events from metastore since the last known createEventId of the
table. However, if there is a significant delay between
(more than 24hrs) between the time a table is created
or invalidated, and the table is queried, it is possible that
the metastore cleanup thread deletes the events which are generated
since the table's createEventId. In such a case, the HMS Client method
getNextNotification() throws an IllegalStateException due to the missing
events. This exception causes the Table load to fail and query to error
out.

The fix is to not rely on the HMS Client method which throws the
IllegalStateException. Instead we use the backing thrift API directly.

Testing:
1. Introduced a custom cluster test which can reproduce this issue.
2. Test works after the patch.
3. Core tests.

Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
---
M bin/create-test-configuration.sh
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/resources/hive-site.xml.py
4 files changed, 32 insertions(+), 31 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/18038/5
--
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 5
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................


Patch Set 14:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9834/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 14
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Nov 2021 01:50:48 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................


Patch Set 7:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9824/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 7
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Nov 2021 05:33:08 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................


Patch Set 11:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9829/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 11
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Nov 2021 17:47:11 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9808/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 5
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Nov 2021 03:17:34 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................

IMPALA-11028: Table loading can fail when events are cleaned up

The fix is to not rely on the HMS Client method which throws the
IllegalStateException. Instead we use the backing thrift API directly.

Testing:
1. Introduced a custom cluster test which can reproduce this issue.
2. Test works after the patch.
3. Core tests.

Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
---
M bin/create-test-configuration.sh
M fe/src/main/java/org/apache/impala/catalog/TableLoader.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M fe/src/test/resources/hive-site.xml.py
A tests/custom_cluster/test_metastore_events_cleanup.py
8 files changed, 107 insertions(+), 41 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/18038/11
--
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 11
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................


Patch Set 7:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7654/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 7
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Nov 2021 05:11:49 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................


Patch Set 7:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/18038/7/fe/src/main/java/org/apache/impala/catalog/TableLoader.java
File fe/src/main/java/org/apache/impala/catalog/TableLoader.java:

http://gerrit.cloudera.org:8080/#/c/18038/7/fe/src/main/java/org/apache/impala/catalog/TableLoader.java@86
PS7, Line 86:         events = MetastoreEventsProcessor.getNextMetastoreEventsInBatches(catalog_, eventId,
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/18038/7/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:

http://gerrit.cloudera.org:8080/#/c/18038/7/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@305
PS7, Line 305:     List<NotificationEvent> events = MetastoreEventsProcessor.getNextMetastoreEventsInBatches(
line too long (94 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 7
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Nov 2021 05:12:09 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................

IMPALA-11028: Table loading can fail when events are cleaned up

The fix is to not rely on the HMS Client method which throws the
IllegalStateException. Instead we use the backing thrift API directly.

Testing:
1. Introduced a custom cluster test which can reproduce this issue.
2. Test works after the patch.
3. Core tests.

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/18038/9
--
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 9
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................

IMPALA-11028: Table loading can fail when events are cleaned up

The fix is to not rely on the HMS Client method which throws the
IllegalStateException. Instead we use the backing thrift API directly.

Testing:
1. Introduced a custom cluster test which can reproduce this issue.
2. Test works after the patch.
3. Core tests.

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/18038/13
--
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 13
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7649/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 5
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Nov 2021 02:55:05 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.

Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................


Patch Set 5: Code-Review+2

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18038/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/18038/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@583
PS5, Line 583:    * Gets metastore notification events from the given eventId. The returned list of
nit: since this  method is directly referencing the event processor's api, do the comments for this method above need to be updated  ? This is now more like a wrapper function.



-- 
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 5
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Comment-Date: Sun, 21 Nov 2021 04:44:01 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................


Patch Set 15: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 15
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Nov 2021 07:45:46 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................

IMPALA-11028: Table loading can fail when events are cleaned up

The fix is to not rely on the HMS Client method which throws the
IllegalStateException. Instead we use the backing thrift API directly.

Testing:
1. Introduced a custom cluster test which can reproduce this issue.
2. Test works after the patch.
3. Core tests.

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/18038/14
--
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 14
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................


Patch Set 7: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7654/


-- 
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 7
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Nov 2021 11:30:19 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................


Patch Set 15: Code-Review+2

(4 comments)

Carrying forward Aman's +2 from earlier. There were mostly formatting changes and a new test file added since that last +2.

http://gerrit.cloudera.org:8080/#/c/18038/8/fe/src/main/java/org/apache/impala/catalog/TableLoader.java
File fe/src/main/java/org/apache/impala/catalog/TableLoader.java:

http://gerrit.cloudera.org:8080/#/c/18038/8/fe/src/main/java/org/apache/impala/catalog/TableLoader.java@86
PS8, Line 86:         events = MetastoreEventsProcessor.getNextMetastoreEventsInBatches(catalog_,
> line too long (92 > 90)
Done


http://gerrit.cloudera.org:8080/#/c/18038/8/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:

http://gerrit.cloudera.org:8080/#/c/18038/8/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@305
PS8, Line 305:   @Test
> line too long (94 > 90)
Done


http://gerrit.cloudera.org:8080/#/c/18038/11/tests/custom_cluster/test_metastore_events_cleanup.py
File tests/custom_cluster/test_metastore_events_cleanup.py:

http://gerrit.cloudera.org:8080/#/c/18038/11/tests/custom_cluster/test_metastore_events_cleanup.py@24
PS11, Line 24: 
> flake8: E501 line too long (95 > 90 characters)
Done


http://gerrit.cloudera.org:8080/#/c/18038/11/tests/custom_cluster/test_metastore_events_cleanup.py@26
PS11, Line 26: 
> flake8: E302 expected 2 blank lines, found 1
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 15
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Nov 2021 01:34:22 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................


Patch Set 12:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7660/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 12
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Nov 2021 17:26:51 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................


Patch Set 15:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7664/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 15
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Nov 2021 01:29:16 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................

IMPALA-11028: Table loading can fail when events are cleaned up

The fix is to not rely on the HMS Client method which throws the
IllegalStateException. Instead we use the backing thrift API directly.

Testing:
1. Introduced a custom cluster test which can reproduce this issue.
2. Test works after the patch.
3. Core tests.

Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Reviewed-on: http://gerrit.cloudera.org:8080/18038
Reviewed-by: Vihang Karajgaonkar <vi...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M bin/create-test-configuration.sh
M fe/src/main/java/org/apache/impala/catalog/TableLoader.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M fe/src/test/resources/hive-site.xml.py
A tests/custom_cluster/test_metastore_events_cleanup.py
8 files changed, 111 insertions(+), 39 deletions(-)

Approvals:
Vihang Karajgaonkar: Looks good to me, approved
Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 16
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................


Patch Set 13:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/18038/13/tests/custom_cluster/test_metastore_events_cleanup.py
File tests/custom_cluster/test_metastore_events_cleanup.py:

http://gerrit.cloudera.org:8080/#/c/18038/13/tests/custom_cluster/test_metastore_events_cleanup.py@24
PS13, Line 24: a
flake8: E501 line too long (95 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/18038/13/tests/custom_cluster/test_metastore_events_cleanup.py@26
PS13, Line 26: class TestTableLoadingWithEventsCleanUp(CustomClusterTestSuite):
flake8: E302 expected 2 blank lines, found 1



-- 
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 13
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Nov 2021 01:17:56 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................


Patch Set 8:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/18038/8/fe/src/main/java/org/apache/impala/catalog/TableLoader.java
File fe/src/main/java/org/apache/impala/catalog/TableLoader.java:

http://gerrit.cloudera.org:8080/#/c/18038/8/fe/src/main/java/org/apache/impala/catalog/TableLoader.java@86
PS8, Line 86:         events = MetastoreEventsProcessor.getNextMetastoreEventsInBatches(catalog_, eventId,
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/18038/8/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:

http://gerrit.cloudera.org:8080/#/c/18038/8/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@305
PS8, Line 305:     List<NotificationEvent> events = MetastoreEventsProcessor.getNextMetastoreEventsInBatches(
line too long (94 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 8
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Nov 2021 16:35:47 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................


Patch Set 8:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9827/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 8
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Nov 2021 16:58:08 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................

IMPALA-11028: Table loading can fail when events are cleaned up

The fix is to not rely on the HMS Client method which throws the
IllegalStateException. Instead we use the backing thrift API directly.

Testing:
1. Introduced a custom cluster test which can reproduce this issue.
2. Test works after the patch.
3. Core tests.

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/18038/7
--
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 7
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.

Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................


Patch Set 7: Code-Review+2

(2 comments)

A minor comment about logging. Feel free to carry forward the +2 once you address it.

http://gerrit.cloudera.org:8080/#/c/18038/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/18038/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@586
PS7, Line 586: getAllEvents
Yes, this name is more intuitive and accurate.


http://gerrit.cloudera.org:8080/#/c/18038/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@621
PS7, Line 621:       LOG.info(String.format("Received %d events. Start event id : %d",
A minor change in logging behavior: previously the log message was showing the filtered events. This one shows the unfiltered events.  It might be useful to also log the filtered ones.



-- 
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 7
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Nov 2021 06:57:58 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................

IMPALA-11028: Table loading can fail when events are cleaned up

The fix is to not rely on the HMS Client method which throws the
IllegalStateException. Instead we use the backing thrift API directly.

Testing:
1. Introduced a custom cluster test which can reproduce this issue.
2. Test works after the patch.
3. Core tests.

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/18038/8
--
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 8
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................


Patch Set 11:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/18038/11/tests/custom_cluster/test_metastore_events_cleanup.py
File tests/custom_cluster/test_metastore_events_cleanup.py:

http://gerrit.cloudera.org:8080/#/c/18038/11/tests/custom_cluster/test_metastore_events_cleanup.py@24
PS11, Line 24: a
flake8: E501 line too long (95 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/18038/11/tests/custom_cluster/test_metastore_events_cleanup.py@26
PS11, Line 26: class TestTableLoadingWithEventsCleanUp(CustomClusterTestSuite):
flake8: E302 expected 2 blank lines, found 1



-- 
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 11
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Nov 2021 17:25:26 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................


Patch Set 7:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18038/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/18038/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@586
PS7, Line 586:    * @param getAllEvents If this is true all the events since eventId are returned.
@Vihang: Looks like setting getAllEvents to true is *not* sufficient to get all events since eventId. It is because HMSHandler has a property metastore.max.event.response which decides the max number of events returned by HMS in a single api call. Though the default value of the property is 1M which is fairly large but the above semantics would break for a customer who has set this property value to few thousands.



-- 
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 7
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Nov 2021 11:30:59 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................


Patch Set 10:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7659/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 10
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Nov 2021 16:39:16 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................


Patch Set 5: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 5
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Nov 2021 09:11:00 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................


Patch Set 7:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18038/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/18038/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@583
PS5, Line 583:    * NotificationEvents are filtered using the NotificationFilter provided if it is not
> nit: since this  method is directly referencing the event processor's api, 
Actually, I realized that my patch changed the behavior of this method. This method actually uses skipBatching flag to either get all the events since eventId or just the next 1000 events since eventId depending on the value of the skipBatching flag. I have renamed the method and the variable name to make it more clear.



-- 
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 7
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Nov 2021 05:11:24 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................


Patch Set 9:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9828/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 9
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Nov 2021 17:01:14 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................


Patch Set 12: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7660/


-- 
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 12
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Nov 2021 23:45:33 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11028: Table loading can fail when events are cleaned up

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18038 )

Change subject: IMPALA-11028: Table loading can fail when events are cleaned up
......................................................................


Patch Set 13:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9832/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18038
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Gerrit-Change-Number: 18038
Gerrit-PatchSet: 13
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Nov 2021 01:39:58 +0000
Gerrit-HasComments: No