You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Anonymous Coward (Code Review)" <ge...@cloudera.org> on 2025/12/16 20:27:12 UTC

[Impala-ASF-CR] IMPALA-14628: Fix mismatch between greatest synced event time and last synced event time

k.venureddy2103@gmail.com has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23797


Change subject: IMPALA-14628: Fix mismatch between greatest synced event time and last synced event time
......................................................................

IMPALA-14628: Fix mismatch between greatest synced event time and last synced event time

The greatest synced event id and its corresponding event time
must match the last synced event id and time once all outstanding
events have been fully processed.

Previously, for batch events, timestamp of the first event in
the batch is recorded when transitioning the batch event from
the in-progress log to the processed log in EventExecutorService.
As a result, the greatest synced event time incorrectly reflected
the first event’s timestamp, even though the greatest synced
event id corresponded to the last event in the batch, leading to
a mismatch

This fix ensures that, for batch events, the timestamp of the
last event in the batch is recorded when the batch is marked as
processed. Consequently, the greatest synced event time now
correctly aligns with the last synced event id after all
outstanding events are processed.

Testing:
    - Added an FE test and ran the existing tests.

Change-Id: I26550441bc0d5e7736c3b900c3e5178103157446
---
M fe/src/main/java/org/apache/impala/catalog/events/EventExecutorService.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
2 files changed, 29 insertions(+), 3 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/23797/1
-- 
To view, visit http://gerrit.cloudera.org:8080/23797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I26550441bc0d5e7736c3b900c3e5178103157446
Gerrit-Change-Number: 23797
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward <k....@gmail.com>

[Impala-ASF-CR] IMPALA-14628: Fix mismatch between greatest synced event time and last synced event time

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/23797 )

Change subject: IMPALA-14628: Fix mismatch between greatest synced event time and last synced event time
......................................................................


Patch Set 1: Code-Review+1

(2 comments)

http://gerrit.cloudera.org:8080/#/c/23797/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/23797/1//COMMIT_MSG@19
PS1, Line 19: a mismatch
nit: add a period.


http://gerrit.cloudera.org:8080/#/c/23797/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:

http://gerrit.cloudera.org:8080/#/c/23797/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@4381
PS1, Line 4381:     Thread.sleep(1000);
Could you add a comment on why we need this sleep?



-- 
To view, visit http://gerrit.cloudera.org:8080/23797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I26550441bc0d5e7736c3b900c3e5178103157446
Gerrit-Change-Number: 23797
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 18 Dec 2025 01:58:48 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14628: Fix mismatch between greatest synced event time and last synced event time

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/23797 )

Change subject: IMPALA-14628: Fix mismatch between greatest synced event time and last synced event time
......................................................................


Patch Set 2: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/23797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I26550441bc0d5e7736c3b900c3e5178103157446
Gerrit-Change-Number: 23797
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Dec 2025 09:05:59 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14628: Fix mismatch between greatest synced event time and last synced event time

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/23797 )

Change subject: IMPALA-14628: Fix mismatch between greatest synced event time and last synced event time
......................................................................

IMPALA-14628: Fix mismatch between greatest synced event time and last synced event time

The greatest synced event id and its corresponding event time
must match the last synced event id and time once all outstanding
events have been fully processed.

Previously, for batch events, timestamp of the first event in
the batch is recorded when transitioning the batch event from
the in-progress log to the processed log in EventExecutorService.
As a result, the greatest synced event time incorrectly reflected
the first event’s timestamp, even though the greatest synced
event id corresponded to the last event in the batch, leading to
a mismatch.

This fix ensures that, for batch events, the timestamp of the
last event in the batch is recorded when the batch is marked as
processed. Consequently, the greatest synced event time now
correctly aligns with the last synced event id after all
outstanding events are processed.

Testing:
    - Added an FE test and ran the existing tests.

Change-Id: I26550441bc0d5e7736c3b900c3e5178103157446
Reviewed-on: http://gerrit.cloudera.org:8080/23797
Reviewed-by: Quanlong Huang <hu...@gmail.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/catalog/events/EventExecutorService.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
2 files changed, 34 insertions(+), 3 deletions(-)

Approvals:
  Quanlong Huang: Looks good to me, approved
  Impala Public Jenkins: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/23797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I26550441bc0d5e7736c3b900c3e5178103157446
Gerrit-Change-Number: 23797
Gerrit-PatchSet: 3
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-14628: Fix mismatch between greatest synced event time and last synced event time

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23797 )

Change subject: IMPALA-14628: Fix mismatch between greatest synced event time and last synced event time
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/21188/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/23797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I26550441bc0d5e7736c3b900c3e5178103157446
Gerrit-Change-Number: 23797
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 16 Dec 2025 20:50:02 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14628: Fix mismatch between greatest synced event time and last synced event time

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23797 )

Change subject: IMPALA-14628: Fix mismatch between greatest synced event time and last synced event time
......................................................................


Patch Set 2: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/23797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I26550441bc0d5e7736c3b900c3e5178103157446
Gerrit-Change-Number: 23797
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Dec 2025 14:00:56 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14628: Fix mismatch between greatest synced event time and last synced event time

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23797 )

Change subject: IMPALA-14628: Fix mismatch between greatest synced event time and last synced event time
......................................................................


Patch Set 1: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/23797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I26550441bc0d5e7736c3b900c3e5178103157446
Gerrit-Change-Number: 23797
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Dec 2025 01:54:53 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14628: Fix mismatch between greatest synced event time and last synced event time

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23797 )

Change subject: IMPALA-14628: Fix mismatch between greatest synced event time and last synced event time
......................................................................


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12802/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/23797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I26550441bc0d5e7736c3b900c3e5178103157446
Gerrit-Change-Number: 23797
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 16 Dec 2025 20:54:13 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14628: Fix mismatch between greatest synced event time and last synced event time

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23797 )

Change subject: IMPALA-14628: Fix mismatch between greatest synced event time and last synced event time
......................................................................


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12816/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/23797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I26550441bc0d5e7736c3b900c3e5178103157446
Gerrit-Change-Number: 23797
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Dec 2025 09:06:28 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14628: Fix mismatch between greatest synced event time and last synced event time

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23797 )

Change subject: IMPALA-14628: Fix mismatch between greatest synced event time and last synced event time
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/21213/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/23797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I26550441bc0d5e7736c3b900c3e5178103157446
Gerrit-Change-Number: 23797
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Dec 2025 09:21:11 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14628: Fix mismatch between greatest synced event time and last synced event time

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Sai Hemanth Gantasala, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/23797

to look at the new patch set (#2).

Change subject: IMPALA-14628: Fix mismatch between greatest synced event time and last synced event time
......................................................................

IMPALA-14628: Fix mismatch between greatest synced event time and last synced event time

The greatest synced event id and its corresponding event time
must match the last synced event id and time once all outstanding
events have been fully processed.

Previously, for batch events, timestamp of the first event in
the batch is recorded when transitioning the batch event from
the in-progress log to the processed log in EventExecutorService.
As a result, the greatest synced event time incorrectly reflected
the first event’s timestamp, even though the greatest synced
event id corresponded to the last event in the batch, leading to
a mismatch.

This fix ensures that, for batch events, the timestamp of the
last event in the batch is recorded when the batch is marked as
processed. Consequently, the greatest synced event time now
correctly aligns with the last synced event id after all
outstanding events are processed.

Testing:
    - Added an FE test and ran the existing tests.

Change-Id: I26550441bc0d5e7736c3b900c3e5178103157446
---
M fe/src/main/java/org/apache/impala/catalog/events/EventExecutorService.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
2 files changed, 34 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/23797/2
-- 
To view, visit http://gerrit.cloudera.org:8080/23797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I26550441bc0d5e7736c3b900c3e5178103157446
Gerrit-Change-Number: 23797
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-14628: Fix mismatch between greatest synced event time and last synced event time

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
k.venureddy2103@gmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/23797 )

Change subject: IMPALA-14628: Fix mismatch between greatest synced event time and last synced event time
......................................................................


Patch Set 2:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/23797/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/23797/1//COMMIT_MSG@19
PS1, Line 19: a mismatch.
> nit: add a period.
Done


http://gerrit.cloudera.org:8080/#/c/23797/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:

http://gerrit.cloudera.org:8080/#/c/23797/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@4381
PS1, Line 4381:     // HMS notification event time has second-level timestamp granularity. Add a delay to
> Could you add a comment on why we need this sleep?
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/23797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I26550441bc0d5e7736c3b900c3e5178103157446
Gerrit-Change-Number: 23797
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Dec 2025 08:56:05 +0000
Gerrit-HasComments: Yes