You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Anonymous Coward (Code Review)" <ge...@cloudera.org> on 2023/09/15 11:53:12 UTC

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

k.venureddy2103@gmail.com has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20486


Change subject:     IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................

IMPALA-12356: Alter partition event from hive is treated as self event

    When a non-transactional table is created and a partition is added using
    insert from impala, alter partition event from hive is treated as self
    event and ignored processing. Self event check for add partition event
    is done only for the transactional tables with IMPALA-10502 fix. It was
    missed to avoid the addition of version number to partition's
    inFlightEvents_ when a partition is added to non-transactional table.
    Thus the version number in inFlightEvents_ is detecting the next alter
    partition event from hive as self event.

    Another issue found during test:
    After the IMPALA-12399 fix, certain event types like OPEN_TXN are not
    fetched with get_next_notification() as an opmization. It can cause a
    deadloop in MetastoreEventsProcessor#getNextMetastoreEventsInBatches()
    when skipped events are present after currentEventId.
    Need to return when events received in response is empty.

    Testing:
    - Manually tested in cluster and added a testcase

Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
---
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
3 files changed, 77 insertions(+), 0 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/20486/3
-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 3
Gerrit-Owner: Anonymous Coward <k....@gmail.com>

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 17:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14921/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 17
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 10 Jan 2024 19:12:30 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 18: Code-Review+2

Thanks for making this fixed!


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 18
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 11 Jan 2024 01:53:46 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Sai Hemanth Gantasala, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20486

to look at the new patch set (#18).

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................

IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be
treated as self event

Self event check for add partition event is done only for the
transactional tables with IMPALA-10502 (commit id: 7f7a631). But
during addition of new partition(with insert statement), catalog
service id and version number are added to partition params of the
parition irrespective of whether the table is transactional or not.
Thus the version number is added to partition's inFlightEvents_ and
remained in it until the next alter partition event from hive. Thus
led to detection of the alter partition event as self event.

This commit ensures the catalog service id and version number are not
added to partition params if the partition is added to a
non-transactional table.

Also fixed another bug in reload event. Reload event self check
fails due to the above fix as it expects catalog service id and
version number in the partition params. Fixed to use last refreshed
event id to skip the self reload events.

Testing:
- Manually tested in cluster and added testcases

Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_events_custom_configs.py
M tests/metadata/test_event_processing.py
6 files changed, 101 insertions(+), 60 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/20486/18
-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 18
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 15:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10119/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 15
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 04 Jan 2024 16:40:29 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 13:

> Patch Set 13: Verified-1
> 
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10099/

The failure in test_orc_stats is resolved in IMPALA-12630. Please rebase this patch.

> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10102/

The failure in test_mixed_catalog_ddls_with_invalidate_metadata is tracked in IMPALA-12670.


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 13
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Jan 2024 05:18:08 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
k.venureddy2103@gmail.com has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/20486 )

Change subject:     IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................

IMPALA-12356: Alter partition event from hive is treated as self event

    When a non-transactional table is created and a partition is added using
    insert from impala, alter partition event from hive is treated as self
    event and ignored processing. Self event check for add partition event
    is done only for the transactional tables with IMPALA-10502 fix. It was
    missed to avoid the addition of version number to partition's
    inFlightEvents_ when a partition is added to non-transactional table.
    Thus the version number in inFlightEvents_ is detecting the next alter
    partition event from hive as self event.

    Testing:
    - Manually tested in cluster and added a testcase

Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
---
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
2 files changed, 70 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/20486/5
-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 8: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 8
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Sep 2023 12:11:01 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 4: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9720/


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 4
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 15 Sep 2023 18:54:33 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
k.venureddy2103@gmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 18:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/20486/16/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
File fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java:

http://gerrit.cloudera.org:8080/#/c/20486/16/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java@762
PS16, Line 762:   private final long lastRefreshEventId_;
> HdfsPartition instances are immutable. We should keep the 'final' keyword a
Done


http://gerrit.cloudera.org:8080/#/c/20486/16/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/20486/16/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@a6820
PS16, Line 6820: 
               : 
               : 
> Please keep using the original way to update the partition, i.e. using Hdfs
Done


http://gerrit.cloudera.org:8080/#/c/20486/16/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4565
PS16, Line 4565:         }
> Nit: this can go in above line
It is formatted with:-

git diff -U0 --no-color HEAD^ | "${IMPALA_TOOLCHAIN_PACKAGES_HOME}/llvm-${IMPALA_LLVM_VERSION}/share/clang/clang-format-diff.py" -binary "${IMPALA_TOOLCHAIN_PACKAGES_HOME}/llvm-${IMPALA_LLVM_VERSION}/bin/clang-format" -p1 -i


http://gerrit.cloudera.org:8080/#/c/20486/16/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:

http://gerrit.cloudera.org:8080/#/c/20486/16/tests/custom_cluster/test_events_custom_configs.py@250
PS16, Line 250:     enable_sync_to_latest_event_on_ddls=False):
> Nit: 4 empty spaces to start new line.
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 18
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 11 Jan 2024 01:20:01 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12356: First alter partition event from hive is treated as self event when the partition is added from impala(with insert statement) for a non-transactional table.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: First alter partition event from hive is treated as self event when the partition is added from impala(with insert statement) for a non-transactional table.
......................................................................


Patch Set 11: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10041/


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 11
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 15 Dec 2023 14:54:27 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: First alter partition event from hive is treated as self event when the partition is added from impala(with insert statement) for a non-transactional table.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: First alter partition event from hive is treated as self event when the partition is added from impala(with insert statement) for a non-transactional table.
......................................................................


Patch Set 11:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14738/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 11
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 15 Dec 2023 10:52:29 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Laszlo Gaal (Code Review)" <ge...@cloudera.org>.
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 4:

Fixed a typo in the commit message to force a new patchset


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 4
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 15 Sep 2023 14:30:21 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Sai Hemanth Gantasala, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20486

to look at the new patch set (#10).

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................

IMPALA-12356: Alter partition event from hive is treated as self event

Self event check for add partition event is done only for the
transactional tables with IMPALA-10502 (commit id: 7f7a631).
Thus the version number remained in inFlightEvents_ and is
used in detecting the next alter partition event from hive as
self event.
This commit does add partition self event check for both
non-transactional and transactional tables.

Testing:
- Manually tested in cluster and added a testcase

Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsSyncToLatestEventIdTest.java
M tests/metadata/test_event_processing.py
3 files changed, 45 insertions(+), 16 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/20486/10
-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 10
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 13:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10102/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 13
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 02 Jan 2024 21:28:23 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
k.venureddy2103@gmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 10:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/20486/10/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/20486/10/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@a2049
PS10, Line 2049: 
> So we are reverting some changes of 7f7a631. Does it mean 'createEventId' c
Yes. self event detection is common for both transactional and non-transactional tables. We use version number for add PartitionEvent self event detection and eventId for InsertEvent self event detection. I have verified AddPartitionEvent, AlterPartitionEvent, InsertEvent, BatchPartitionEvent(with AlterPartitionEvent and InsertEvent) and DropPartitionEvent events. Have tested add/drop with multiple partitions as well.


http://gerrit.cloudera.org:8080/#/c/20486/10/fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsSyncToLatestEventIdTest.java
File fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsSyncToLatestEventIdTest.java:

http://gerrit.cloudera.org:8080/#/c/20486/10/fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsSyncToLatestEventIdTest.java@165
PS10, Line 165:         catalog_.setMetastoreEventProcessor(eventsProcessor_);
> Could you explain why do we need this for this patch?
Events generated from previous tests are not processed with eventsProcessor_. So they are being processed in the subsequent tests when eventsProcessor_.processEvents() is called. CatalogServiceCatalog#isEventProcessingActive() fails during CatalogServiceCatalog#evaluateSelfEvent() because catalog_.getMetastoreEventProcessor() is not the instance of MetastoreEventsProcessor. It is a NoOpEventProcessor instance.

So doing this similar to MetastoreEventsProcessorTest, to ensure the tests don't process previous testcase events. 

This is observed with isSelfEvent() during AddPartitionEvent.process()



-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 10
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Dec 2023 14:27:45 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12356: First alter partition event from hive is treated as self event when the partition is added from impala(with insert statement) for a non-transactional table.

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Sai Hemanth Gantasala, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20486

to look at the new patch set (#11).

Change subject: IMPALA-12356: First alter partition event from hive is treated as self event when the partition is added from impala(with insert statement) for a non-transactional table.
......................................................................

IMPALA-12356: First alter partition event from hive is treated as self
event when the partition is added from impala(with insert statement)
for a non-transactional table.

Self event check for add partition event is done only for the
transactional tables with IMPALA-10502 (commit id: 7f7a631). But
during addition of new partition(with insert statement), catalog
service id and version number are added to partition params of the
parition irrespective of whether the table is transactional or not.
Thus the version number is added to partition's inFlightEvents_ and
remained in it until the next alter partition event from hive. Thus
led to detection of the alter partition event as self event.

This commit ensures the catalog service id and version number are not
added to partition params if the partition is added to a
non-transactinal table.

Testing:
- Manually tested in cluster and added a testcase

Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
---
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/metadata/test_event_processing.py
2 files changed, 28 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/20486/11
-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 11
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
k.venureddy2103@gmail.com has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................

IMPALA-12356: Alter partition event from hive is treated as self event

Self event check for add partition event is done only for the
transactional tables with IMPALA-10502 fix. Thus the version
number remained in inFlightEvents_ and is used in detecting
the next alter partition event from hive as self event.

Testing:
- Manually tested in cluster and added a testcase

Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
---
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/metadata/test_event_processing.py
2 files changed, 31 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/20486/8
-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 8
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 13: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10099/


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 13
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 02 Jan 2024 20:44:11 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Sai Hemanth Gantasala, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20486

to look at the new patch set (#15).

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................

IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be
treated as self event

Self event check for add partition event is done only for the
transactional tables with IMPALA-10502 (commit id: 7f7a631). But
during addition of new partition(with insert statement), catalog
service id and version number are added to partition params of the
parition irrespective of whether the table is transactional or not.
Thus the version number is added to partition's inFlightEvents_ and
remained in it until the next alter partition event from hive. Thus
led to detection of the alter partition event as self event.

This commit ensures the catalog service id and version number are not
added to partition params if the partition is added to a
non-transactional table.

Also fixed another bug in reload event. Reload event self check
fails due to the above fix as it expects catalog service id and
version number in the partition params. Fixed to use last refreshed
event id to skip the self reload events.

Testing:
- Manually tested in cluster and added testcases

Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_events_custom_configs.py
M tests/metadata/test_event_processing.py
6 files changed, 103 insertions(+), 62 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/20486/15
-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 15
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 16:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14882/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 16
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 05 Jan 2024 05:33:39 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 10:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/20486/10/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/20486/10/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@a2049
PS10, Line 2049: 
So we are reverting some changes of 7f7a631. Does it mean 'createEventId' can't help in detecting self-events? Can we double check the scenario of consecutive ADD/DROP partition events are handled correctly?


http://gerrit.cloudera.org:8080/#/c/20486/10/fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsSyncToLatestEventIdTest.java
File fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsSyncToLatestEventIdTest.java:

http://gerrit.cloudera.org:8080/#/c/20486/10/fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsSyncToLatestEventIdTest.java@165
PS10, Line 165:         catalog_.setMetastoreEventProcessor(eventsProcessor_);
Could you explain why do we need this for this patch?



-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 10
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 27 Nov 2023 14:02:50 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 10:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/20486/10//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/20486/10//COMMIT_MSG@7
PS10, Line 7: IMPALA-12356: Alter partition event from hive is treated as self event
I think we need a more specific title. Not all the ALTER_PARTITION events are treated as self events.


http://gerrit.cloudera.org:8080/#/c/20486/10/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/20486/10/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@a2049
PS10, Line 2049: 
> Yes. self event detection is common for both transactional and non-transact
I was confused on why the bug of IMPALA-10502 won't occur after reverting this. Later I realized the new mechanism still exists so this is just added back as the first guard so it works.

However, we tend to remove the old mechanism of self-event detection in the future. It might be better to fix the issue in the root source, i.e. not adding the catalog version into the inFlightEvents list when INSERT creates new partitions. I left more details in the JIRA comment: https://issues.apache.org/jira/browse/IMPALA-12356?focusedCommentId=17795150&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17795150

We can continue the discussion there.


http://gerrit.cloudera.org:8080/#/c/20486/10/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/20486/10/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4563
PS10, Line 4563: in
nit: could you add a space before "in" by the way?


http://gerrit.cloudera.org:8080/#/c/20486/10/fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsSyncToLatestEventIdTest.java
File fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsSyncToLatestEventIdTest.java:

http://gerrit.cloudera.org:8080/#/c/20486/10/fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsSyncToLatestEventIdTest.java@165
PS10, Line 165:         catalog_.setMetastoreEventProcessor(eventsProcessor_);
> Events generated from previous tests are not processed with eventsProcessor
Ack



-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 10
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Dec 2023 03:44:00 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 10:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14358/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 10
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 07 Nov 2023 15:26:49 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 16:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/20486/16/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
File fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java:

http://gerrit.cloudera.org:8080/#/c/20486/16/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java@762
PS16, Line 762:   private long lastRefreshEventId_;
HdfsPartition instances are immutable. We should keep the 'final' keyword and update this using HdfsPartition.Builder.


http://gerrit.cloudera.org:8080/#/c/20486/16/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/20486/16/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@a6820
PS16, Line 6820: 
               : 
               : 
Please keep using the original way to update the partition, i.e. using HdfsPartition.Builder.



-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 16
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 10 Jan 2024 05:13:22 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Sai Hemanth Gantasala, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20486

to look at the new patch set (#17).

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................

IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be
treated as self event

Self event check for add partition event is done only for the
transactional tables with IMPALA-10502 (commit id: 7f7a631). But
during addition of new partition(with insert statement), catalog
service id and version number are added to partition params of the
parition irrespective of whether the table is transactional or not.
Thus the version number is added to partition's inFlightEvents_ and
remained in it until the next alter partition event from hive. Thus
led to detection of the alter partition event as self event.

This commit ensures the catalog service id and version number are not
added to partition params if the partition is added to a
non-transactional table.

Also fixed another bug in reload event. Reload event self check
fails due to the above fix as it expects catalog service id and
version number in the partition params. Fixed to use last refreshed
event id to skip the self reload events.

Testing:
- Manually tested in cluster and added testcases

Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_events_custom_configs.py
M tests/metadata/test_event_processing.py
6 files changed, 101 insertions(+), 60 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/20486/17
-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 17
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 18: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 18
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 11 Jan 2024 01:14:37 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 18:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10138/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 18
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 10 Jan 2024 20:31:34 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 7:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/20486/7/tests/metadata/test_event_processing.py
File tests/metadata/test_event_processing.py:

http://gerrit.cloudera.org:8080/#/c/20486/7/tests/metadata/test_event_processing.py@440
PS7, Line 440: %
flake8: E501 line too long (104 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/20486/7/tests/metadata/test_event_processing.py@443
PS7, Line 443: n
flake8: E501 line too long (100 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/20486/7/tests/metadata/test_event_processing.py@449
PS7, Line 449:  
flake8: E501 line too long (114 > 90 characters)



-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 7
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Sep 2023 07:21:33 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12356: First alter partition event from hive is treated as self event when the partition is added from impala(with insert statement) for a non-transactional table.

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: First alter partition event from hive is treated as self event when the partition is added from impala(with insert statement) for a non-transactional table.
......................................................................


Patch Set 11:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/20486/11//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/20486/11//COMMIT_MSG@7
PS11, Line 7: IMPALA-12356: First alter partition event from hive is treated as self
            : event when the partition is added from impala(with insert statement)
            : for a non-transactional table.
nit: this might be too long. We can put this in the commit message body and only leave "Fix first ALTER_PARTITION event from Hive could be treated as self event" in the title.



-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 11
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Sun, 17 Dec 2023 23:44:22 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 16: Code-Review+1

(2 comments)

http://gerrit.cloudera.org:8080/#/c/20486/16/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/20486/16/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4565
PS16, Line 4565:               eventId,
Nit: this can go in above line


http://gerrit.cloudera.org:8080/#/c/20486/16/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:

http://gerrit.cloudera.org:8080/#/c/20486/16/tests/custom_cluster/test_events_custom_configs.py@250
PS16, Line 250:                                          enable_sync_to_latest_event_on_ddls=False):
Nit: 4 empty spaces to start new line.



-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 16
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 10 Jan 2024 02:12:51 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 10: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 10
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 07 Nov 2023 19:58:01 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
k.venureddy2103@gmail.com has removed Sai Hemanth Gantasala from this change.  ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Removed reviewer Sai Hemanth Gantasala.
-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteReviewer
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 6
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
k.venureddy2103@gmail.com has removed Quanlong Huang from this change.  ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Removed reviewer Quanlong Huang.
-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteReviewer
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 6
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Laszlo Gaal (Code Review)" <ge...@cloudera.org>.
Laszlo Gaal has uploaded a new patch set (#4) to the change originally created by k.venureddy2103@gmail.com. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................

IMPALA-12356: Alter partition event from hive is treated as self event

    When a non-transactional table is created and a partition is added using
    insert from impala, alter partition event from hive is treated as self
    event and ignored processing. Self event check for add partition event
    is done only for the transactional tables with IMPALA-10502 fix. It was
    missed to avoid the addition of version number to partition's
    inFlightEvents_ when a partition is added to non-transactional table.
    Thus the version number in inFlightEvents_ is detecting the next alter
    partition event from hive as self event.

    Another issue found during test:
    After the IMPALA-12399 fix, certain event types like OPEN_TXN are not
    fetched with get_next_notification() as an optimization. It can cause a
    deadloop in MetastoreEventsProcessor#getNextMetastoreEventsInBatches()
    when skipped events are present after currentEventId.
    Need to return when events received in response is empty.

    Testing:
    - Manually tested in cluster and added a testcase

Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
---
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
3 files changed, 77 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/20486/4
-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 4
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9720/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 4
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 15 Sep 2023 14:30:49 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject:     IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14010/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 15 Sep 2023 15:07:40 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
k.venureddy2103@gmail.com has removed Laszlo Gaal from this change.  ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Removed reviewer Laszlo Gaal.
-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteReviewer
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 6
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
k.venureddy2103@gmail.com has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................

IMPALA-12356: Alter partition event from hive is treated as self event

When a non-transactional table is created and a partition is added using
insert from impala, alter partition event from hive is treated as self
event and ignored processing. Self event check for add partition event
is done only for the transactional tables with IMPALA-10502 fix. It was
missed to avoid the addition of version number to partition's
inFlightEvents_ when a partition is added to non-transactional table.
Thus the version number in inFlightEvents_ is detecting the next alter
partition event from hive as self event.

Testing:
- Manually tested in cluster and added a testcase

Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
---
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
2 files changed, 70 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/20486/6
-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 6
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 6:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/20486/6//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/20486/6//COMMIT_MSG@9
PS6, Line 9: using
           : insert
Do you mean dynamic partition insert?
Does it happen with alter table add partition operation?


http://gerrit.cloudera.org:8080/#/c/20486/6//COMMIT_MSG@12
PS6, Line 12: It was
            : missed to avoid the addition of version number
nit: can you rephrase this? It is really not clear


http://gerrit.cloudera.org:8080/#/c/20486/6//COMMIT_MSG@19
PS6, Line 19: - Manually tested in cluster and added a testcase
Can you also add an end-to-end test to prove that the alter partition event from hive will not be treated as self event?



-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 6
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Sep 2023 21:08:11 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 14: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 14
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Jan 2024 16:16:34 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
k.venureddy2103@gmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 14:

(3 comments)

Fixed review comments

http://gerrit.cloudera.org:8080/#/c/20486/11//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/20486/11//COMMIT_MSG@7
PS11, Line 7: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be
            : treated as self event
            : 
> nit: this might be too long. We can put this in the commit message body and
Done


http://gerrit.cloudera.org:8080/#/c/20486/11/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/20486/11/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@6914
PS11, Line 6914:           Lists.newArrayList();
> nit: could you add a comment here mentioning self-event detection is deprec
Done


http://gerrit.cloudera.org:8080/#/c/20486/11/tests/metadata/test_event_processing.py
File tests/metadata/test_event_processing.py:

http://gerrit.cloudera.org:8080/#/c/20486/11/tests/metadata/test_event_processing.py@440
PS11, Line 440:   def test_partition_location_change_from_hive(self, unique_database):
> It'd be nice to also add test coverage on transactional tables. Impala can 
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 14
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 04 Jan 2024 06:22:13 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 15:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10116/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 15
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 04 Jan 2024 12:01:47 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Sai Hemanth Gantasala, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20486

to look at the new patch set (#13).

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................

IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be
treated as self event

Self event check for add partition event is done only for the
transactional tables with IMPALA-10502 (commit id: 7f7a631). But
during addition of new partition(with insert statement), catalog
service id and version number are added to partition params of the
parition irrespective of whether the table is transactional or not.
Thus the version number is added to partition's inFlightEvents_ and
remained in it until the next alter partition event from hive. Thus
led to detection of the alter partition event as self event.

This commit ensures the catalog service id and version number are not
added to partition params if the partition is added to a
non-transactinal table.

Also fixed another bug in reload event. Reload event self check
fails due to the above fix as it expects catalog service id and
version number in the partition params. Fixed to use last refreshed
event id to skip the self reload events.

Testing:
- Manually tested in cluster and added testcases

Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_events_custom_configs.py
M tests/metadata/test_event_processing.py
6 files changed, 103 insertions(+), 62 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/20486/13
-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 13
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 19: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 19
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 11 Jan 2024 07:07:15 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 18:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14925/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 18
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 10 Jan 2024 20:50:32 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 9:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14274/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 9
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 27 Oct 2023 08:09:43 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14013/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 6
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 15 Sep 2023 15:43:24 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 9:

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9878/


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 9
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Nov 2023 05:11:40 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
k.venureddy2103@gmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 10:

(2 comments)

reworked

http://gerrit.cloudera.org:8080/#/c/20486/8//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/20486/8//COMMIT_MSG@10
PS8, Line 10: IMPALA-10502 (co
> Could you mention the commit id? There are several commits in IMPALA-10502.
It was done with commit id: 7f7a631. Have added this id in the commit message now.


http://gerrit.cloudera.org:8080/#/c/20486/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/20486/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4538
PS8, Line 4538:       return partsToBeAdded;
> Is it true that for non-transactional ADD_PARTITION self-events, this is th
"Is it true that for non-transactional ADD_PARTITION self-events, this is the only place to remove items in the inflight event list? and after IMPALA-10502, we don't have a place to do so which cause the next event on the partition being always treated as self-event?"
-> Yes.

"The method name doesn't indicate it has side effect (modify the partitions). It'd be better to do this in other place or modify the method name."
-> Agreed. Have made the change at common place for both transactional and non-transactional cases.



-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 10
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 08 Nov 2023 03:19:04 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 13:

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10102/


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 13
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Jan 2024 02:01:58 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Sai Hemanth Gantasala, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20486

to look at the new patch set (#16).

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................

IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be
treated as self event

Self event check for add partition event is done only for the
transactional tables with IMPALA-10502 (commit id: 7f7a631). But
during addition of new partition(with insert statement), catalog
service id and version number are added to partition params of the
parition irrespective of whether the table is transactional or not.
Thus the version number is added to partition's inFlightEvents_ and
remained in it until the next alter partition event from hive. Thus
led to detection of the alter partition event as self event.

This commit ensures the catalog service id and version number are not
added to partition params if the partition is added to a
non-transactional table.

Also fixed another bug in reload event. Reload event self check
fails due to the above fix as it expects catalog service id and
version number in the partition params. Fixed to use last refreshed
event id to skip the self reload events.

Testing:
- Manually tested in cluster and added testcases

Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_events_custom_configs.py
M tests/metadata/test_event_processing.py
6 files changed, 103 insertions(+), 62 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/20486/16
-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 16
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 8:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9770/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 8
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Sep 2023 07:49:37 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
k.venureddy2103@gmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 8:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/20486/6//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/20486/6//COMMIT_MSG@12
PS6, Line 12: 
            : 
> nit: can you rephrase this? It is really not clear
Done


http://gerrit.cloudera.org:8080/#/c/20486/6//COMMIT_MSG@19
PS6, Line 19: 
> Can you also add an end-to-end test to prove that the alter partition event
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 8
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Sep 2023 12:48:49 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 9:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9855/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 9
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 27 Oct 2023 09:01:23 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 9: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9855/


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 9
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 27 Oct 2023 13:29:55 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................

IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be
treated as self event

Self event check for add partition event is done only for the
transactional tables with IMPALA-10502 (commit id: 7f7a631). But
during addition of new partition(with insert statement), catalog
service id and version number are added to partition params of the
parition irrespective of whether the table is transactional or not.
Thus the version number is added to partition's inFlightEvents_ and
remained in it until the next alter partition event from hive. Thus
led to detection of the alter partition event as self event.

This commit ensures the catalog service id and version number are not
added to partition params if the partition is added to a
non-transactional table.

Also fixed another bug in reload event. Reload event self check
fails due to the above fix as it expects catalog service id and
version number in the partition params. Fixed to use last refreshed
event id to skip the self reload events.

Testing:
- Manually tested in cluster and added testcases

Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Reviewed-on: http://gerrit.cloudera.org:8080/20486
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_events_custom_configs.py
M tests/metadata/test_event_processing.py
6 files changed, 101 insertions(+), 60 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 20
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 7:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14097/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 7
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Sep 2023 07:49:27 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 8:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/20486/8//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/20486/8//COMMIT_MSG@10
PS8, Line 10: IMPALA-10502 fix
Could you mention the commit id? There are several commits in IMPALA-10502. It's unclear to me why IMPALA-10502 is related to transactional tables.


http://gerrit.cloudera.org:8080/#/c/20486/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/20486/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4538
PS8, Line 4538:           hdfsPartition.removeFromVersionsForInflightEvents(false,
Is it true that for non-transactional ADD_PARTITION self-events, this is the only place to remove items in the inflight event list? and after IMPALA-10502, we don't have a place to do so which cause the next event on the partition being always treated as self-event?

The method name doesn't indicate it has side effect (modify the partitions). It'd be better to do this in other place or modify the method name.



-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 8
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Oct 2023 08:32:08 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 13:

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10104/


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 13
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Jan 2024 08:43:55 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 12:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14852/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 12
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 02 Jan 2024 15:56:10 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: First alter partition event from hive is treated as self event when the partition is added from impala(with insert statement) for a non-transactional table.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: First alter partition event from hive is treated as self event when the partition is added from impala(with insert statement) for a non-transactional table.
......................................................................


Patch Set 11:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10041/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 11
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 15 Dec 2023 10:28:37 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 16: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10124/


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 16
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 05 Jan 2024 09:43:24 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 9:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9878/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 9
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Nov 2023 00:30:31 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Sai Hemanth Gantasala, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20486

to look at the new patch set (#9).

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................

IMPALA-12356: Alter partition event from hive is treated as self event

Self event check for add partition event is done only for the
transactional tables with IMPALA-10502 (commit id: 7f7a631).
Thus the version number remained in inFlightEvents_ and is
used in detecting the next alter partition event from hive as
self event.
This commit does add partition self event check for both
non-transactional and transactional tables.

Testing:
- Manually tested in cluster and added a testcase

Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M tests/metadata/test_event_processing.py
2 files changed, 37 insertions(+), 16 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/20486/9
-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 9
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 12:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/20486/12/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:

http://gerrit.cloudera.org:8080/#/c/20486/12/tests/custom_cluster/test_events_custom_configs.py@249
PS12, Line 249: s
flake8: E501 line too long (126 > 90 characters)



-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 12
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 02 Jan 2024 15:31:08 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 13:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10099/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 13
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 02 Jan 2024 16:16:14 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 16:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10124/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 16
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 05 Jan 2024 05:08:11 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 15: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10116/


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 15
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 04 Jan 2024 16:34:13 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 15:

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10119/


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 15
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 04 Jan 2024 21:14:00 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 19:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10142/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 19
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 11 Jan 2024 01:54:21 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 19: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 19
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 11 Jan 2024 01:54:20 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 7:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9769/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 7
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Sep 2023 07:21:02 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
k.venureddy2103@gmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 8:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/20486/6//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/20486/6//COMMIT_MSG@9
PS6, Line 9: 
           : transa
> Can you answer this question?
Yes. This issue happens only with dynamic partition insert. Not with alter table add partition.


http://gerrit.cloudera.org:8080/#/c/20486/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/20486/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4537
PS8, Line 4537:                 params.get(MetastoreEventPropertyKey.CATALOG_SERVICE_ID.getKey()))) {
> Should we check whether the key 'CATALOG_SERVICE_ID' exists in the table pa
No need to check explicitly. NPE doesn't happen since String.equals() returns false when anObject passed to it is null.


http://gerrit.cloudera.org:8080/#/c/20486/8/tests/metadata/test_event_processing.py
File tests/metadata/test_event_processing.py:

http://gerrit.cloudera.org:8080/#/c/20486/8/tests/metadata/test_event_processing.py@456
PS8, Line 456:     assert [(new_part_location,)] == partitions
> Can you also assert that the number of events processed before and after ch
get_impala_partition_info() at line 455 executes the show partitions query on impala. So if alter cmd from hive(at line 451) is not applied on catalogd, the location returned would still be old location. I think, location check will suffice.



-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 8
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Oct 2023 06:59:32 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12356: First alter partition event from hive is treated as self event when the partition is added from impala(with insert statement) for a non-transactional table.

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: First alter partition event from hive is treated as self event when the partition is added from impala(with insert statement) for a non-transactional table.
......................................................................


Patch Set 11:

(2 comments)

> Patch Set 11: Verified-1
> 
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10041/

There is a test failure. Not sure if it's related:
https://jenkins.impala.io/job/ubuntu-20.04-from-scratch/1174

custom_cluster/test_events_custom_configs.py:274: in test_refresh_invalidate_events
    .format(unique_database, test_reload_table))
custom_cluster/test_events_custom_configs.py:271: in check_self_events
    assert events_skipped_after > events_skipped_before
E   assert 4 > 4

http://gerrit.cloudera.org:8080/#/c/20486/11/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/20486/11/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@6914
PS11, Line 6914:               if (AcidUtils.isTransactionalTable(msTbl.getParameters())) {
nit: could you add a comment here mentioning self-event detection is deprecated for non-transactional tables so we only add catalog service identifiers for transactional tables?


http://gerrit.cloudera.org:8080/#/c/20486/11/tests/metadata/test_event_processing.py
File tests/metadata/test_event_processing.py:

http://gerrit.cloudera.org:8080/#/c/20486/11/tests/metadata/test_event_processing.py@440
PS11, Line 440:     self.client.execute("create table %s (i int) partitioned by(j int) stored as parquet"
It'd be nice to also add test coverage on transactional tables. Impala can write to insert-only transactional tables.



-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 11
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Sun, 17 Dec 2023 23:41:43 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject:     IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14008/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 3
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 15 Sep 2023 12:20:12 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 8:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/20486/6//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/20486/6//COMMIT_MSG@9
PS6, Line 9: 
           : transa
> Do you mean dynamic partition insert?
Can you answer this question?


http://gerrit.cloudera.org:8080/#/c/20486/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/20486/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4537
PS8, Line 4537:                 params.get(MetastoreEventPropertyKey.CATALOG_SERVICE_ID.getKey()))) {
Should we check whether the key 'CATALOG_SERVICE_ID' exists in the table parameters or not? Or you could use map.getOrDefult('CATALOG_SERVICE_ID', ""); Otherwise we would be hitting NullPointerException here.


http://gerrit.cloudera.org:8080/#/c/20486/8/tests/metadata/test_event_processing.py
File tests/metadata/test_event_processing.py:

http://gerrit.cloudera.org:8080/#/c/20486/8/tests/metadata/test_event_processing.py@456
PS8, Line 456:     assert [(new_part_location,)] == partitions
Can you also assert that the number of events processed before and after changed? Just to verify that query from hive is not treated as self event?



-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 8
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Oct 2023 16:11:14 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
k.venureddy2103@gmail.com has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................

IMPALA-12356: Alter partition event from hive is treated as self event

Self event check for add partition event is done only for the
transactional tables with IMPALA-10502 fix. Thus the version
number remained in inFlightEvents_ and is used in detecting
the next alter partition event from hive as self event.

Testing:
- Manually tested in cluster and added a testcase

Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
---
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/metadata/test_event_processing.py
2 files changed, 28 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/20486/7
-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 7
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 8:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14099/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 8
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Sep 2023 08:07:05 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 7: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 7
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Sep 2023 11:47:48 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 14:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10107/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 14
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Jan 2024 11:36:17 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Alter partition event from hive is treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Alter partition event from hive is treated as self event
......................................................................


Patch Set 10:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9897/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 10
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 07 Nov 2023 15:32:37 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Sai Hemanth Gantasala, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20486

to look at the new patch set (#12).

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................

IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be
treated as self event

Self event check for add partition event is done only for the
transactional tables with IMPALA-10502 (commit id: 7f7a631). But
during addition of new partition(with insert statement), catalog
service id and version number are added to partition params of the
parition irrespective of whether the table is transactional or not.
Thus the version number is added to partition's inFlightEvents_ and
remained in it until the next alter partition event from hive. Thus
led to detection of the alter partition event as self event.

This commit ensures the catalog service id and version number are not
added to partition params if the partition is added to a
non-transactinal table.

Also fixed another bug in reload event. Reload event self check
fails due to the above fix as it expects catalog service id and
version number in the partition params. Fixed to use last refreshed
event id to skip the self reload events.

Testing:
- Manually tested in cluster and added testcases

Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_events_custom_configs.py
M tests/metadata/test_event_processing.py
6 files changed, 102 insertions(+), 62 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/20486/12
-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 12
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 13:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14853/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 13
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 02 Jan 2024 16:02:19 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 13:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10104/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 13
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Jan 2024 04:13:09 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 16:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10126/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 16
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 05 Jan 2024 11:18:49 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12356: Fix first ALTER PARTITION event from Hive could be treated as self event

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20486 )

Change subject: IMPALA-12356: Fix first ALTER_PARTITION event from Hive could be treated as self event
......................................................................


Patch Set 16: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/20486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c2affa3fe32c0b3843bff5e4c0018dce9060d3
Gerrit-Change-Number: 20486
Gerrit-PatchSet: 16
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 05 Jan 2024 15:57:46 +0000
Gerrit-HasComments: No