You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org> on 2024/10/24 04:46:53 UTC

[Impala-ASF-CR] IMPALA-13403: Trivial changes in StorageDescriptor of ALTER TABLE event is not enough to decide file metadata reload can be skipped

Sai Hemanth Gantasala has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21971


Change subject: IMPALA-13403: Trivial changes in StorageDescriptor of ALTER_TABLE event is not enough to decide file metadata reload can be skipped
......................................................................

IMPALA-13403: Trivial changes in StorageDescriptor of ALTER_TABLE
event is not enough to decide file metadata reload can be skipped

IMPALA-12487 adds an optimization that if an ALTER_TABLE event has
trivial changes in StorageDescriptor (e.g. removing optional field
'storedAsSubDirectories'=false which defaults to false), file
metadata reload will be skipped, no matter what changes are in the
table properties. This is problematic since some HMS clients (e.g.
Spark) could modify both the table properties and StorageDescriptor.
If there is a non-trivial changes in table properties (e.g. 'location'
change), we shouldn't skip reloading file metadata.

Testing:
- Added a unit test to verify the same

Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
2 files changed, 24 insertions(+), 5 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/21971/1
-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-13403: Trivial changes in StorageDescriptor of ALTER TABLE event is not enough to decide file metadata reload can be skipped

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Trivial changes in StorageDescriptor of ALTER_TABLE event is not enough to decide file metadata reload can be skipped
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/17256/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 24 Oct 2024 05:13:35 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER TABLE events

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER_TABLE events
......................................................................


Patch Set 7: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 7
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 10 Jan 2025 09:55:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13403: Trivial changes in StorageDescriptor of ALTER TABLE event is not enough to decide file metadata reload can be skipped

Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Trivial changes in StorageDescriptor of ALTER_TABLE event is not enough to decide file metadata reload can be skipped
......................................................................


Patch Set 2:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/21971/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/21971/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1940
PS1, Line 1940:         skipFileMetadata = true;
> Can't field schema / owner change at the same time as non trivial sd props?
I agree to your suggestion. There is no DDL to change both of them at the same time but this can be done in single Alter_table() API in HMS . Spark is changing SD and table properties at the same time through HMS API.


http://gerrit.cloudera.org:8080/#/c/21971/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1948
PS1, Line 1948:         if (isTrivialSdPropsChanged(beforeTable.getSd(), afterTable.getSd())) {
              :           skipFileMetadata = true;
              :         } else {
> The logic could be a bit simpler if isCustomTblPropsChanged() would return 
Ack



-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Oct 2024 21:01:35 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13403: Trivial changes in StorageDescriptor of ALTER TABLE event is not enough to decide file metadata reload can be skipped

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Trivial changes in StorageDescriptor of ALTER_TABLE event is not enough to decide file metadata reload can be skipped
......................................................................


Patch Set 2: Code-Review+1

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21971/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/21971/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1949
PS2, Line 1949:           skipFileMetadata = true;
              :         } else {
              :           skipFileMetadata = false;
IMO the function could be simpler (no need for skipFileMetadata/sdOrOwnerChanged) by:
- moving SD check earlier and return true/false instead of setting skipFileMetadata
- after the above checking owner change/file schema change and return true if any is true
- return false at the end the cause of the alter is unknown, better be safe



-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Sun, 03 Nov 2024 08:27:42 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER TABLE events

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER_TABLE events
......................................................................


Patch Set 6:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11160/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Nov 2024 17:35:46 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13403: Trivial changes in StorageDescriptor of ALTER TABLE event is not enough to decide file metadata reload can be skipped

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Trivial changes in StorageDescriptor of ALTER_TABLE event is not enough to decide file metadata reload can be skipped
......................................................................


Patch Set 1:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/21971/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/21971/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1940
PS1, Line 1940:         skipFileMetadata = true;
Can't field schema / owner change at the same time as non trivial sd props? I think that similar issues in the future could be avoided by doing these checks independently and always return false if something non-trivial changed.


http://gerrit.cloudera.org:8080/#/c/21971/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1948
PS1, Line 1948:       if (isCustomTblPropsChanged(whitelistedTblProperties, beforeTable,
              :           afterTable)) {
              :         return false;
The logic could be a bit simpler if isCustomTblPropsChanged() would return early before checking owner/SD



-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Oct 2024 08:28:57 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER TABLE events

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
k.venureddy2103@gmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER_TABLE events
......................................................................


Patch Set 4:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21971/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/21971/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1937
PS4, Line 1937:       if (!Objects.equals(beforeTable.getSd(), afterTable.getSd())) {
              :         if (isTrivialSdPropsChanged(beforeTable.getSd(), afterTable.getSd())) {
              :           skipFileMetadata = true;
              :         } else {
              :           return false;
              :         }
              :       }
              :       if (isFieldSchemaChanged(beforeTable, afterTable) ||
              :           isTableOwnerChanged(beforeTable.getOwner(), afterTable.getOwner())) {
              :         skipFileMetadata = true;
              :       }
              :       if (!isCustomTblPropsChanged(whitelistedTblProperties, beforeTable, afterTable)) {
              :         skipFileMetadata = true;
              :       } else {
              :         return false;
              :       }
              :       infoLog("Skipping reloading of file metadata for table {}.{} since SD and " +
              :           "whitelistedTblProperties has not changed.", dbName_, tblName_);
              :       return skipFileMetadata;
Field schema change seem to be already considered as trivial change with isTrivialSdPropsChanged() since we do sd.unsetCols() in normalizeStorageDescriptor() before comparing storage descriptors. I think, we do not need explicit isFieldSchemaChanged(beforeTable, afterTable) again. What is your opinion ?

How about we reorder something like this -  
// If SD has not trivial changes return false i.e., cannot skip file meta reload
if (!Objects.equals(beforeTable.getSd(), afterTable.getSd()) && !isTrivialSdPropsChanged(beforeTable.getSd(), afterTable.getSd())) {
        return false;
      }

// If custom table properties are changed, return false
      if (isCustomTblPropsChanged(whitelistedTblProperties, beforeTable, afterTable)) {
        return false;
      }

// Check for table object fields such as owner etc
...



-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Nov 2024 09:43:11 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13403: Trivial changes in StorageDescriptor of ALTER TABLE event is not enough to decide file metadata reload can be skipped

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Trivial changes in StorageDescriptor of ALTER_TABLE event is not enough to decide file metadata reload can be skipped
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/17311/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Oct 2024 21:28:17 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13403: Trivial changes in StorageDescriptor of ALTER TABLE event is not enough to decide file metadata reload can be skipped

Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Csaba Ringhofer, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/21971

to look at the new patch set (#3).

Change subject: IMPALA-13403: Trivial changes in StorageDescriptor of ALTER_TABLE event is not enough to decide file metadata reload can be skipped
......................................................................

IMPALA-13403: Trivial changes in StorageDescriptor of ALTER_TABLE
event is not enough to decide file metadata reload can be skipped

IMPALA-12487 adds an optimization that if an ALTER_TABLE event has
trivial changes in StorageDescriptor (e.g. removing optional field
'storedAsSubDirectories'=false which defaults to false), file
metadata reload will be skipped, no matter what changes are in the
table properties. This is problematic since some HMS clients (e.g.
Spark) could modify both the table properties and StorageDescriptor.
If there is a non-trivial changes in table properties (e.g. 'location'
change), we shouldn't skip reloading file metadata.

Testing:
- Added a unit test to verify the same

Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
2 files changed, 21 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/21971/3
-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-13403: Trivial changes in StorageDescriptor of ALTER TABLE event is not enough to decide file metadata reload can be skipped

Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Csaba Ringhofer, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/21971

to look at the new patch set (#2).

Change subject: IMPALA-13403: Trivial changes in StorageDescriptor of ALTER_TABLE event is not enough to decide file metadata reload can be skipped
......................................................................

IMPALA-13403: Trivial changes in StorageDescriptor of ALTER_TABLE
event is not enough to decide file metadata reload can be skipped

IMPALA-12487 adds an optimization that if an ALTER_TABLE event has
trivial changes in StorageDescriptor (e.g. removing optional field
'storedAsSubDirectories'=false which defaults to false), file
metadata reload will be skipped, no matter what changes are in the
table properties. This is problematic since some HMS clients (e.g.
Spark) could modify both the table properties and StorageDescriptor.
If there is a non-trivial changes in table properties (e.g. 'location'
change), we shouldn't skip reloading file metadata.

Testing:
- Added a unit test to verify the same

Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
2 files changed, 33 insertions(+), 12 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/21971/2
-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-13403: Trivial changes in StorageDescriptor of ALTER TABLE event is not enough to decide file metadata reload can be skipped

Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Trivial changes in StorageDescriptor of ALTER_TABLE event is not enough to decide file metadata reload can be skipped
......................................................................


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21971/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/21971/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1949
PS2, Line 1949:         skipFileMetadata = true;
              :       } else {
              :         return false;
> IMO the function could be simpler (no need for skipFileMetadata/sdOrOwnerCh
This scenario would fail the existing test case whether we don't expect to reload file metadata when same value is set for custom table property. Can you see if my latest change looks better?



-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Nov 2024 00:56:31 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER TABLE events

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER_TABLE events
......................................................................


Patch Set 6: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 10 Jan 2025 09:53:55 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER TABLE events

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER_TABLE events
......................................................................


Patch Set 6:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11207/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Sat, 14 Dec 2024 03:07:44 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER TABLE events

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER_TABLE events
......................................................................


Patch Set 6:

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/11160/


-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Nov 2024 17:53:33 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER TABLE events

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER_TABLE events
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/17500/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Nov 2024 23:38:31 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER TABLE events

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER_TABLE events
......................................................................

IMPALA-13403: Refactor the checks of skip reloading file metadata for
ALTER_TABLE events

IMPALA-12487 adds an optimization that if an ALTER_TABLE event has
trivial changes in StorageDescriptor (e.g. removing optional field
'storedAsSubDirectories'=false which defaults to false), file
metadata reload will be skipped, no matter what changes are in the
table properties. This is problematic since some HMS clients (e.g.
Spark) could modify both the table properties and StorageDescriptor.
If there is a non-trivial changes in table properties (e.g. 'location'
change), we shouldn't skip reloading file metadata.

Testing:
- Added a unit test to verify the same

Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Reviewed-on: http://gerrit.cloudera.org:8080/21971
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
2 files changed, 19 insertions(+), 11 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 8
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER TABLE events

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER_TABLE events
......................................................................


Patch Set 6:

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/11155/


-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Nov 2024 10:53:27 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER TABLE events

Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, k.venureddy2103@gmail.com, Csaba Ringhofer, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/21971

to look at the new patch set (#5).

Change subject: IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER_TABLE events
......................................................................

IMPALA-13403: Refactor the checks of skip reloading file metadata for
ALTER_TABLE events

IMPALA-12487 adds an optimization that if an ALTER_TABLE event has
trivial changes in StorageDescriptor (e.g. removing optional field
'storedAsSubDirectories'=false which defaults to false), file
metadata reload will be skipped, no matter what changes are in the
table properties. This is problematic since some HMS clients (e.g.
Spark) could modify both the table properties and StorageDescriptor.
If there is a non-trivial changes in table properties (e.g. 'location'
change), we shouldn't skip reloading file metadata.

Testing:
- Added a unit test to verify the same

Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
2 files changed, 20 insertions(+), 7 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/21971/5
-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-13403: Trivial changes in StorageDescriptor of ALTER TABLE event is not enough to decide file metadata reload can be skipped

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Trivial changes in StorageDescriptor of ALTER_TABLE event is not enough to decide file metadata reload can be skipped
......................................................................


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21971/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/21971/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1953
PS3, Line 1953: skipFileMetadata
When we reach here, 'skipFileMetadata' is always true since it's either set to true at L1949 or we return false at L1951. Then the if-branch at L1944 of checking columns and owner doesn't affect the final result, and we can remove it.

This means if the ALTER_TABLE event doesn't change SD and whitelistedTblProperties, we will skip file metadata reloading. Can we add a log about the checks and the decision? E.g. "Skip reloading file metadata of table xxx since SD and whitelistedTblProperties not changed".



-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Nov 2024 01:02:47 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13403: Trivial changes in StorageDescriptor of ALTER TABLE event is not enough to decide file metadata reload can be skipped

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Trivial changes in StorageDescriptor of ALTER_TABLE event is not enough to decide file metadata reload can be skipped
......................................................................


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21971/3//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/21971/3//COMMIT_MSG@7
PS3, Line 7: IMPALA-13403: Trivial changes in StorageDescriptor of ALTER_TABLE
           : event is not enough to decide file metadata reload can be skipped
nit: Can we use a shorter title, e.g. "refactor the checks of skip reloading file metadata for ALTER_TABLE events"?



-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Nov 2024 01:04:05 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER TABLE events

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER_TABLE events
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/17477/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Nov 2024 23:38:19 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER TABLE events

Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Csaba Ringhofer, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/21971

to look at the new patch set (#4).

Change subject: IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER_TABLE events
......................................................................

IMPALA-13403: Refactor the checks of skip reloading file metadata for
ALTER_TABLE events

IMPALA-12487 adds an optimization that if an ALTER_TABLE event has
trivial changes in StorageDescriptor (e.g. removing optional field
'storedAsSubDirectories'=false which defaults to false), file
metadata reload will be skipped, no matter what changes are in the
table properties. This is problematic since some HMS clients (e.g.
Spark) could modify both the table properties and StorageDescriptor.
If there is a non-trivial changes in table properties (e.g. 'location'
change), we shouldn't skip reloading file metadata.

Testing:
- Added a unit test to verify the same

Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
2 files changed, 24 insertions(+), 7 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/21971/4
-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER TABLE events

Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER_TABLE events
......................................................................


Patch Set 4:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/21971/3//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/21971/3//COMMIT_MSG@7
PS3, Line 7: IMPALA-13403: Refactor the checks of skip reloading file metadata for
           : ALTER_TABLE events
> nit: Can we use a shorter title, e.g. "refactor the checks of skip reloadin
Ack


http://gerrit.cloudera.org:8080/#/c/21971/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/21971/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1953
PS3, Line 1953: ("Skipping reloa
> When we reach here, 'skipFileMetadata' is always true since it's either set
Ack



-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Nov 2024 00:44:42 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER TABLE events

Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER_TABLE events
......................................................................


Patch Set 4:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/21971/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/21971/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1953
PS3, Line 1953: ("Skipping reloa
> I just see a new log is added. What do you think about the first part - do 
I don't see any risks, so I'm removing it.


http://gerrit.cloudera.org:8080/#/c/21971/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/21971/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1937
PS4, Line 1937:       if (!Objects.equals(beforeTable.getSd(), afterTable.getSd())) {
              :         if (isTrivialSdPropsChanged(beforeTable.getSd(), afterTable.getSd())) {
              :           skipFileMetadata = true;
              :         } else {
              :           return false;
              :         }
              :       }
              :       if (isFieldSchemaChanged(beforeTable, afterTable) ||
              :           isTableOwnerChanged(beforeTable.getOwner(), afterTable.getOwner())) {
              :         skipFileMetadata = true;
              :       }
              :       if (!isCustomTblPropsChanged(whitelistedTblProperties, beforeTable, afterTable)) {
              :         skipFileMetadata = true;
              :       } else {
              :         return false;
              :       }
              :       infoLog("Skipping reloading of file metadata for table {}.{} since SD and " +
              :           "whitelistedTblProperties has not changed.", dbName_, tblName_);
              :       return skipFileMetadata;
> Field schema change seem to be already considered as trivial change with is
In the latest patchset, I'll only consider SD and custom table props and ignore others.



-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Nov 2024 23:10:01 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER TABLE events

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER_TABLE events
......................................................................


Patch Set 6:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11150/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 19 Nov 2024 21:49:01 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER TABLE events

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
k.venureddy2103@gmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER_TABLE events
......................................................................


Patch Set 6: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 30 Dec 2024 06:12:04 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER TABLE events

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER_TABLE events
......................................................................


Patch Set 7: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 7
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 10 Jan 2025 15:21:26 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER TABLE events

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER_TABLE events
......................................................................


Patch Set 7:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11301/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 7
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 10 Jan 2025 09:55:58 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER TABLE events

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER_TABLE events
......................................................................


Patch Set 5:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/21971/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/21971/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1938
PS5, Line 1938:         if (isTrivialSdPropsChanged(beforeTable.getSd(), afterTable.getSd())) {
              :           skipFileMetadata = true;
              :         } else {
              :           return false;
              :         }
nit: we can simplify this to

        if (!isTrivialSdPropsChanged(beforeTable.getSd(), afterTable.getSd())) {
          return false;
        }

It seems variable 'skipFileMetadata' is useless.


http://gerrit.cloudera.org:8080/#/c/21971/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1944
PS5, Line 1944:       if (!isCustomTblPropsChanged(whitelistedTblProperties, beforeTable, afterTable)) {
              :         skipFileMetadata = true;
              :       } else {
              :         return false;
              :       }
              :       infoLog("Skipping reloading of file metadata for table {}.{} since SD and " +
              :           "whitelistedTblProperties has not changed.", dbName_, tblName_);
              :       return skipFileMetadata;
nit: we can simplify this to

      if (isCustomTblPropsChanged(whitelistedTblProperties, beforeTable, afterTable)) {
        return false;
      }
      infoLog("Skipping reloading of file metadata for table {}.{} since SD and " +
          "whitelistedTblProperties has not changed.", dbName_, tblName_);
      return true;



-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Nov 2024 08:20:58 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER TABLE events

Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER_TABLE events
......................................................................


Patch Set 6:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/21971/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/21971/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1938
PS5, Line 1938:           return false;
              :         }
              :       }
              :       if (isCustomTblPropsChanged(whitelistedTblProperties, beforeTable, afterTable)) {
              :         r
> nit: we can simplify this to
Ack


http://gerrit.cloudera.org:8080/#/c/21971/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1944
PS5, Line 1944:       infoLog("Skipping reloading of file metadata for table {}.{} since SD and " +
              :           "whitelistedTblProperties has not changed.", dbName_, tblName_);
              :       return true;
              :     }
              : 
              :     private boolean isFieldSchemaChanged(
              :         org.apache.hadoop.hive.metastore.api.Table beforeTable,
              :         org.apache.hadoop.hive
> nit: we can simplify this to
Ack



-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Nov 2024 23:11:47 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER TABLE events

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER_TABLE events
......................................................................


Patch Set 6: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Sat, 14 Dec 2024 08:44:37 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER TABLE events

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER_TABLE events
......................................................................


Patch Set 4:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/17442/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Nov 2024 01:02:01 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13403: Trivial changes in StorageDescriptor of ALTER TABLE event is not enough to decide file metadata reload can be skipped

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Trivial changes in StorageDescriptor of ALTER_TABLE event is not enough to decide file metadata reload can be skipped
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/17387/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Nov 2024 01:23:24 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13403: Trivial changes in StorageDescriptor of ALTER TABLE event is not enough to decide file metadata reload can be skipped

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Trivial changes in StorageDescriptor of ALTER_TABLE event is not enough to decide file metadata reload can be skipped
......................................................................


Patch Set 3: Code-Review+1

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21971/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/21971/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1949
PS2, Line 1949:         skipFileMetadata = true;
              :       } else {
              :         return false;
> This scenario would fail the existing test case whether we don't expect to 
thanks, looks good!



-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Nov 2024 08:38:12 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER TABLE events

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER_TABLE events
......................................................................


Patch Set 4:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21971/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/21971/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1953
PS3, Line 1953: ("Skipping reloa
> Ack
I just see a new log is added. What do you think about the first part - do we still need to check schema and owner changes? Any risks if we just check SD and whitelistedTblProperties changes?



-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Nov 2024 01:36:27 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER TABLE events

Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, k.venureddy2103@gmail.com, Csaba Ringhofer, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/21971

to look at the new patch set (#6).

Change subject: IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER_TABLE events
......................................................................

IMPALA-13403: Refactor the checks of skip reloading file metadata for
ALTER_TABLE events

IMPALA-12487 adds an optimization that if an ALTER_TABLE event has
trivial changes in StorageDescriptor (e.g. removing optional field
'storedAsSubDirectories'=false which defaults to false), file
metadata reload will be skipped, no matter what changes are in the
table properties. This is problematic since some HMS clients (e.g.
Spark) could modify both the table properties and StorageDescriptor.
If there is a non-trivial changes in table properties (e.g. 'location'
change), we shouldn't skip reloading file metadata.

Testing:
- Added a unit test to verify the same

Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
2 files changed, 19 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/21971/6
-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER TABLE events

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER_TABLE events
......................................................................


Patch Set 6: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/11150/


-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 19 Nov 2024 22:06:56 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER TABLE events

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER_TABLE events
......................................................................


Patch Set 6: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Sun, 15 Dec 2024 00:17:50 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER TABLE events

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21971 )

Change subject: IMPALA-13403: Refactor the checks of skip reloading file metadata for ALTER_TABLE events
......................................................................


Patch Set 6:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11155/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/21971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia969dd32385ac5a1a9a65890a5ccc8cd257f4b97
Gerrit-Change-Number: 21971
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Nov 2024 10:36:16 +0000
Gerrit-HasComments: No