You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org> on 2023/04/26 03:22:52 UTC
[Impala-ASF-CR] IMPALA-12099: Catalogd throw NPE when refreshing a partitioned table
Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19804
Change subject: IMPALA-12099: Catalogd throw NPE when refreshing a partitioned table
......................................................................
IMPALA-12099: Catalogd throw NPE when refreshing a partitioned table
Impala Catalogd assumes that if a table is partitioned then all
CompactionInfoStruct entries should not have null partitionname.
This assumption is not true. As according to [1], it is possible
for a compaction queue entry for a partitioned table to have null
partitionname.
This patch handles the null case.
Testing:
- Ran core test.
[1] https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L345
Change-Id: If37559f3c74ac517d1403ddb82ca28708015d6bf
---
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
1 file changed, 8 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/19804/1
--
To view, visit http://gerrit.cloudera.org:8080/19804
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: If37559f3c74ac517d1403ddb82ca28708015d6bf
Gerrit-Change-Number: 19804
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] IMPALA-12099: Catalogd throws NPE when refreshing a partitioned table
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19804 )
Change subject: IMPALA-12099: Catalogd throws NPE when refreshing a partitioned table
......................................................................
IMPALA-12099: Catalogd throws NPE when refreshing a partitioned table
Impala Catalogd assumes that if a table is partitioned then all
CompactionInfoStruct entries should not have null partitionname.
This assumption is not true. As according to [1], it is possible
for a compaction queue entry for a partitioned table to have null
partitionname.
This patch handles the null case.
Testing:
- Ran core test.
[1] https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L345
Change-Id: If37559f3c74ac517d1403ddb82ca28708015d6bf
Reviewed-on: http://gerrit.cloudera.org:8080/19804
Reviewed-by: Quanlong Huang <hu...@gmail.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
1 file changed, 16 insertions(+), 4 deletions(-)
Approvals:
Quanlong Huang: Looks good to me, approved
Impala Public Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/19804
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: If37559f3c74ac517d1403ddb82ca28708015d6bf
Gerrit-Change-Number: 19804
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] IMPALA-12099: Catalogd throw NPE when refreshing a partitioned table
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19804 )
Change subject: IMPALA-12099: Catalogd throw NPE when refreshing a partitioned table
......................................................................
Patch Set 1: Code-Review+1
(2 comments)
http://gerrit.cloudera.org:8080/#/c/19804/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/19804/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@697
PS1, Line 697: }
For supportability, can we add a warning log when the partition name is null? We can print the required fields of CompactionInfoStruct and probably the 'errorMessage' if exists, or just dump the whole CompactionInfoStruct object.
http://gerrit.cloudera.org:8080/#/c/19804/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@757
PS1, Line 757: }
Same here, can we add a warning log when the partition name is null?
--
To view, visit http://gerrit.cloudera.org:8080/19804
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If37559f3c74ac517d1403ddb82ca28708015d6bf
Gerrit-Change-Number: 19804
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 26 Apr 2023 04:55:38 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12099: Catalogd throws NPE when refreshing a partitioned table
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19804 )
Change subject: IMPALA-12099: Catalogd throws NPE when refreshing a partitioned table
......................................................................
Patch Set 2:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/19804/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19804/1//COMMIT_MSG@7
PS1, Line 7: throw
> Nit: throws
Done
http://gerrit.cloudera.org:8080/#/c/19804/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/19804/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@697
PS1, Line 697: } else {
> For supportability, can we add a warning log when the partition name is nul
dump the whole object if the partition name is null
http://gerrit.cloudera.org:8080/#/c/19804/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@757
PS1, Line 757: if (table.isPartitioned()) {
> Same here, can we add a warning log when the partition name is null?
Done
--
To view, visit http://gerrit.cloudera.org:8080/19804
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If37559f3c74ac517d1403ddb82ca28708015d6bf
Gerrit-Change-Number: 19804
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 26 Apr 2023 22:28:14 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12099: Catalogd throws NPE when refreshing a partitioned table
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19804 )
Change subject: IMPALA-12099: Catalogd throws NPE when refreshing a partitioned table
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9257/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/19804
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If37559f3c74ac517d1403ddb82ca28708015d6bf
Gerrit-Change-Number: 19804
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Apr 2023 01:25:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12099: Catalogd throws NPE when refreshing a partitioned table
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19804 )
Change subject: IMPALA-12099: Catalogd throws NPE when refreshing a partitioned table
......................................................................
Patch Set 2: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/19804
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If37559f3c74ac517d1403ddb82ca28708015d6bf
Gerrit-Change-Number: 19804
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Apr 2023 06:42:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12099: Catalogd throw NPE when refreshing a partitioned table
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19804 )
Change subject: IMPALA-12099: Catalogd throw NPE when refreshing a partitioned table
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/12869/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19804
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If37559f3c74ac517d1403ddb82ca28708015d6bf
Gerrit-Change-Number: 19804
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 26 Apr 2023 03:43:49 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12099: Catalogd throw NPE when refreshing a partitioned table
Posted by "Daniel Becker (Code Review)" <ge...@cloudera.org>.
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/19804 )
Change subject: IMPALA-12099: Catalogd throw NPE when refreshing a partitioned table
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19804/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19804/1//COMMIT_MSG@7
PS1, Line 7: throw
Nit: throws
--
To view, visit http://gerrit.cloudera.org:8080/19804
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If37559f3c74ac517d1403ddb82ca28708015d6bf
Gerrit-Change-Number: 19804
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 26 Apr 2023 11:04:11 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12099: Catalogd throws NPE when refreshing a partitioned table
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/19804 )
Change subject: IMPALA-12099: Catalogd throws NPE when refreshing a partitioned table
......................................................................
IMPALA-12099: Catalogd throws NPE when refreshing a partitioned table
Impala Catalogd assumes that if a table is partitioned then all
CompactionInfoStruct entries should not have null partitionname.
This assumption is not true. As according to [1], it is possible
for a compaction queue entry for a partitioned table to have null
partitionname.
This patch handles the null case.
Testing:
- Ran core test.
[1] https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L345
Change-Id: If37559f3c74ac517d1403ddb82ca28708015d6bf
---
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
1 file changed, 16 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/19804/2
--
To view, visit http://gerrit.cloudera.org:8080/19804
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If37559f3c74ac517d1403ddb82ca28708015d6bf
Gerrit-Change-Number: 19804
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-12099: Catalogd throws NPE when refreshing a partitioned table
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19804 )
Change subject: IMPALA-12099: Catalogd throws NPE when refreshing a partitioned table
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/12874/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19804
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If37559f3c74ac517d1403ddb82ca28708015d6bf
Gerrit-Change-Number: 19804
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 26 Apr 2023 22:49:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12099: Catalogd throws NPE when refreshing a partitioned table
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19804 )
Change subject: IMPALA-12099: Catalogd throws NPE when refreshing a partitioned table
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/19804
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If37559f3c74ac517d1403ddb82ca28708015d6bf
Gerrit-Change-Number: 19804
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Apr 2023 01:24:29 +0000
Gerrit-HasComments: No