You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org> on 2024/12/02 16:58:53 UTC
[PR] HIVE-28654: MSCK repair fails for default partition when partition column is of numeric type [hive]
Aggarwal-Raghav opened a new pull request, #5568:
URL: https://github.com/apache/hive/pull/5568
### What changes were proposed in this pull request?
[HIVE-28654](https://issues.apache.org/jira/browse/HIVE-28654)
### Why are the changes needed?
To ensure msck repair command works when default partition "__HIVE_DEFAULT_PARTITION__" is present
### Does this PR introduce _any_ user-facing change?
NO
### Is the change a dependency upgrade?
NO
### How was this patch tested?
Using q files (present in PR)
```
mvn test -Dtest=TestMiniLlapLocalCliDriver -Dqfile=msck_repair_9.q -Drat.skip -Dtest.output.overwrite
mvn test -Dtest=TestMiniLlapLocalCliDriver -Dqfile=msck_repair_8.q -Drat.skip -Dtest.output.overwrite
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28654: MSCK repair fails for default partition when partition column is of numeric type [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5568:
URL: https://github.com/apache/hive/pull/5568#issuecomment-2527122008
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5568) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5568&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5568&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5568&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5568&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5568&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5568)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28654: MSCK repair fails for default partition when partition column is of numeric type [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on code in PR #5568:
URL: https://github.com/apache/hive/pull/5568#discussion_r1875400308
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java:
##########
@@ -440,12 +440,13 @@ void findUnknownPartitions(Table table, Set<Path> missingPartDirs,
partColNames.add(fSchema.getName());
}
+ String defaultPartName = MetastoreConf.getVar(conf, MetastoreConf.ConfVars.DEFAULTPARTITIONNAME);
Review Comment:
@ayushtkn Thanks for looking into this. Yes, I have update the PR based on your suggestion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28654: MSCK repair fails for default partition when partition column is of numeric type [hive]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #5568:
URL: https://github.com/apache/hive/pull/5568#discussion_r1875047399
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java:
##########
@@ -440,12 +440,13 @@ void findUnknownPartitions(Table table, Set<Path> missingPartDirs,
partColNames.add(fSchema.getName());
}
+ String defaultPartName = MetastoreConf.getVar(conf, MetastoreConf.ConfVars.DEFAULTPARTITIONNAME);
Review Comment:
I just had a quick look seems like the problem is coming post HIVE-23347, which added this method ``getNormalisedPartitionValue`` which does this parsing for numeric values. Before that patch things should have worked & Scenario-1 would have passed as well, with both values being shown in `show partitions`
This method ``getNormalisedPartitionValue`` looks like is valid only for numeric values to sort out some 0s or so. Maybe we can just have a numeric check & return rather than this default partition value matching
```
if(!NumberUtils.isParsable(partitionValue)) {
return partitionValue;
}
```
If we check the regular case without MSCK for scenario-1, the show partitions would show both the values, I tried locally
```
+------------------------------------------+
| partition |
+------------------------------------------+
| month=12/day=2 |
| month=12/day=3 |
| month=12/day=__HIVE_DEFAULT_PARTITION__ |
| month=__ANOTHER_PARTITION__/day=9 |
| month=__HIVE_DEFAULT_PARTITION__/day=3 |
+------------------------------------------+
5 rows selected (0.052 seconds)
0: jdbc:hive2://localhost:10000/>
```
So, post MSCK also we should show both those values. So, I believe we should just check whether the value is numeric or not, if not, just return the value as is..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28654: MSCK repair fails for default partition when partition column is of numeric type [hive]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn merged PR #5568:
URL: https://github.com/apache/hive/pull/5568
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28654: MSCK repair fails for default partition when partition column is of numeric type [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #5568:
URL: https://github.com/apache/hive/pull/5568#discussion_r1875666862
##########
ql/src/test/queries/clientpositive/msck_repair_9.q:
##########
@@ -0,0 +1,22 @@
+DROP TABLE IF EXISTS tbl_x;
+DROP TABLE IF EXISTS tbl_y;
+
+CREATE EXTERNAL TABLE tbl_x (id INT, name STRING) PARTITIONED BY (month INT, day INT) stored as ORC location '${system:test.tmp.dir}/apps/hive/warehouse/test.db/tbl_x/';
+
+INSERT INTO tbl_x values(1, 'aaa', 12, 2);
+INSERT INTO tbl_x values(2, 'bbb', 12, 3);
+INSERT INTO tbl_x (id, name, month) values(3, 'ccc', 12);
+
+SET hive.exec.default.partition.name=ANOTHER_PARTITION;
+INSERT INTO tbl_x (id, name, day) values(4, 'ddd', 3);
+
+SHOW PARTITIONS tbl_x;
+
+CREATE EXTERNAL TABLE tbl_y (id INT, name STRING) PARTITIONED BY (month INT, day INT) stored as ORC location '${system:test.tmp.dir}/apps/hive/warehouse/test.db/tbl_x/';
+
+MSCK REPAIR TABLE tbl_y;
+
+SHOW PARTITIONS tbl_y;
+
+DROP TABLE tbl_x;
+DROP TABLE tbl_y;
Review Comment:
Thanks for adding great test cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28654: MSCK repair fails for default partition when partition column is of numeric type [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on code in PR #5568:
URL: https://github.com/apache/hive/pull/5568#discussion_r1868785747
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java:
##########
@@ -1636,6 +1636,9 @@ public static String getPartitionName(Path tablePath, Path partitionPath, Set<St
public static String getNormalisedPartitionValue(String partitionValue, String type) {
+ if (partitionValue.equals("__HIVE_DEFAULT_PARTITION__"))
Review Comment:
@ngsg, thanks for reviewing the PR. Yes, the issue is regarding `hive.exec.default.partition.name`. As hive-common is not a dependency in _standalone-metastore/metastore-server/pom.xml_ , i think it better to use MetaConf as done in `getPartitionListByFilterExp` https://github.com/apache/hive/blob/2ff3ed6d83a1633d17bbf5b3c852b34b51548ef8/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java#L270
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28654: MSCK repair fails for default partition when partition column is of numeric type [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5568:
URL: https://github.com/apache/hive/pull/5568#issuecomment-2516405787
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5568) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5568&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5568&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5568&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5568&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5568&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5568)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28654: MSCK repair fails for default partition when partition column is of numeric type [hive]
Posted by "Indhumathi27 (via GitHub)" <gi...@apache.org>.
Indhumathi27 commented on code in PR #5568:
URL: https://github.com/apache/hive/pull/5568#discussion_r1870606329
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java:
##########
@@ -1634,7 +1634,10 @@ public static String getPartitionName(Path tablePath, Path partitionPath, Set<St
return result;
}
- public static String getNormalisedPartitionValue(String partitionValue, String type) {
+ public static String getNormalisedPartitionValue(String partitionValue, String type, String defaultPartName) {
Review Comment:
why dont we get and check MetastoreConf.ConfVars.DEFAULTPARTITIONNAME here itself, rather than passing from the caller ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28654: MSCK repair fails for default partition when partition column is of numeric type [hive]
Posted by "ngsg (via GitHub)" <gi...@apache.org>.
ngsg commented on code in PR #5568:
URL: https://github.com/apache/hive/pull/5568#discussion_r1867406987
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java:
##########
@@ -1636,6 +1636,9 @@ public static String getPartitionName(Path tablePath, Path partitionPath, Set<St
public static String getNormalisedPartitionValue(String partitionValue, String type) {
+ if (partitionValue.equals("__HIVE_DEFAULT_PARTITION__"))
Review Comment:
I have a quick question here. Does the patch solve the issue regardless of the value of `hive.exec.default.partition.name`? If not, I think it would be better to retreive the default partition name from `HiveConf` rather than using `__HIVE_DEFAULT_PARTITION__`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28654: MSCK repair fails for default partition when partition column is of numeric type [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on code in PR #5568:
URL: https://github.com/apache/hive/pull/5568#discussion_r1871190026
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java:
##########
@@ -1634,7 +1634,10 @@ public static String getPartitionName(Path tablePath, Path partitionPath, Set<St
return result;
}
- public static String getNormalisedPartitionValue(String partitionValue, String type) {
+ public static String getNormalisedPartitionValue(String partitionValue, String type, String defaultPartName) {
Review Comment:
We would need a conf object to fetch the value of DEFAULTPARTITIONNAME and conf is not attribute of MetaStoreServerUtils class. Therefore, String is passed in the function argument.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28654: MSCK repair fails for default partition when partition column is of numeric type [hive]
Posted by "Indhumathi27 (via GitHub)" <gi...@apache.org>.
Indhumathi27 commented on code in PR #5568:
URL: https://github.com/apache/hive/pull/5568#discussion_r1871258402
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java:
##########
@@ -440,12 +440,13 @@ void findUnknownPartitions(Table table, Set<Path> missingPartDirs,
partColNames.add(fSchema.getName());
}
+ String defaultPartName = MetastoreConf.getVar(conf, MetastoreConf.ConfVars.DEFAULTPARTITIONNAME);
Review Comment:
I think, this solution may not work in all scenarios. Since **hive.exec.default.partition.name** is session configurable, that means, we can use different default partition names with multiple inserts. And when user does MSCK repair, if we take current configuration value of hive.exec.default.partition.name, it can fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28654: MSCK repair fails for default partition when partition column is of numeric type [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on code in PR #5568:
URL: https://github.com/apache/hive/pull/5568#discussion_r1875735238
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java:
##########
@@ -1636,6 +1637,10 @@ public static String getPartitionName(Path tablePath, Path partitionPath, Set<St
public static String getNormalisedPartitionValue(String partitionValue, String type) {
+ if (!NumberUtils.isParsable(partitionValue)) {
+ return partitionValue;
+ }
Review Comment:
@okumin , thanks for the input. I think there is already a UT for this and passing post the patch. Please check https://github.com/apache/hive/blob/0e75930d619e96a32ae35b59df9eeeb6466ea2c6/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/TestMetaStoreServerUtils.java#L965
Let me know if I am missing something, but **NumberUtils.isParsable(partitionValue)** will return true for a decimal and later in same function it will be handled. https://github.com/apache/hive/blob/0e75930d619e96a32ae35b59df9eeeb6466ea2c6/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java#L1653
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28654: MSCK repair fails for default partition when partition column is of numeric type [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5568:
URL: https://github.com/apache/hive/pull/5568#issuecomment-2512300643
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5568) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5568&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5568&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5568&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5568&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5568&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5568)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28654: MSCK repair fails for default partition when partition column is of numeric type [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5568:
URL: https://github.com/apache/hive/pull/5568#issuecomment-2513681681
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5568) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5568&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5568&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5568&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5568&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5568&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5568)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28654: MSCK repair fails for default partition when partition column is of numeric type [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #5568:
URL: https://github.com/apache/hive/pull/5568#discussion_r1875668896
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java:
##########
@@ -1636,6 +1637,10 @@ public static String getPartitionName(Path tablePath, Path partitionPath, Set<St
public static String getNormalisedPartitionValue(String partitionValue, String type) {
+ if (!NumberUtils.isParsable(partitionValue)) {
+ return partitionValue;
+ }
Review Comment:
I slightly suspect this can skip the case of `decimal`.
https://commons.apache.org/proper/commons-lang/apidocs/org/apache/commons/lang3/math/NumberUtils.html#isParsable(java.lang.String)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28654: MSCK repair fails for default partition when partition column is of numeric type [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #5568:
URL: https://github.com/apache/hive/pull/5568#discussion_r1875773175
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java:
##########
@@ -1636,6 +1637,10 @@ public static String getPartitionName(Path tablePath, Path partitionPath, Set<St
public static String getNormalisedPartitionValue(String partitionValue, String type) {
+ if (!NumberUtils.isParsable(partitionValue)) {
+ return partitionValue;
+ }
Review Comment:
I originally thought it could not work when `Double.parseDouble` can not decode it but `new BigDecimal` can decode it. An example in my mind was a huge number. Looking at the implementation, now I feel the current implementation would work
https://github.com/apache/commons-lang/blob/daa63aa595ea39a69e1b44ac23bc50fdcf2afd62/src/main/java/org/apache/commons/lang3/math/NumberUtils.java#L1794-L1810
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28654: MSCK repair fails for default partition when partition column is of numeric type [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5568:
URL: https://github.com/apache/hive/pull/5568#issuecomment-2527710446
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5568) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5568&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5568&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5568&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5568&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5568&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5568)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28654: MSCK repair fails for default partition when partition column is of numeric type [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on code in PR #5568:
URL: https://github.com/apache/hive/pull/5568#discussion_r1874930571
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java:
##########
@@ -440,12 +440,13 @@ void findUnknownPartitions(Table table, Set<Path> missingPartDirs,
partColNames.add(fSchema.getName());
}
+ String defaultPartName = MetastoreConf.getVar(conf, MetastoreConf.ConfVars.DEFAULTPARTITIONNAME);
Review Comment:
**Yes, that case the solution won't work.**
But IMO, its user's responsibility to create the default partition of same type as that of the partition column type and ensure no other datatype partition column is present.
Consider 2 scenarios where partition column is of **INT** type:
**Scenario1:**
User1 sets default partition conf value to "ANOTHER_PARTITION" and user 2 in another beeline session tries to insert data in default partition, then hdfs dir structure will look something like this:
```
path to table/part=ANOTHER_PARITION
path to table/part=__HIVE_DEFAULT_PARTITION__
```
and running msck repair will never succeed for any user. One user will get this
```
Caused by: java.lang.NumberFormatException: For input string: "ANOTHER_PARITION"
```
and other will get this
```
Caused by: java.lang.NumberFormatException: For input string: "__HIVE_DEFAULT_PARTITION__"
```
**Scenario2:**
User set default partition conf value to INT. Lets say 13, so hdfs dir structure looks like this
```
path to table/part=13
path to table/part=__HIVE_DEFAULT_PARTITION__
```
and in same session user creates a new table pointing of location of old table and run msck repair then it will fail for \_\_HIVE_DEFAULT_PARTITION__.
Unless the user change the config back to \_\_HIVE_DEFAULT_PARTITION__
**Questions that I also have**, is going forward what should be the expected behaviour? That the user has to ensure that there shouldn't be 2 partitions of different datatype than that of partition column as shown in scenario 1?
CC @ayushtkn @deniskuzZ -- please provide your insights on this?
Also, there are multiple places where \_\_HIVE_DEFAULT_PARTITION__ is explicitly used instead of the config value. For instance here:
https://github.com/apache/hive/blob/c27d31722c2a7426f3236d3d892dbe1e206e840d/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/FileUtils.java#L307
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org