You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "VenkatSNarayanan (via GitHub)" <gi...@apache.org> on 2024/12/26 22:11:05 UTC
[PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
VenkatSNarayanan opened a new pull request, #5591:
URL: https://github.com/apache/hive/pull/5591
Adds support for custom partitioning patterns to MSCK repair table.
### What changes were proposed in this pull request?
Adds support for custom partitioning patterns to MSCK repair table.
### Why are the changes needed?
HCatStorer supports custom partitioning patterns when using dynamic partitioning, but Hive itself does not support this. This change adds support for non-pathological cases to Hive.
### Does this PR introduce _any_ user-facing change?
MSCK repair table with a configured custom partition pattern would previously ignore that pattern and error on finding nonstandard paths. With the code from this PR, it will respect the defined custom pattern when it extracts partition key values from the paths.
### Is the change a dependency upgrade?
No
### How was this patch tested?
A test was added to TestHiveMetastoreChecker to test the common kinds of custom patterns supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5591:
URL: https://github.com/apache/hive/pull/5591#issuecomment-2755890657
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5591) **Quality Gate passed**
Issues
 [7 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5591&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5591&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5591&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5591&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5591&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5591)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5591:
URL: https://github.com/apache/hive/pull/5591#discussion_r2082316741
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java:
##########
@@ -678,7 +710,7 @@ void checkPartitionDirs(final ExecutorService executor,
//process each level in parallel
while(!nextLevel.isEmpty()) {
futures.add(
- executor.submit(new PathDepthInfoCallable(nextLevel.poll(), partColNames, fs, tempQueue)));
+ executor.submit(new PathDepthInfoCallable(nextLevel.poll(), partColNames, fs, tempQueue, conf.get("hcat.dynamic.partitioning.custom.pattern"))));
Review Comment:
use HiveConf var
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5591:
URL: https://github.com/apache/hive/pull/5591#discussion_r2082322365
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java:
##########
@@ -1729,4 +1753,68 @@ public static List<String> getColumnNames(List<FieldSchema> cols) {
public static boolean isCompactionTxn(TxnType txnType) {
return TxnType.COMPACTION.equals(txnType) || TxnType.REBALANCE_COMPACTION.equals(txnType);
}
+
+ /**
+ * When using msck repair table with custom partitioning patterns, we need to capture
+ * the partition keys from the pattern, and use those to construct a regex which will
+ * match the paths and extract the partition key values.*/
+ public static class DynamicPartitioningCustomPattern {
+
+ private final String customPattern;
+ private final Pattern partitionCapturePattern;
+ private final List<String> partitionColumns;
+
+ private DynamicPartitioningCustomPattern(String customPattern, Pattern partitionCapturePattern, List<String> partitionColumns) {
+ this.customPattern = customPattern;
+ this.partitionCapturePattern = partitionCapturePattern;
+ this.partitionColumns = partitionColumns;
+ }
+
+ /**
+ * @return stored custom pattern string
+ * */
+ public String getCustomPattern() {
+ return customPattern;
+ }
+
+ /**
+ * @return stored custom pattern regex matcher
+ * */
+ public Pattern getPartitionCapturePattern() {
+ return partitionCapturePattern;
+ }
+
+ /**
+ * @return list of partition key columns
+ * */
+ public List<String> getPartitionColumns() {
+ return partitionColumns;
+ }
+
+ public static class Builder {
+ private String customPattern;
+
+ public Builder setCustomPattern(String customPattern) {
+ this.customPattern = customPattern;
+ return this;
+ }
+
+ /**
+ * Constructs the regex to match the partition values in a path based on the custom pattern.
+ *
+ * @return custom partition pattern matcher */
+ public DynamicPartitioningCustomPattern build() {
+ Pattern stringPattern = Pattern.compile("(\\$\\{)([^\\s/\\{\\}\\\\]+)(\\})");
Review Comment:
please move Pattern to constant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5591:
URL: https://github.com/apache/hive/pull/5591#discussion_r2082321231
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java:
##########
@@ -1729,4 +1753,68 @@ public static List<String> getColumnNames(List<FieldSchema> cols) {
public static boolean isCompactionTxn(TxnType txnType) {
return TxnType.COMPACTION.equals(txnType) || TxnType.REBALANCE_COMPACTION.equals(txnType);
}
+
+ /**
+ * When using msck repair table with custom partitioning patterns, we need to capture
+ * the partition keys from the pattern, and use those to construct a regex which will
+ * match the paths and extract the partition key values.*/
+ public static class DynamicPartitioningCustomPattern {
Review Comment:
should it be extracted into it's own class file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5591:
URL: https://github.com/apache/hive/pull/5591#discussion_r2082323066
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java:
##########
@@ -1729,4 +1753,68 @@ public static List<String> getColumnNames(List<FieldSchema> cols) {
public static boolean isCompactionTxn(TxnType txnType) {
return TxnType.COMPACTION.equals(txnType) || TxnType.REBALANCE_COMPACTION.equals(txnType);
}
+
+ /**
+ * When using msck repair table with custom partitioning patterns, we need to capture
+ * the partition keys from the pattern, and use those to construct a regex which will
+ * match the paths and extract the partition key values.*/
+ public static class DynamicPartitioningCustomPattern {
+
+ private final String customPattern;
+ private final Pattern partitionCapturePattern;
+ private final List<String> partitionColumns;
+
+ private DynamicPartitioningCustomPattern(String customPattern, Pattern partitionCapturePattern, List<String> partitionColumns) {
+ this.customPattern = customPattern;
+ this.partitionCapturePattern = partitionCapturePattern;
+ this.partitionColumns = partitionColumns;
+ }
+
+ /**
+ * @return stored custom pattern string
+ * */
+ public String getCustomPattern() {
+ return customPattern;
+ }
+
+ /**
+ * @return stored custom pattern regex matcher
+ * */
+ public Pattern getPartitionCapturePattern() {
+ return partitionCapturePattern;
+ }
+
+ /**
+ * @return list of partition key columns
+ * */
+ public List<String> getPartitionColumns() {
+ return partitionColumns;
+ }
+
+ public static class Builder {
+ private String customPattern;
+
+ public Builder setCustomPattern(String customPattern) {
+ this.customPattern = customPattern;
+ return this;
+ }
+
+ /**
+ * Constructs the regex to match the partition values in a path based on the custom pattern.
+ *
+ * @return custom partition pattern matcher */
+ public DynamicPartitioningCustomPattern build() {
+ Pattern stringPattern = Pattern.compile("(\\$\\{)([^\\s/\\{\\}\\\\]+)(\\})");
+ StringBuffer sb = new StringBuffer();
+ Matcher m = stringPattern.matcher(customPattern);
+ ArrayList<String> partColumns = new ArrayList<>();
Review Comment:
please use interfaces
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5591:
URL: https://github.com/apache/hive/pull/5591#discussion_r2082294933
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java:
##########
@@ -445,7 +447,7 @@ void findUnknownPartitions(Table table, Set<Path> missingPartDirs,
for (Path partPath : missingPartDirs) {
FileSystem fs = partPath.getFileSystem(conf);
String partitionName = getPartitionName(fs.makeQualified(tablePath),
- partPath, partColNames, partitionColToTypeMap);
+ partPath, partColNames, partitionColToTypeMap, conf.get("hcat.dynamic.partitioning.custom.pattern"));
Review Comment:
use constant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on PR #5591:
URL: https://github.com/apache/hive/pull/5591#issuecomment-2606070138
@VenkatSNarayanan you can login in this page by your github account.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5591:
URL: https://github.com/apache/hive/pull/5591#discussion_r2082288994
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java:
##########
@@ -414,7 +414,7 @@ private static AlterTableAddPartitionDesc getBaseAddPartitionDescFromPartition(P
partitionSpec, location, partition.getParameters(), sd.getInputFormat(), sd.getOutputFormat(),
sd.getNumBuckets(), sd.getCols(), sd.getSerdeInfo().getSerializationLib(), sd.getSerdeInfo().getParameters(),
sd.getBucketCols(), sd.getSortCols(), null, writeId);
- return new AlterTableAddPartitionDesc(dbName, tblDesc.getTableName(), true, ImmutableList.of(partitionDesc));
+ return new AlterTableAddPartitionDesc(dbName, tblDesc.getTableName(), true, ImmutableList.of(partitionDesc), conf.get("hcat.dynamic.partitioning.custom.pattern", null));
Review Comment:
use HiveConf var
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5591:
URL: https://github.com/apache/hive/pull/5591#issuecomment-2574205750
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5591) **Quality Gate passed**
Issues
 [7 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5591&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5591&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5591&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5591&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5591&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5591)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5591:
URL: https://github.com/apache/hive/pull/5591#issuecomment-2608359211
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5591) **Quality Gate passed**
Issues
 [7 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5591&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5591&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5591&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5591&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5591&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5591)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5591:
URL: https://github.com/apache/hive/pull/5591#issuecomment-2563162831
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5591) **Quality Gate passed**
Issues
 [13 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5591&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5591&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5591&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5591&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5591&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5591)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5591:
URL: https://github.com/apache/hive/pull/5591#issuecomment-2756502305
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5591) **Quality Gate passed**
Issues
 [7 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5591&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5591&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5591&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5591&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5591&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5591)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #5591:
URL: https://github.com/apache/hive/pull/5591#issuecomment-2867529764
re-triggered the build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5591:
URL: https://github.com/apache/hive/pull/5591#discussion_r2082280621
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java:
##########
@@ -141,6 +148,16 @@ public static org.apache.hadoop.hive.metastore.api.Partition createMetaPartition
if (!tbl.isView()) {
tpart.setSd(tbl.getSd().deepCopy());
+ if (location == null && customPattern != null) { //should only happen for msck repair table with custom pattern
+ Pattern p = Pattern.compile("(\\$\\{)([^\\s/\\{\\}\\\\]+)(\\})");
Review Comment:
can be moved to constants
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5591:
URL: https://github.com/apache/hive/pull/5591#discussion_r2082260133
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/add/AbstractAddPartitionAnalyzer.java:
##########
@@ -70,7 +70,7 @@ protected void analyzeCommand(TableName tableName, Map<String, String> partition
}
AlterTableAddPartitionDesc desc = new AlterTableAddPartitionDesc(table.getDbName(), table.getTableName(),
- ifNotExists, partitions);
+ ifNotExists, partitions, conf.get("hcat.dynamic.partitioning.custom.pattern", null));
Review Comment:
why do you need to specify `null` default and not use HiveConf constant?
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/add/AbstractAddPartitionAnalyzer.java:
##########
@@ -70,7 +70,7 @@ protected void analyzeCommand(TableName tableName, Map<String, String> partition
}
AlterTableAddPartitionDesc desc = new AlterTableAddPartitionDesc(table.getDbName(), table.getTableName(),
- ifNotExists, partitions);
+ ifNotExists, partitions, conf.get("hcat.dynamic.partitioning.custom.pattern", null));
Review Comment:
why do you need to specify `null` default and not use HiveConf var?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5591:
URL: https://github.com/apache/hive/pull/5591#discussion_r2082260133
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/add/AbstractAddPartitionAnalyzer.java:
##########
@@ -70,7 +70,7 @@ protected void analyzeCommand(TableName tableName, Map<String, String> partition
}
AlterTableAddPartitionDesc desc = new AlterTableAddPartitionDesc(table.getDbName(), table.getTableName(),
- ifNotExists, partitions);
+ ifNotExists, partitions, conf.get("hcat.dynamic.partitioning.custom.pattern", null));
Review Comment:
why do you need to specify `null` default and why not use. constant `HCAT_DYNAMIC_CUSTOM_PATTERN`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5591:
URL: https://github.com/apache/hive/pull/5591#discussion_r2082270285
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/events/filesystem/FSTableEvent.java:
##########
@@ -179,7 +179,7 @@ private AlterTableAddPartitionDesc addPartitionDesc(Path fromPath, ImportTableDe
sd.getBucketCols(), sd.getSortCols(), columnStatistics, writeId);
AlterTableAddPartitionDesc addPartitionDesc = new AlterTableAddPartitionDesc(tblDesc.getDatabaseName(),
- tblDesc.getTableName(), true, ImmutableList.of(partitionDesc));
+ tblDesc.getTableName(), true, ImmutableList.of(partitionDesc), hiveConf.get("hcat.dynamic.partitioning.custom.pattern"));
Review Comment:
same, use HiveConf var
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "VenkatSNarayanan (via GitHub)" <gi...@apache.org>.
VenkatSNarayanan commented on PR #5591:
URL: https://github.com/apache/hive/pull/5591#issuecomment-2605753350
> @VenkatSNarayanan Please check the failed tests. https://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-5591/2/tests. Thx
I can't seem to access that page, do I need to do something to see it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #5591:
URL: https://github.com/apache/hive/pull/5591#issuecomment-2746242228
few tests are failing on Jenkins
````
org.apache.hadoop.hive.metastore.api.MetastoreException: org.apache.hadoop.hive.metastore.api.MetastoreException: MSCK is missing partition columns under pfile:/home/jenkins/agent/workspace/hive-precommit_PR-5591/ql/target/warehouse/testhivemetastorechecker_db.db/testhivemetastorechecker_table/partdate=2008-01-02/partcity=stockholm
at org.apache.hadoop.hive.metastore.HiveMetaStoreChecker.checkPartitionDirs(HiveMetaStoreChecker.java:726)
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5591:
URL: https://github.com/apache/hive/pull/5591#discussion_r2082288994
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java:
##########
@@ -414,7 +414,7 @@ private static AlterTableAddPartitionDesc getBaseAddPartitionDescFromPartition(P
partitionSpec, location, partition.getParameters(), sd.getInputFormat(), sd.getOutputFormat(),
sd.getNumBuckets(), sd.getCols(), sd.getSerdeInfo().getSerializationLib(), sd.getSerdeInfo().getParameters(),
sd.getBucketCols(), sd.getSortCols(), null, writeId);
- return new AlterTableAddPartitionDesc(dbName, tblDesc.getTableName(), true, ImmutableList.of(partitionDesc));
+ return new AlterTableAddPartitionDesc(dbName, tblDesc.getTableName(), true, ImmutableList.of(partitionDesc), conf.get("hcat.dynamic.partitioning.custom.pattern", null));
Review Comment:
use constant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5591:
URL: https://github.com/apache/hive/pull/5591#discussion_r2082260133
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/add/AbstractAddPartitionAnalyzer.java:
##########
@@ -70,7 +70,7 @@ protected void analyzeCommand(TableName tableName, Map<String, String> partition
}
AlterTableAddPartitionDesc desc = new AlterTableAddPartitionDesc(table.getDbName(), table.getTableName(),
- ifNotExists, partitions);
+ ifNotExists, partitions, conf.get("hcat.dynamic.partitioning.custom.pattern", null));
Review Comment:
why do you need to specify `null` default and not use constant (shouldn't be hcat specific)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5591:
URL: https://github.com/apache/hive/pull/5591#discussion_r2082319531
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java:
##########
@@ -1598,39 +1599,62 @@ public static Partition getPartition(IMetaStoreClient msc, Table tbl, Map<String
* @return Partition name, for example partitiondate=2008-01-01
*/
public static String getPartitionName(Path tablePath, Path partitionPath, Set<String> partCols,
- Map<String, String> partitionColToTypeMap) {
+ Map<String, String> partitionColToTypeMap, String customPattern) throws MetastoreException {
String result = null;
Path currPath = partitionPath;
LOG.debug("tablePath:" + tablePath + ", partCols: " + partCols);
+ if (customPattern != null) {
+ DynamicPartitioningCustomPattern compiledCustomPattern = new DynamicPartitioningCustomPattern.Builder()
+ .setCustomPattern(customPattern)
+ .build();
+ Pattern customPathPattern = compiledCustomPattern.getPartitionCapturePattern();
+ List<String> patternPartCols = compiledCustomPattern.getPartitionColumns(); //partition columns in order that they appear in the pattern
+ String relPath = partitionPath.toString().substring(tablePath.toString().length() + 1); //start after tablepath and the / afterwards
+ Matcher pathMatcher = customPathPattern.matcher(relPath);
+ boolean didMatch = pathMatcher.matches();
Review Comment:
please remove local var and inline
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "VenkatSNarayanan (via GitHub)" <gi...@apache.org>.
VenkatSNarayanan commented on PR #5591:
URL: https://github.com/apache/hive/pull/5591#issuecomment-2860576727
@deniskuzZ Fixed the MSCK test failures. There is an Iceberg test failure but it seems unrelated to the changes I've made (it complains about some output being too long).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5591:
URL: https://github.com/apache/hive/pull/5591#discussion_r2082322365
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java:
##########
@@ -1729,4 +1753,68 @@ public static List<String> getColumnNames(List<FieldSchema> cols) {
public static boolean isCompactionTxn(TxnType txnType) {
return TxnType.COMPACTION.equals(txnType) || TxnType.REBALANCE_COMPACTION.equals(txnType);
}
+
+ /**
+ * When using msck repair table with custom partitioning patterns, we need to capture
+ * the partition keys from the pattern, and use those to construct a regex which will
+ * match the paths and extract the partition key values.*/
+ public static class DynamicPartitioningCustomPattern {
+
+ private final String customPattern;
+ private final Pattern partitionCapturePattern;
+ private final List<String> partitionColumns;
+
+ private DynamicPartitioningCustomPattern(String customPattern, Pattern partitionCapturePattern, List<String> partitionColumns) {
+ this.customPattern = customPattern;
+ this.partitionCapturePattern = partitionCapturePattern;
+ this.partitionColumns = partitionColumns;
+ }
+
+ /**
+ * @return stored custom pattern string
+ * */
+ public String getCustomPattern() {
+ return customPattern;
+ }
+
+ /**
+ * @return stored custom pattern regex matcher
+ * */
+ public Pattern getPartitionCapturePattern() {
+ return partitionCapturePattern;
+ }
+
+ /**
+ * @return list of partition key columns
+ * */
+ public List<String> getPartitionColumns() {
+ return partitionColumns;
+ }
+
+ public static class Builder {
+ private String customPattern;
+
+ public Builder setCustomPattern(String customPattern) {
+ this.customPattern = customPattern;
+ return this;
+ }
+
+ /**
+ * Constructs the regex to match the partition values in a path based on the custom pattern.
+ *
+ * @return custom partition pattern matcher */
+ public DynamicPartitioningCustomPattern build() {
+ Pattern stringPattern = Pattern.compile("(\\$\\{)([^\\s/\\{\\}\\\\]+)(\\})");
Review Comment:
move to constant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5591:
URL: https://github.com/apache/hive/pull/5591#discussion_r2082316741
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java:
##########
@@ -678,7 +710,7 @@ void checkPartitionDirs(final ExecutorService executor,
//process each level in parallel
while(!nextLevel.isEmpty()) {
futures.add(
- executor.submit(new PathDepthInfoCallable(nextLevel.poll(), partColNames, fs, tempQueue)));
+ executor.submit(new PathDepthInfoCallable(nextLevel.poll(), partColNames, fs, tempQueue, conf.get("hcat.dynamic.partitioning.custom.pattern"))));
Review Comment:
use constant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5591:
URL: https://github.com/apache/hive/pull/5591#issuecomment-3050666735
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on PR #5591:
URL: https://github.com/apache/hive/pull/5591#issuecomment-2591547813
@VenkatSNarayanan Please check the failed tests. https://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-5591/2/tests. Thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5591:
URL: https://github.com/apache/hive/pull/5591#discussion_r2082270285
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/events/filesystem/FSTableEvent.java:
##########
@@ -179,7 +179,7 @@ private AlterTableAddPartitionDesc addPartitionDesc(Path fromPath, ImportTableDe
sd.getBucketCols(), sd.getSortCols(), columnStatistics, writeId);
AlterTableAddPartitionDesc addPartitionDesc = new AlterTableAddPartitionDesc(tblDesc.getDatabaseName(),
- tblDesc.getTableName(), true, ImmutableList.of(partitionDesc));
+ tblDesc.getTableName(), true, ImmutableList.of(partitionDesc), hiveConf.get("hcat.dynamic.partitioning.custom.pattern"));
Review Comment:
same, use. constant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5591:
URL: https://github.com/apache/hive/pull/5591#discussion_r2082294933
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java:
##########
@@ -445,7 +447,7 @@ void findUnknownPartitions(Table table, Set<Path> missingPartDirs,
for (Path partPath : missingPartDirs) {
FileSystem fs = partPath.getFileSystem(conf);
String partitionName = getPartitionName(fs.makeQualified(tablePath),
- partPath, partColNames, partitionColToTypeMap);
+ partPath, partColNames, partitionColToTypeMap, conf.get("hcat.dynamic.partitioning.custom.pattern"));
Review Comment:
use HiveConf var
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #5591: HIVE-28676: Support custom partition patterns in MSCK repair table
URL: https://github.com/apache/hive/pull/5591
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28676: Support custom partition patterns in MSCK repair table [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #5591:
URL: https://github.com/apache/hive/pull/5591#issuecomment-3077586480
hi @VenkatSNarayanan, please address the review comments and re-open the PR if you are still interested to get this in. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org