You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "difin (via GitHub)" <gi...@apache.org> on 2024/03/08 17:03:32 UTC
[PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
difin opened a new pull request, #5123:
URL: https://github.com/apache/hive/pull/5123
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Adding support for compacting a specific Iceberg partition
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Until now it was possible to only compact whole the table at once. If a table is partitioned, it would compact all partitions.
Now adding a new feature to compact a single specified partition.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No
### Is the change a dependency upgrade?
<!--
If yes, please attach a file with output from mvn dependency:tree to validate a complete upgrade of dependency.
If no, write 'No'.
-->
No
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
New q-test was added that compacts a partitioned table partition-by-partition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-1996200651
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [9 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2050724931
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [19 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1532288946
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##########
@@ -544,14 +549,20 @@ private void commitWrite(Table table, String branchName, Long snapshotId, long s
* @param rewritePolicy The rewrite policy to use for the insert overwrite commit
*/
private void commitOverwrite(Table table, String branchName, long startTime, FilesForCommit results,
- RewritePolicy rewritePolicy) {
+ RewritePolicy rewritePolicy, String partitionSpec) {
Preconditions.checkArgument(results.deleteFiles().isEmpty(), "Can not handle deletes with overwrite");
if (!results.dataFiles().isEmpty()) {
Transaction transaction = table.newTransaction();
if (rewritePolicy == RewritePolicy.ALL_PARTITIONS) {
DeleteFiles delete = transaction.newDelete();
delete.deleteFromRowFilter(Expressions.alwaysTrue());
delete.commit();
+ } else if (rewritePolicy == RewritePolicy.SINGLE_PARTITION) {
Review Comment:
can't we use the DEFAULT? rewrite policy enum is either all or not all (custom set)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1532302493
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -1110,8 +1110,9 @@ public void validateSinkDesc(FileSinkDesc sinkDesc) throws SemanticException {
// If the table is empty we don't have any danger that some data can get lost.
return;
}
- if (RewritePolicy.fromString(conf.get(ConfVars.REWRITE_POLICY.varname, RewritePolicy.DEFAULT.name())) ==
- RewritePolicy.ALL_PARTITIONS) {
+ RewritePolicy rewritePolicy = RewritePolicy.fromString(conf.get(ConfVars.REWRITE_POLICY.varname,
+ RewritePolicy.DEFAULT.name()));
+ if (rewritePolicy == RewritePolicy.ALL_PARTITIONS || rewritePolicy == RewritePolicy.SINGLE_PARTITION) {
Review Comment:
do we need this check since we add support for partition-level IOW?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2053761766
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [17 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1575777876
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -1915,6 +1917,46 @@ private boolean hasUndergonePartitionEvolution(Table table) {
.anyMatch(id -> id < table.spec().specId());
}
+ private boolean isIdentityPartitionTable(org.apache.hadoop.hive.ql.metadata.Table table) {
+ return getPartitionTransformSpec(table).stream().map(TransformSpec::getTransformType)
+ .allMatch(type -> type == TransformSpec.TransformType.IDENTITY);
+ }
+
+ @Override
+ public org.apache.commons.lang3.tuple.Pair<Boolean, ErrorMsg> isEligibleForCompaction(
+ org.apache.hadoop.hive.ql.metadata.Table table, Map<String, String> partitionSpec) {
+ if (partitionSpec != null) {
+ Table icebergTable = IcebergTableUtil.getTable(conf, table.getTTable());
+ if (hasUndergonePartitionEvolution(icebergTable)) {
+ return org.apache.commons.lang3.tuple.Pair.of(false, ErrorMsg.COMPACTION_PARTITION_EVOLUTION);
+ }
+ if (!isIdentityPartitionTable(table)) {
+ return org.apache.commons.lang3.tuple.Pair.of(false, ErrorMsg.COMPACTION_NON_IDENTITY_PARTITION_SPEC);
+ }
+ }
+ return org.apache.commons.lang3.tuple.Pair.of(true, null);
+ }
+
+ @Override
+ public List<Partition> getPartitions(org.apache.hadoop.hive.ql.metadata.Table table,
+ Map<String, String> partitionSpec, short limit) throws SemanticException {
+ return table.getStorageHandler().getPartitionNames(table, partitionSpec).stream()
Review Comment:
why do you need `table.getStorageHandler()`, call `getPartitionNames` directly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2065596782
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [10 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1576421625
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +55,71 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ Table table = IcebergTableUtil.getTable(conf, context.getTable());
+ List<String> partFields = table.spec().fields().stream().map(PartitionField::name).collect(Collectors.toList());
+ String queryFields = table.schema().columns().stream().map(Types.NestedField::name)
+ .filter(f -> !partFields.contains(f)).collect(Collectors.joining(","));
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select %4$s from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "), queryFields);
+ }
+
+ SessionState sessionState = setupQueryCompactionSession(conf, context.getCompactionInfo(), tblProperties);
- SessionState sessionState = setupQueryCompactionSession(context.getConf(),
- context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
try {
- DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
+ DriverUtils.runOnDriver(conf, sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
+ return true;
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
- throw new RuntimeException(e);
+ LOG.error("Failed compacting table {}", compactTableName, e);
+ throw e;
} finally {
sessionState.setCompaction(false);
}
+ }
+
+ private String partDataToSQL(PartitionData partitionData, String partSpec, String delimiter)
+ throws HiveException {
+
+ try {
+ Map<String, String> partSpecMap = Warehouse.makeSpecFromName(partSpec);
+ StringBuilder sb = new StringBuilder();
+
+ for (int i = 0; i < partitionData.size(); ++i) {
+ if (i > 0) {
+ sb.append(delimiter);
+ }
+
+ String quoteOpt = "";
+ if (partitionData.getType(i).typeId() == Type.TypeID.STRING ||
+ partitionData.getType(i).typeId() == Type.TypeID.DATE ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIME ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIMESTAMP ||
+ partitionData.getType(i).typeId() == Type.TypeID.BINARY) {
+ quoteOpt = "'";
+ }
- return true;
+ String fieldName = partitionData.getSchema().getFields().get(i).name();
+
+ sb.append(fieldName)
+ .append("=")
+ .append(quoteOpt)
+ .append(partSpecMap.get(fieldName))
+ .append(quoteOpt);
Review Comment:
Thanks, done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1571017320
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/compact/AlterTableCompactOperation.java:
##########
@@ -62,6 +65,12 @@ public AlterTableCompactOperation(DDLOperationContext context, AlterTableCompact
throw new HiveException(ErrorMsg.NONACID_COMPACTION_NOT_SUPPORTED, table.getDbName(), table.getTableName());
}
+ if (desc.getPartitionSpec() != null && DDLUtils.isIcebergTable(table) &&
Review Comment:
could we move this logic into AlterTableUtils?
````
public boolean isIdentityPartitionSpec(table) {
return table.getStorageHandler() != null && table.getStorageHandler().getPartitionTransformSpec(table).stream()
.map(TransformSpec::getTransformType)
.allMatch(type -> type == TransformSpec.TransformType.IDENTITY);
````
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1576402233
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -1915,6 +1917,46 @@ private boolean hasUndergonePartitionEvolution(Table table) {
.anyMatch(id -> id < table.spec().specId());
}
+ private boolean isIdentityPartitionTable(org.apache.hadoop.hive.ql.metadata.Table table) {
+ return getPartitionTransformSpec(table).stream().map(TransformSpec::getTransformType)
+ .allMatch(type -> type == TransformSpec.TransformType.IDENTITY);
+ }
+
+ @Override
+ public org.apache.commons.lang3.tuple.Pair<Boolean, ErrorMsg> isEligibleForCompaction(
+ org.apache.hadoop.hive.ql.metadata.Table table, Map<String, String> partitionSpec) {
+ if (partitionSpec != null) {
+ Table icebergTable = IcebergTableUtil.getTable(conf, table.getTTable());
+ if (hasUndergonePartitionEvolution(icebergTable)) {
+ return org.apache.commons.lang3.tuple.Pair.of(false, ErrorMsg.COMPACTION_PARTITION_EVOLUTION);
+ }
+ if (!isIdentityPartitionTable(table)) {
+ return org.apache.commons.lang3.tuple.Pair.of(false, ErrorMsg.COMPACTION_NON_IDENTITY_PARTITION_SPEC);
+ }
+ }
+ return org.apache.commons.lang3.tuple.Pair.of(true, null);
+ }
+
+ @Override
+ public List<Partition> getPartitions(org.apache.hadoop.hive.ql.metadata.Table table,
+ Map<String, String> partitionSpec, short limit) throws SemanticException {
+ return table.getStorageHandler().getPartitionNames(table, partitionSpec).stream()
+ .limit(limit > 0 ? limit : Long.MAX_VALUE)
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1577440548
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -777,6 +783,16 @@ default List<Partition> getPartitionsByExpr(org.apache.hadoop.hive.ql.metadata.T
"for a table.");
}
+ default List<Partition> getPartitions(org.apache.hadoop.hive.ql.metadata.Table table,
+ Map<String, String> partitionSpec) throws SemanticException {
+ throw new UnsupportedOperationException("Storage handler does not support getting partitions for a table.");
+ }
+
+ default Partition getPartition(org.apache.hadoop.hive.ql.metadata.Table table, Map<String, String> partSpec)
Review Comment:
`partitionSpec`
please keep param names consistent, that is an interface
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1576852118
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -777,6 +783,16 @@ default List<Partition> getPartitionsByExpr(org.apache.hadoop.hive.ql.metadata.T
"for a table.");
}
+ default List<Partition> getPartitions(org.apache.hadoop.hive.ql.metadata.Table table,
+ Map<String, String> partialPartSpec) throws SemanticException {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1575083584
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/PartitionUtils.java:
##########
@@ -94,10 +98,20 @@ public static ExprNodeGenericFuncDesc makeUnaryPredicate(String fn, ExprNodeDesc
public static Partition getPartition(Hive db, Table table, Map<String, String> partitionSpec, boolean throwException)
throws SemanticException {
Partition partition;
- try {
- partition = db.getPartition(table, partitionSpec, false);
- } catch (Exception e) {
- throw new SemanticException(toMessage(ErrorMsg.INVALID_PARTITION, partitionSpec), e);
+ if (table.getStorageHandler() != null && table.getStorageHandler().alwaysUnpartitioned()) {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1567652089
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +52,68 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.DEFAULT.name());
+ Table table = IcebergTableUtil.getTable(context.getConf(), context.getTable());
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ context.getConf().set(Context.compactPartitionSpec, partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select * from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "));
+ }
SessionState sessionState = setupQueryCompactionSession(context.getConf(),
context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+
try {
DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
+ return true;
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
- throw new RuntimeException(e);
+ LOG.error("Failed compacting table {}", compactTableName, e);
+ throw e;
} finally {
sessionState.setCompaction(false);
}
+ }
+
+ private String partDataToSQL(PartitionData partitionData, String partSpec, String delimiter)
Review Comment:
Done
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +52,68 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.DEFAULT.name());
+ Table table = IcebergTableUtil.getTable(context.getConf(), context.getTable());
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ context.getConf().set(Context.compactPartitionSpec, partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select * from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "));
+ }
SessionState sessionState = setupQueryCompactionSession(context.getConf(),
context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+
try {
DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
+ return true;
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
- throw new RuntimeException(e);
+ LOG.error("Failed compacting table {}", compactTableName, e);
+ throw e;
} finally {
sessionState.setCompaction(false);
}
+ }
+
+ private String partDataToSQL(PartitionData partitionData, String partSpec, String delimiter)
+ throws HiveException {
- return true;
+ try {
+ Map<String, String> partSpecMap = Warehouse.makeSpecFromName(partSpec);
+ StringBuilder sb = new StringBuilder();
+
+ for (int i = 0; i < partitionData.size(); ++i) {
+ if (i > 0) {
+ sb.append(delimiter);
+ }
+
+ String quoteOpt = "";
+ if (partitionData.getType(i).typeId() == Type.TypeID.STRING ||
+ partitionData.getType(i).typeId() == Type.TypeID.DATE ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIME ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIMESTAMP ||
+ partitionData.getType(i).typeId() == Type.TypeID.BINARY) {
+ quoteOpt = "'";
+ }
+
+ sb.append(partitionData.getSchema().getFields().get(i).name())
+ .append("=")
+ .append(quoteOpt)
+ .append(partSpecMap.get(partitionData.getPartitionType().fields().get(i).name()))
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1567652570
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##########
@@ -569,14 +575,25 @@ private void commitWrite(Table table, String branchName, Long snapshotId, long s
* @param rewritePolicy The rewrite policy to use for the insert overwrite commit
*/
private void commitOverwrite(Table table, String branchName, long startTime, FilesForCommit results,
- RewritePolicy rewritePolicy) {
+ RewritePolicy rewritePolicy, String partitionSpec) {
Preconditions.checkArgument(results.deleteFiles().isEmpty(), "Can not handle deletes with overwrite");
if (!results.dataFiles().isEmpty()) {
Transaction transaction = table.newTransaction();
if (rewritePolicy == RewritePolicy.ALL_PARTITIONS) {
DeleteFiles delete = transaction.newDelete();
delete.deleteFromRowFilter(Expressions.alwaysTrue());
delete.commit();
+ LOG.info("Overwrite commit: truncated all partitions for table: {} with {} file(s)",
+ table, results.dataFiles().size());
+ } else if (!partitionSpec.isEmpty()) {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1565823941
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##########
@@ -569,14 +575,25 @@ private void commitWrite(Table table, String branchName, Long snapshotId, long s
* @param rewritePolicy The rewrite policy to use for the insert overwrite commit
*/
private void commitOverwrite(Table table, String branchName, long startTime, FilesForCommit results,
- RewritePolicy rewritePolicy) {
+ RewritePolicy rewritePolicy, String partitionSpec) {
Preconditions.checkArgument(results.deleteFiles().isEmpty(), "Can not handle deletes with overwrite");
if (!results.dataFiles().isEmpty()) {
Transaction transaction = table.newTransaction();
if (rewritePolicy == RewritePolicy.ALL_PARTITIONS) {
DeleteFiles delete = transaction.newDelete();
delete.deleteFromRowFilter(Expressions.alwaysTrue());
delete.commit();
+ LOG.info("Overwrite commit: truncated all partitions for table: {} with {} file(s)",
+ table, results.dataFiles().size());
+ } else if (!partitionSpec.isEmpty()) {
Review Comment:
This is not for partition evolution case. This is for compaction on partition level without evolution. The partition needs to be truncated before adding compacted files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1568989334
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +51,69 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ Table table = IcebergTableUtil.getTable(conf, context.getTable());
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select * from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "));
+ }
+
+ SessionState sessionState = setupQueryCompactionSession(conf, context.getCompactionInfo(), tblProperties);
+ sessionState.setConf(conf);
Review Comment:
When the cloned conf is passed to the session, and the session is not null, the conf is not set in the session and any changes defined in it get lost. I found it during debugging when trying to understand why in a q-test that has 2 compaction calls the second compaction command is not working.
```
public static SessionState setUpSessionState(HiveConf conf, String user, boolean doStart) {
SessionState sessionState = SessionState.get();
if (sessionState == null) {
// Note: we assume that workers run on the same threads repeatedly, so we can set up
// the session here and it will be reused without explicitly storing in the worker.
sessionState = new SessionState(conf, user);
if (doStart) {
// TODO: Required due to SessionState.getHDFSSessionPath. Why wasn't it required before?
sessionState.setIsHiveServerQuery(true);
SessionState.start(sessionState);
}
SessionState.setCurrentSessionState(sessionState);
}
return sessionState;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1569002561
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/PartitionUtils.java:
##########
@@ -94,10 +97,21 @@ public static ExprNodeGenericFuncDesc makeUnaryPredicate(String fn, ExprNodeDesc
public static Partition getPartition(Hive db, Table table, Map<String, String> partitionSpec, boolean throwException)
throws SemanticException {
Partition partition;
- try {
- partition = db.getPartition(table, partitionSpec, false);
- } catch (Exception e) {
- throw new SemanticException(toMessage(ErrorMsg.INVALID_PARTITION, partitionSpec), e);
+ if (table.getStorageHandler() != null && table.getStorageHandler().alwaysUnpartitioned()) {
+ table.getStorageHandler().validatePartSpec(table, partitionSpec);
+ try {
+ String partName = Warehouse.makePartName(partitionSpec, false);
+ partition = new DummyPartition(table, partName, partitionSpec);
+ } catch (MetaException e) {
+ throw new SemanticException("Unable to construct name for dummy partition due to: ", e);
+ }
+ }
+ else {
Review Comment:
please move it to prev line with `}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1533045609
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##########
@@ -8810,7 +8810,8 @@ private Operator genConversionSelectOperatorByAddPartition(String dest, QB qb, O
List<ExprNodeDesc> expressions = new ArrayList<>(outColumnCnt);
AtomicBoolean convert = new AtomicBoolean(false);
- if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt) {
+ if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
Review Comment:
I modified the check for Iceberg case as following, is it fine now?
```
if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
((table.getStorageHandler() != null && table.getStorageHandler().alwaysUnpartitioned() && inColumnCnt != outColumnCnt) ||
!(table.getStorageHandler() != null && table.getStorageHandler().alwaysUnpartitioned()))) {
String reason = "Table " + dest + " has " + outColumnCnt
+ " columns, but query has " + (inColumnCnt + staticPartitionColumnCnt) + " columns.";
throw new SemanticException(ASTErrorUtils.getMsg(
ErrorMsg.TARGET_TABLE_COLUMN_MISMATCH.getMsg(),
qb.getParseInfo().getDestForClause(dest), reason));
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2016351860
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [19 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2032798526
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [16 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1570827056
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/compact/AlterTableCompactOperation.java:
##########
@@ -133,7 +140,7 @@ private List<Partition> getPartitions(Table table, AlterTableCompactDesc desc, D
}
} else {
Map<String, String> partitionSpec = desc.getPartitionSpec();
- partitions = context.getDb().getPartitions(table, partitionSpec);
+ partitions.addAll(PartitionUtils.getPartitions(context.getDb(), table, partitionSpec, false));
Review Comment:
is that used for the scenarios like below, when we provide partial spec?
````
/country=Ukraine/city=Kharkiv
/country=Ukraine/city=Kyiv
````
alter table T partition(country=Ukraine) compact 'MAJOR'
how would we handle this in Iceberg ATM?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1572020032
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##########
@@ -8815,6 +8815,23 @@ These props are now enabled elsewhere (see commit diffs). It would be better in
conf.set(AcidUtils.CONF_ACID_KEY, "true");
SessionState.get().getConf().set(AcidUtils.CONF_ACID_KEY, "true");
}
+
+ private void validateColumnCounts(String dest, Table table, Map<String, String> partitionSpec,
+ int inColumnCnt, int outColumnCnt) throws SemanticException {
+
+ long staticPartitionColumnCnt = partitionSpec != null ?
+ partitionSpec.values().stream().filter(Objects::nonNull).count() : 0;
+
+ if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
Review Comment:
if it could be `inColumnCnt + staticPartitionColumnCnt == outColumnCnt where staticPartitionColumnCnt is > 0 and in other cases it can be inColumnCnt == outColumnCnt` why do we even need to handle iceberg differently?
if(inColumnCnt + staticPartitionColumnCnt != outColumnCnt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1568961100
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +51,69 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ Table table = IcebergTableUtil.getTable(conf, context.getTable());
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select * from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "));
+ }
+
+ SessionState sessionState = setupQueryCompactionSession(conf, context.getCompactionInfo(), tblProperties);
+ sessionState.setConf(conf);
Review Comment:
check MinorQueryCompactor for example, it just clones the conf and passes that to session
````
HiveConf conf = new HiveConf(hiveConf);
conf.set(HiveConf.ConfVars.SPLIT_GROUPING_MODE.varname, CompactorUtil.COMPACTOR);
conf.setBoolVar(HiveConf.ConfVars.HIVE_STATS_FETCH_COLUMN_STATS, false);
conf.setBoolVar(HiveConf.ConfVars.HIVE_STATS_ESTIMATE_STATS, false);
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1569497420
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -1110,8 +1110,9 @@ public void validateSinkDesc(FileSinkDesc sinkDesc) throws SemanticException {
// If the table is empty we don't have any danger that some data can get lost.
return;
}
- if (RewritePolicy.fromString(conf.get(ConfVars.REWRITE_POLICY.varname, RewritePolicy.DEFAULT.name())) ==
- RewritePolicy.ALL_PARTITIONS) {
+ RewritePolicy rewritePolicy = RewritePolicy.fromString(conf.get(ConfVars.REWRITE_POLICY.varname,
+ RewritePolicy.DEFAULT.name()));
+ if (rewritePolicy == RewritePolicy.ALL_PARTITIONS || rewritePolicy == RewritePolicy.SINGLE_PARTITION) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1572034783
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/compact/AlterTableCompactOperation.java:
##########
@@ -62,6 +65,17 @@ public AlterTableCompactOperation(DDLOperationContext context, AlterTableCompact
throw new HiveException(ErrorMsg.NONACID_COMPACTION_NOT_SUPPORTED, table.getDbName(), table.getTableName());
}
+ if (desc.getPartitionSpec() != null && !AlterTableUtils.isIdentityPartitionTable(table)) {
+ throw new HiveException("Compaction is not supported on the table " +
+ table.getDbName() + "." + table.getTableName() + " with non-identity partition spec");
+ }
+
+ if (desc.getPartitionSpec() != null && DDLUtils.isIcebergTable(table) &&
Review Comment:
i think would be better if we move checks under something like
````
AlterTableUtils.isEligibleForCompaction(table,desc.getPartitionSpec() ){
if (!AcidUtils.isTransactionalTable(table) && !AcidUtils.isNonNativeAcidTable(table)) {
throw new HiveException(ErrorMsg.NONACID_COMPACTION_NOT_SUPPORTED, table.getDbName(), table.getTableName());
}
if (desc.getPartitionSpec() != null && table.getStorageHandler() != null) {
if (isIdentityPartitionTable(table)) {
throw new HiveException("Compaction is not supported on the table " +
table.getDbName() + "." + table.getTableName() + " with non-identity partition spec");
}
if(hasUndergonePartitionEvolution(table)) {
throw new HiveException("Compaction on partition level is not supported on the table " +
table.getDbName() + "." + table.getTableName() + " which has undergone partition evolution");
}
}
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1572034783
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/compact/AlterTableCompactOperation.java:
##########
@@ -62,6 +65,17 @@ public AlterTableCompactOperation(DDLOperationContext context, AlterTableCompact
throw new HiveException(ErrorMsg.NONACID_COMPACTION_NOT_SUPPORTED, table.getDbName(), table.getTableName());
}
+ if (desc.getPartitionSpec() != null && !AlterTableUtils.isIdentityPartitionTable(table)) {
+ throw new HiveException("Compaction is not supported on the table " +
+ table.getDbName() + "." + table.getTableName() + " with non-identity partition spec");
+ }
+
+ if (desc.getPartitionSpec() != null && DDLUtils.isIcebergTable(table) &&
Review Comment:
i think would be better if we move checks under something like
AlterTableUtils.isEligibleForCompaction(table,desc.getPartitionSpec() ){
if (!AcidUtils.isTransactionalTable(table) && !AcidUtils.isNonNativeAcidTable(table)) {
throw new HiveException(ErrorMsg.NONACID_COMPACTION_NOT_SUPPORTED, table.getDbName(), table.getTableName());
}
if (desc.getPartitionSpec() != null && table.getStorageHandler() != null) {
if (isIdentityPartitionTable(table)) {
throw new HiveException("Compaction is not supported on the table " +
table.getDbName() + "." + table.getTableName() + " with non-identity partition spec");
}
if(hasUndergonePartitionEvolution(table)) {
throw new HiveException("Compaction on partition level is not supported on the table " +
table.getDbName() + "." + table.getTableName() + " which has undergone partition evolution");
}
}
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1574882384
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java:
##########
@@ -782,7 +782,7 @@ private void setCommonHmsTablePropertiesForIceberg(org.apache.hadoop.hive.metast
* @param hmsTable Table for which we are calculating the properties
* @return The properties we can provide for Iceberg functions, like {@link Catalogs}
*/
- private static Properties getCatalogProperties(org.apache.hadoop.hive.metastore.api.Table hmsTable) {
+ public static Properties getCatalogProperties(org.apache.hadoop.hive.metastore.api.Table hmsTable) {
Review Comment:
Fixed. Was needed before, but not anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1575838382
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +55,71 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ Table table = IcebergTableUtil.getTable(conf, context.getTable());
+ List<String> partFields = table.spec().fields().stream().map(PartitionField::name).collect(Collectors.toList());
+ String queryFields = table.schema().columns().stream().map(Types.NestedField::name)
+ .filter(f -> !partFields.contains(f)).collect(Collectors.joining(","));
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select %4$s from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "), queryFields);
+ }
+
+ SessionState sessionState = setupQueryCompactionSession(conf, context.getCompactionInfo(), tblProperties);
- SessionState sessionState = setupQueryCompactionSession(context.getConf(),
- context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
try {
- DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
+ DriverUtils.runOnDriver(conf, sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
+ return true;
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
- throw new RuntimeException(e);
+ LOG.error("Failed compacting table {}", compactTableName, e);
+ throw e;
} finally {
sessionState.setCompaction(false);
}
+ }
+
+ private String partDataToSQL(PartitionData partitionData, String partSpec, String delimiter)
+ throws HiveException {
+
+ try {
+ Map<String, String> partSpecMap = Warehouse.makeSpecFromName(partSpec);
+ StringBuilder sb = new StringBuilder();
+
+ for (int i = 0; i < partitionData.size(); ++i) {
+ if (i > 0) {
+ sb.append(delimiter);
+ }
+
+ String quoteOpt = "";
+ if (partitionData.getType(i).typeId() == Type.TypeID.STRING ||
+ partitionData.getType(i).typeId() == Type.TypeID.DATE ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIME ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIMESTAMP ||
+ partitionData.getType(i).typeId() == Type.TypeID.BINARY) {
+ quoteOpt = "'";
+ }
- return true;
+ String fieldName = partitionData.getSchema().getFields().get(i).name();
+
+ sb.append(fieldName)
+ .append("=")
+ .append(quoteOpt)
+ .append(partSpecMap.get(fieldName))
+ .append(quoteOpt);
Review Comment:
how about
````
Map<String, String> partSpecMap = Warehouse.makeSpecFromName(partSpec);
List<FieldSchema> partitionKeys = hiveTable.getStorageHandler().getPartitionKeys(hiveTable);
List<String> partValues = partitionKeys.stream().map(
fs -> String.join("=", HiveUtils.unparseIdentifier(fs.getName()),
TypeInfoUtils.convertStringToLiteralForSQL(partSpecMap.get(fs.getName()),
((PrimitiveTypeInfo) TypeInfoUtils.getTypeInfoFromTypeString(fs.getType())).getPrimitiveCategory())
)
).collect(Collectors.toList());
String queryFields = hiveTable.getCols().stream().filter(col -> !partitionKeys.contains(col))
.map(FieldSchema::getName)
.collect(Collectors.joining(","));
compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select %4$s from %1$s where %3$s",
compactTableName,
StringUtils.join(partValues, ","),
StringUtils.join(partValues, " and "),
queryFields);
````
table.getStorageHandler().getColumnInfo looks expensive and unnecessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1576721296
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -777,6 +783,16 @@ default List<Partition> getPartitionsByExpr(org.apache.hadoop.hive.ql.metadata.T
"for a table.");
}
+ default List<Partition> getPartitions(org.apache.hadoop.hive.ql.metadata.Table table,
+ Map<String, String> partialPartSpec) throws SemanticException {
Review Comment:
`partitionSpec`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2059679977
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [10 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1569000652
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +51,69 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ Table table = IcebergTableUtil.getTable(conf, context.getTable());
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select * from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "));
+ }
+
+ SessionState sessionState = setupQueryCompactionSession(conf, context.getCompactionInfo(), tblProperties);
+ sessionState.setConf(conf);
Review Comment:
````
else {
sessionState.setConf(conf);
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1568995126
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +51,69 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ Table table = IcebergTableUtil.getTable(conf, context.getTable());
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select * from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "));
+ }
+
+ SessionState sessionState = setupQueryCompactionSession(conf, context.getCompactionInfo(), tblProperties);
+ sessionState.setConf(conf);
Review Comment:
how does it work then for ACID compactions? shouldn't we fix that in setUpSessionState and always update the conf?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1571193903
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##########
@@ -8815,6 +8815,23 @@ These props are now enabled elsewhere (see commit diffs). It would be better in
conf.set(AcidUtils.CONF_ACID_KEY, "true");
SessionState.get().getConf().set(AcidUtils.CONF_ACID_KEY, "true");
}
+
+ private void validateColumnCounts(String dest, Table table, Map<String, String> partitionSpec,
+ int inColumnCnt, int outColumnCnt) throws SemanticException {
+
+ long staticPartitionColumnCnt = partitionSpec != null ?
+ partitionSpec.values().stream().filter(Objects::nonNull).count() : 0;
+
+ if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
Review Comment:
Extracted the code into a local var, but about the second part, unfortunately it won't work because I saw that for Iceberg it can be `inColumnCnt + staticPartitionColumnCnt == outColumnCnt` or` inColumnCnt == outColumnCnt` and the suggested expression won't cover the first case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2047933984
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [14 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1561219672
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##########
@@ -485,7 +485,7 @@ private void commitTable(FileIO io, ExecutorService executor, OutputTable output
}
}
- private Long getSnapshotId(Table table, String branchName) {
+ public static Long getSnapshotId(Table table, String branchName) {
Review Comment:
maybe move it to `IcebergTableUtils`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1561232301
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -21,45 +21,159 @@
import java.io.IOException;
import java.util.Map;
+import jline.internal.Log;
import org.apache.hadoop.hive.conf.HiveConf;
import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.metastore.Warehouse;
+import org.apache.hadoop.hive.metastore.api.MetaException;
import org.apache.hadoop.hive.ql.Context.RewritePolicy;
import org.apache.hadoop.hive.ql.DriverUtils;
import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.parse.AlterTableSnapshotRefSpec;
import org.apache.hadoop.hive.ql.session.SessionState;
import org.apache.hadoop.hive.ql.txn.compactor.CompactorContext;
import org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor;
import org.apache.hive.iceberg.org.apache.orc.storage.common.TableName;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.PartitionData;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter;
+import org.apache.iceberg.mr.hive.IcebergBranchExec;
+import org.apache.iceberg.mr.hive.IcebergTableUtil;
+import org.apache.iceberg.types.Type;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class IcebergMajorQueryCompactor extends QueryCompactor {
private static final Logger LOG = LoggerFactory.getLogger(IcebergMajorQueryCompactor.class.getName());
+ private static final String branchName = "compaction";
+ private String compactTableName;
@Override
public boolean run(CompactorContext context) throws IOException, HiveException, InterruptedException {
- String compactTableName = TableName.getDbTable(context.getTable().getDbName(), context.getTable().getTableName());
+ compactTableName = TableName.getDbTable(context.getTable().getDbName(), context.getTable().getTableName());
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
SessionState sessionState = setupQueryCompactionSession(context.getConf(),
context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+
try {
- DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
- LOG.info("Completed compaction for table {}", compactTableName);
- } catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
- throw new RuntimeException(e);
+ if (partSpec == null) {
+ HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+
+ executeHiveQuery(context, compactionQuery, sessionState,
+ String.format("compacting table %s", compactTableName));
+ } else {
+ Table table = IcebergTableUtil.getTable(context.getConf(), context.getTable());
+ AlterTableSnapshotRefSpec.DropSnapshotRefSpec dropBranchSpec =
+ new AlterTableSnapshotRefSpec.DropSnapshotRefSpec(branchName, true);
+ IcebergBranchExec.dropBranch(table, dropBranchSpec);
+
+ AlterTableSnapshotRefSpec.CreateSnapshotRefSpec createBranchSpec =
+ new AlterTableSnapshotRefSpec.CreateSnapshotRefSpec(branchName);
+ IcebergBranchExec.createBranch(table, createBranchSpec);
+ table.refresh();
+ Long createBranchSnapshotId = HiveIcebergOutputCommitter.getSnapshotId(table, "branch_" + branchName);
+
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ String truncateQuery = String.format("truncate table %s.branch_%s partition(%s)",
+ compactTableName, branchName, partDataToSQL(partitionData, partSpec, ","));
+ executeHiveQuery(context, truncateQuery, sessionState,
+ String.format("truncating partition %s of table %s.branch_%s", partSpec, compactTableName,
+ branchName));
+ table.refresh();
Review Comment:
why is that needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-1990909679
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [16 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1575838382
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +55,71 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ Table table = IcebergTableUtil.getTable(conf, context.getTable());
+ List<String> partFields = table.spec().fields().stream().map(PartitionField::name).collect(Collectors.toList());
+ String queryFields = table.schema().columns().stream().map(Types.NestedField::name)
+ .filter(f -> !partFields.contains(f)).collect(Collectors.joining(","));
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select %4$s from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "), queryFields);
+ }
+
+ SessionState sessionState = setupQueryCompactionSession(conf, context.getCompactionInfo(), tblProperties);
- SessionState sessionState = setupQueryCompactionSession(context.getConf(),
- context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
try {
- DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
+ DriverUtils.runOnDriver(conf, sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
+ return true;
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
- throw new RuntimeException(e);
+ LOG.error("Failed compacting table {}", compactTableName, e);
+ throw e;
} finally {
sessionState.setCompaction(false);
}
+ }
+
+ private String partDataToSQL(PartitionData partitionData, String partSpec, String delimiter)
+ throws HiveException {
+
+ try {
+ Map<String, String> partSpecMap = Warehouse.makeSpecFromName(partSpec);
+ StringBuilder sb = new StringBuilder();
+
+ for (int i = 0; i < partitionData.size(); ++i) {
+ if (i > 0) {
+ sb.append(delimiter);
+ }
+
+ String quoteOpt = "";
+ if (partitionData.getType(i).typeId() == Type.TypeID.STRING ||
+ partitionData.getType(i).typeId() == Type.TypeID.DATE ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIME ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIMESTAMP ||
+ partitionData.getType(i).typeId() == Type.TypeID.BINARY) {
+ quoteOpt = "'";
+ }
- return true;
+ String fieldName = partitionData.getSchema().getFields().get(i).name();
+
+ sb.append(fieldName)
+ .append("=")
+ .append(quoteOpt)
+ .append(partSpecMap.get(fieldName))
+ .append(quoteOpt);
Review Comment:
how about
````
PrimitiveObjectInspector primitiveObjInspector = IcebergObjectInspector.primitive(partitionData.getType(i))
TypeInfoUtils.convertStringToLiteralForSQL(value, primitiveObjInspector.getPrimitiveCategory())
````
table.getStorageHandler().getColumnInfo looks expensive and unnecessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1575838382
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +55,71 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ Table table = IcebergTableUtil.getTable(conf, context.getTable());
+ List<String> partFields = table.spec().fields().stream().map(PartitionField::name).collect(Collectors.toList());
+ String queryFields = table.schema().columns().stream().map(Types.NestedField::name)
+ .filter(f -> !partFields.contains(f)).collect(Collectors.joining(","));
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select %4$s from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "), queryFields);
+ }
+
+ SessionState sessionState = setupQueryCompactionSession(conf, context.getCompactionInfo(), tblProperties);
- SessionState sessionState = setupQueryCompactionSession(context.getConf(),
- context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
try {
- DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
+ DriverUtils.runOnDriver(conf, sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
+ return true;
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
- throw new RuntimeException(e);
+ LOG.error("Failed compacting table {}", compactTableName, e);
+ throw e;
} finally {
sessionState.setCompaction(false);
}
+ }
+
+ private String partDataToSQL(PartitionData partitionData, String partSpec, String delimiter)
+ throws HiveException {
+
+ try {
+ Map<String, String> partSpecMap = Warehouse.makeSpecFromName(partSpec);
+ StringBuilder sb = new StringBuilder();
+
+ for (int i = 0; i < partitionData.size(); ++i) {
+ if (i > 0) {
+ sb.append(delimiter);
+ }
+
+ String quoteOpt = "";
+ if (partitionData.getType(i).typeId() == Type.TypeID.STRING ||
+ partitionData.getType(i).typeId() == Type.TypeID.DATE ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIME ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIMESTAMP ||
+ partitionData.getType(i).typeId() == Type.TypeID.BINARY) {
+ quoteOpt = "'";
+ }
- return true;
+ String fieldName = partitionData.getSchema().getFields().get(i).name();
+
+ sb.append(fieldName)
+ .append("=")
+ .append(quoteOpt)
+ .append(partSpecMap.get(fieldName))
+ .append(quoteOpt);
Review Comment:
how about
````
Map<String, String> partSpecMap = new LinkedHashMap<>();
Warehouse.makeSpecFromName(partSpecMap, new Path(partSpec), null);
List<FieldSchema> partitionKeys = hiveTable.getStorageHandler().getPartitionKeys(hiveTable);
List<String> partValues = partitionKeys.stream().map(
fs -> String.join("=", HiveUtils.unparseIdentifier(fs.getName()),
TypeInfoUtils.convertStringToLiteralForSQL(partSpecMap.get(fs.getName()),
((PrimitiveTypeInfo) TypeInfoUtils.getTypeInfoFromTypeString(fs.getType())).getPrimitiveCategory())
)
).collect(Collectors.toList());
String queryFields = hiveTable.getCols().stream().filter(col -> !partSpecMap.containsKey(col.getName()))
.map(FieldSchema::getName)
.collect(Collectors.joining(","));
compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select %4$s from %1$s where %3$s",
compactTableName,
StringUtils.join(partValues, ","),
StringUtils.join(partValues, " and "),
queryFields);
````
table.getStorageHandler().getColumnInfo looks expensive and unnecessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1575801527
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/PartitionUtils.java:
##########
@@ -95,7 +95,7 @@ public static Partition getPartition(Hive db, Table table, Map<String, String> p
throws SemanticException {
Partition partition;
try {
- partition = db.getPartition(table, partitionSpec, false);
+ partition = db.getPartition(table, partitionSpec);
Review Comment:
3rd param, `forceCreate` was omitted, why?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1533045609
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##########
@@ -8810,7 +8810,8 @@ private Operator genConversionSelectOperatorByAddPartition(String dest, QB qb, O
List<ExprNodeDesc> expressions = new ArrayList<>(outColumnCnt);
AtomicBoolean convert = new AtomicBoolean(false);
- if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt) {
+ if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
Review Comment:
I modified the check for Iceberg case as following, is it fine now?
```
if (table.getStorageHandler() != null && table.getStorageHandler().alwaysUnpartitioned()) {
if (inColumnCnt != outColumnCnt || staticPartitionColumnCnt > inColumnCnt){
String reason = "Always unpartitioned table " + dest + " has " + outColumnCnt
+ " columns, but query has " + inColumnCnt + " columns.";
throw new SemanticException(ASTErrorUtils.getMsg(
ErrorMsg.TARGET_TABLE_COLUMN_MISMATCH.getMsg(),
qb.getParseInfo().getDestForClause(dest), reason));
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1525221477
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/AbstractBaseAlterTableAnalyzer.java:
##########
@@ -98,6 +100,7 @@ protected void addInputsOutputsAlterTable(TableName tableName, Map<String, Strin
if (AlterTableUtils.isFullPartitionSpec(table, partitionSpec)) {
// Fully specified partition spec
+ SessionState.getSessionConf().set(Context.operationNameConf, op.getName());
Review Comment:
can we get the operation from Context.getOperation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1525216056
##########
ql/src/java/org/apache/hadoop/hive/ql/Compiler.java:
##########
@@ -100,6 +100,7 @@ public QueryPlan compile(String rawCommand, boolean deferClose) throws CommandPr
boolean parsed = false;
QueryPlan plan = null;
try {
+ SessionState.getSessionConf().set(Context.operationNameConf, org.apache.commons.lang3.StringUtils.EMPTY);
Review Comment:
why do we need this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-1997646375
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [9 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1569495716
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##########
@@ -544,14 +549,20 @@ private void commitWrite(Table table, String branchName, Long snapshotId, long s
* @param rewritePolicy The rewrite policy to use for the insert overwrite commit
*/
private void commitOverwrite(Table table, String branchName, long startTime, FilesForCommit results,
- RewritePolicy rewritePolicy) {
+ RewritePolicy rewritePolicy, String partitionSpec) {
Preconditions.checkArgument(results.deleteFiles().isEmpty(), "Can not handle deletes with overwrite");
if (!results.dataFiles().isEmpty()) {
Transaction transaction = table.newTransaction();
if (rewritePolicy == RewritePolicy.ALL_PARTITIONS) {
DeleteFiles delete = transaction.newDelete();
delete.deleteFromRowFilter(Expressions.alwaysTrue());
delete.commit();
+ } else if (rewritePolicy == RewritePolicy.SINGLE_PARTITION) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1561221183
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -21,45 +21,159 @@
import java.io.IOException;
import java.util.Map;
+import jline.internal.Log;
import org.apache.hadoop.hive.conf.HiveConf;
import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.metastore.Warehouse;
+import org.apache.hadoop.hive.metastore.api.MetaException;
import org.apache.hadoop.hive.ql.Context.RewritePolicy;
import org.apache.hadoop.hive.ql.DriverUtils;
import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.parse.AlterTableSnapshotRefSpec;
import org.apache.hadoop.hive.ql.session.SessionState;
import org.apache.hadoop.hive.ql.txn.compactor.CompactorContext;
import org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor;
import org.apache.hive.iceberg.org.apache.orc.storage.common.TableName;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.PartitionData;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter;
+import org.apache.iceberg.mr.hive.IcebergBranchExec;
+import org.apache.iceberg.mr.hive.IcebergTableUtil;
+import org.apache.iceberg.types.Type;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class IcebergMajorQueryCompactor extends QueryCompactor {
private static final Logger LOG = LoggerFactory.getLogger(IcebergMajorQueryCompactor.class.getName());
+ private static final String branchName = "compaction";
Review Comment:
finals should be in uppercase
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2052616730
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [14 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2065721743
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [10 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2067523855
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1574878825
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -750,6 +750,11 @@ default boolean canUseTruncate(org.apache.hadoop.hive.ql.metadata.Table hmsTable
return true;
}
+ default boolean isEligibilityForCompaction(org.apache.hadoop.hive.ql.metadata.Table table,
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1573859264
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/compact/AlterTableCompactOperation.java:
##########
@@ -61,6 +62,10 @@ public AlterTableCompactOperation(DDLOperationContext context, AlterTableCompact
if (!AcidUtils.isTransactionalTable(table) && !AcidUtils.isNonNativeAcidTable(table)) {
throw new HiveException(ErrorMsg.NONACID_COMPACTION_NOT_SUPPORTED, table.getDbName(), table.getTableName());
}
+
+ if (table.getStorageHandler() != null) {
+ table.getStorageHandler().isEligibilityForCompaction(table, desc.getPartitionSpec());
Review Comment:
return if "false"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1573873316
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +55,71 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ Table table = IcebergTableUtil.getTable(conf, context.getTable());
+ List<String> partFields = table.spec().fields().stream().map(PartitionField::name).collect(Collectors.toList());
+ String queryFields = table.schema().columns().stream().map(Types.NestedField::name)
+ .filter(f -> !partFields.contains(f)).collect(Collectors.joining(","));
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select %4$s from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "), queryFields);
+ }
+
+ SessionState sessionState = setupQueryCompactionSession(conf, context.getCompactionInfo(), tblProperties);
- SessionState sessionState = setupQueryCompactionSession(context.getConf(),
- context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
try {
- DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
+ DriverUtils.runOnDriver(conf, sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
+ return true;
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
- throw new RuntimeException(e);
+ LOG.error("Failed compacting table {}", compactTableName, e);
+ throw e;
} finally {
sessionState.setCompaction(false);
}
+ }
+
+ private String partDataToSQL(PartitionData partitionData, String partSpec, String delimiter)
+ throws HiveException {
+
+ try {
+ Map<String, String> partSpecMap = Warehouse.makeSpecFromName(partSpec);
+ StringBuilder sb = new StringBuilder();
+
+ for (int i = 0; i < partitionData.size(); ++i) {
+ if (i > 0) {
+ sb.append(delimiter);
+ }
+
+ String quoteOpt = "";
+ if (partitionData.getType(i).typeId() == Type.TypeID.STRING ||
+ partitionData.getType(i).typeId() == Type.TypeID.DATE ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIME ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIMESTAMP ||
+ partitionData.getType(i).typeId() == Type.TypeID.BINARY) {
+ quoteOpt = "'";
+ }
- return true;
+ String fieldName = partitionData.getSchema().getFields().get(i).name();
+
+ sb.append(fieldName)
+ .append("=")
+ .append(quoteOpt)
+ .append(partSpecMap.get(fieldName))
+ .append(quoteOpt);
Review Comment:
nit: remove extra space
also, you could simplify the code with
````
HiveUtils.unparseIdentifier(fieldName)
.append("=")
.append(TypeInfoUtils.convertStringToLiteralForSQL(
((PrimitiveTypeInfo) TypeInfoUtils.getTypeInfoFromTypeString(partitionData.getType(i).typeId())).getPrimitiveCategory()
````
check `DDLPlanUtils.getPartitionActualName`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1573866442
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/PartitionUtils.java:
##########
@@ -108,10 +122,18 @@ public static Partition getPartition(Hive db, Table table, Map<String, String> p
public static List<Partition> getPartitions(Hive db, Table table, Map<String, String> partitionSpec,
boolean throwException) throws SemanticException {
List<Partition> partitions;
- try {
- partitions = partitionSpec == null ? db.getPartitions(table) : db.getPartitions(table, partitionSpec);
- } catch (Exception e) {
- throw new SemanticException(toMessage(ErrorMsg.INVALID_PARTITION, partitionSpec), e);
+ if (table.getStorageHandler() != null && table.getStorageHandler().alwaysUnpartitioned()) {
Review Comment:
same here: `Hive.getPartitions()` + create method in SH `getPartitions()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1573884289
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -1915,6 +1915,28 @@ private boolean hasUndergonePartitionEvolution(Table table) {
.anyMatch(id -> id < table.spec().specId());
}
+ private boolean isIdentityPartitionTable(org.apache.hadoop.hive.ql.metadata.Table table) {
+ return getPartitionTransformSpec(table).stream().map(TransformSpec::getTransformType)
+ .allMatch(type -> type == TransformSpec.TransformType.IDENTITY);
+ }
+
+ @Override
+ public boolean isEligibilityForCompaction(org.apache.hadoop.hive.ql.metadata.Table table,
+ Map<String, String> partitionSpec) throws HiveException {
+ if (partitionSpec != null) {
+ Table icebergTable = IcebergTableUtil.getTable(conf, table.getTTable());
+ if (hasUndergonePartitionEvolution(icebergTable)) {
+ throw new HiveException("Compaction on partition level is not supported on the table " +
+ table.getDbName() + "." + table.getTableName() + " which has undergone partition evolution");
+ }
+ if (!isIdentityPartitionTable(table)) {
Review Comment:
only primitiveTypes: `partitionData.getType(i).isPrimitiveType()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-1986894016
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [16 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2040556461
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [18 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2041214091
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [15 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1565474973
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##########
@@ -569,14 +575,25 @@ private void commitWrite(Table table, String branchName, Long snapshotId, long s
* @param rewritePolicy The rewrite policy to use for the insert overwrite commit
*/
private void commitOverwrite(Table table, String branchName, long startTime, FilesForCommit results,
- RewritePolicy rewritePolicy) {
+ RewritePolicy rewritePolicy, String partitionSpec) {
Preconditions.checkArgument(results.deleteFiles().isEmpty(), "Can not handle deletes with overwrite");
if (!results.dataFiles().isEmpty()) {
Transaction transaction = table.newTransaction();
if (rewritePolicy == RewritePolicy.ALL_PARTITIONS) {
Review Comment:
do we need a rewrite policy, since we can check for `partitionSpec.isEmpty()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1535400489
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##########
@@ -8810,7 +8810,8 @@ private Operator genConversionSelectOperatorByAddPartition(String dest, QB qb, O
List<ExprNodeDesc> expressions = new ArrayList<>(outColumnCnt);
AtomicBoolean convert = new AtomicBoolean(false);
- if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt) {
+ if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
Review Comment:
previously this check was done regardless of `alwaysUnpartitioned` flag and now we moved it inside the if?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1571113887
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/compact/AlterTableCompactOperation.java:
##########
@@ -62,6 +65,12 @@ public AlterTableCompactOperation(DDLOperationContext context, AlterTableCompact
throw new HiveException(ErrorMsg.NONACID_COMPACTION_NOT_SUPPORTED, table.getDbName(), table.getTableName());
}
+ if (desc.getPartitionSpec() != null && DDLUtils.isIcebergTable(table) &&
Review Comment:
we should also check if table `hasUndergonePartitionEvolution`, see method in HiveIcebergStorageHandler
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1575083584
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/PartitionUtils.java:
##########
@@ -94,10 +98,20 @@ public static ExprNodeGenericFuncDesc makeUnaryPredicate(String fn, ExprNodeDesc
public static Partition getPartition(Hive db, Table table, Map<String, String> partitionSpec, boolean throwException)
throws SemanticException {
Partition partition;
- try {
- partition = db.getPartition(table, partitionSpec, false);
- } catch (Exception e) {
- throw new SemanticException(toMessage(ErrorMsg.INVALID_PARTITION, partitionSpec), e);
+ if (table.getStorageHandler() != null && table.getStorageHandler().alwaysUnpartitioned()) {
Review Comment:
Done. This is not used in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1575801527
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/PartitionUtils.java:
##########
@@ -95,7 +95,7 @@ public static Partition getPartition(Hive db, Table table, Map<String, String> p
throws SemanticException {
Partition partition;
try {
- partition = db.getPartition(table, partitionSpec, false);
+ partition = db.getPartition(table, partitionSpec);
Review Comment:
3rd param, `forceCreate` was omitted, why?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1569497802
##########
ql/src/java/org/apache/hadoop/hive/ql/Context.java:
##########
@@ -256,6 +257,7 @@ public String toString() {
public enum RewritePolicy {
DEFAULT,
+ SINGLE_PARTITION,
Review Comment:
done
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -21,45 +21,159 @@
import java.io.IOException;
import java.util.Map;
+import jline.internal.Log;
import org.apache.hadoop.hive.conf.HiveConf;
import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.metastore.Warehouse;
+import org.apache.hadoop.hive.metastore.api.MetaException;
import org.apache.hadoop.hive.ql.Context.RewritePolicy;
import org.apache.hadoop.hive.ql.DriverUtils;
import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.parse.AlterTableSnapshotRefSpec;
import org.apache.hadoop.hive.ql.session.SessionState;
import org.apache.hadoop.hive.ql.txn.compactor.CompactorContext;
import org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor;
import org.apache.hive.iceberg.org.apache.orc.storage.common.TableName;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.PartitionData;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter;
+import org.apache.iceberg.mr.hive.IcebergBranchExec;
+import org.apache.iceberg.mr.hive.IcebergTableUtil;
+import org.apache.iceberg.types.Type;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class IcebergMajorQueryCompactor extends QueryCompactor {
private static final Logger LOG = LoggerFactory.getLogger(IcebergMajorQueryCompactor.class.getName());
+ private static final String branchName = "compaction";
+ private String compactTableName;
@Override
public boolean run(CompactorContext context) throws IOException, HiveException, InterruptedException {
- String compactTableName = TableName.getDbTable(context.getTable().getDbName(), context.getTable().getTableName());
+ compactTableName = TableName.getDbTable(context.getTable().getDbName(), context.getTable().getTableName());
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
SessionState sessionState = setupQueryCompactionSession(context.getConf(),
context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+
try {
- DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
- LOG.info("Completed compaction for table {}", compactTableName);
- } catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
- throw new RuntimeException(e);
+ if (partSpec == null) {
+ HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+
+ executeHiveQuery(context, compactionQuery, sessionState,
+ String.format("compacting table %s", compactTableName));
+ } else {
+ Table table = IcebergTableUtil.getTable(context.getConf(), context.getTable());
+ AlterTableSnapshotRefSpec.DropSnapshotRefSpec dropBranchSpec =
+ new AlterTableSnapshotRefSpec.DropSnapshotRefSpec(branchName, true);
+ IcebergBranchExec.dropBranch(table, dropBranchSpec);
+
+ AlterTableSnapshotRefSpec.CreateSnapshotRefSpec createBranchSpec =
+ new AlterTableSnapshotRefSpec.CreateSnapshotRefSpec(branchName);
+ IcebergBranchExec.createBranch(table, createBranchSpec);
+ table.refresh();
+ Long createBranchSnapshotId = HiveIcebergOutputCommitter.getSnapshotId(table, "branch_" + branchName);
+
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ String truncateQuery = String.format("truncate table %s.branch_%s partition(%s)",
+ compactTableName, branchName, partDataToSQL(partitionData, partSpec, ","));
+ executeHiveQuery(context, truncateQuery, sessionState,
+ String.format("truncating partition %s of table %s.branch_%s", partSpec, compactTableName,
+ branchName));
+ table.refresh();
Review Comment:
not needed anymore, removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1568977615
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -2127,4 +2127,26 @@ public List<FileStatus> getMergeTaskInputFiles(Properties properties) throws IOE
public MergeTaskProperties getMergeTaskProperties(Properties properties) {
return new IcebergMergeTaskProperties(properties);
}
+
+ @Override
+ public boolean isIdentityPartitionSpec(org.apache.hadoop.hive.metastore.api.Table table,
Review Comment:
why should it be in API, could we move this to TableUtils?
````
Utils.isIdentityPartitionSpec(table.getStorageHandler().getPartitionTransformSpec)
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1569079683
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/PartitionUtils.java:
##########
@@ -94,10 +97,21 @@ public static ExprNodeGenericFuncDesc makeUnaryPredicate(String fn, ExprNodeDesc
public static Partition getPartition(Hive db, Table table, Map<String, String> partitionSpec, boolean throwException)
throws SemanticException {
Partition partition;
- try {
- partition = db.getPartition(table, partitionSpec, false);
- } catch (Exception e) {
- throw new SemanticException(toMessage(ErrorMsg.INVALID_PARTITION, partitionSpec), e);
+ if (table.getStorageHandler() != null && table.getStorageHandler().alwaysUnpartitioned()) {
+ table.getStorageHandler().validatePartSpec(table, partitionSpec);
+ try {
+ String partName = Warehouse.makePartName(partitionSpec, false);
+ partition = new DummyPartition(table, partName, partitionSpec);
+ } catch (MetaException e) {
+ throw new SemanticException("Unable to construct name for dummy partition due to: ", e);
+ }
+ }
+ else {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1571193903
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##########
@@ -8815,6 +8815,23 @@ These props are now enabled elsewhere (see commit diffs). It would be better in
conf.set(AcidUtils.CONF_ACID_KEY, "true");
SessionState.get().getConf().set(AcidUtils.CONF_ACID_KEY, "true");
}
+
+ private void validateColumnCounts(String dest, Table table, Map<String, String> partitionSpec,
+ int inColumnCnt, int outColumnCnt) throws SemanticException {
+
+ long staticPartitionColumnCnt = partitionSpec != null ?
+ partitionSpec.values().stream().filter(Objects::nonNull).count() : 0;
+
+ if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
Review Comment:
Extracted the code into a local var, but about the second part, unfortunately it won't work because I saw that for Iceberg it can be `inColumnCnt + staticPartitionColumnCnt == outColumnCnt` or` inColumnCnt == outColumnCnt` and the suggested expression won't cover the first case.
Would you accept this instead?
```
if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
((alwaysUnpartitioned && inColumnCnt != outColumnCnt) || !alwaysUnpartitioned)) {
...
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1569041910
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:
##########
@@ -1187,6 +1187,11 @@ private List<Partition> getPartitionsByQuery(String catName, String dbName, Stri
MetastoreDirectSqlUtils.timingTrace(doTrace, queryText, start, queryTime);
}
String partIds = getIdListForIn(partitions.keySet());
+
+ if (partIds.isEmpty()) {
Review Comment:
do we still need this after you've added iceberg partition list functionality in PartitionUtils?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1574879897
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/compact/AlterTableCompactOperation.java:
##########
@@ -61,6 +62,10 @@ public AlterTableCompactOperation(DDLOperationContext context, AlterTableCompact
if (!AcidUtils.isTransactionalTable(table) && !AcidUtils.isNonNativeAcidTable(table)) {
throw new HiveException(ErrorMsg.NONACID_COMPACTION_NOT_SUPPORTED, table.getDbName(), table.getTableName());
}
+
+ if (table.getStorageHandler() != null) {
+ table.getStorageHandler().isEligibilityForCompaction(table, desc.getPartitionSpec());
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1573856606
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -750,6 +750,11 @@ default boolean canUseTruncate(org.apache.hadoop.hive.ql.metadata.Table hmsTable
return true;
}
+ default boolean isEligibilityForCompaction(org.apache.hadoop.hive.ql.metadata.Table table,
Review Comment:
typo: `isEligibleForCompaction()`, default should be `false`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1565694331
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +52,68 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.DEFAULT.name());
+ Table table = IcebergTableUtil.getTable(context.getConf(), context.getTable());
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ context.getConf().set(Context.compactPartitionSpec, partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select * from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "));
+ }
SessionState sessionState = setupQueryCompactionSession(context.getConf(),
context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+
try {
DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
+ return true;
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
- throw new RuntimeException(e);
+ LOG.error("Failed compacting table {}", compactTableName, e);
+ throw e;
} finally {
sessionState.setCompaction(false);
}
+ }
+
+ private String partDataToSQL(PartitionData partitionData, String partSpec, String delimiter)
Review Comment:
you should use logic from `truncatePartitionBySpec` as you can't override partition with non identity spec
if (!TransformSpec.TransformType.IDENTITY.equals(transformType)) {
throw new MetaException(
String.format("Partition transforms are not supported in compact operation: %s", entry.getKey()));
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1565481530
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##########
@@ -569,14 +575,25 @@ private void commitWrite(Table table, String branchName, Long snapshotId, long s
* @param rewritePolicy The rewrite policy to use for the insert overwrite commit
*/
private void commitOverwrite(Table table, String branchName, long startTime, FilesForCommit results,
- RewritePolicy rewritePolicy) {
+ RewritePolicy rewritePolicy, String partitionSpec) {
Preconditions.checkArgument(results.deleteFiles().isEmpty(), "Can not handle deletes with overwrite");
if (!results.dataFiles().isEmpty()) {
Transaction transaction = table.newTransaction();
if (rewritePolicy == RewritePolicy.ALL_PARTITIONS) {
DeleteFiles delete = transaction.newDelete();
delete.deleteFromRowFilter(Expressions.alwaysTrue());
delete.commit();
+ LOG.info("Overwrite commit: truncated all partitions for table: {} with {} file(s)",
+ table, results.dataFiles().size());
+ } else if (!partitionSpec.isEmpty()) {
Review Comment:
why do we need this? that won't help with partition evolution in any case
ATM (no evolution support) partitionSpec is not required
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2053847911
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [14 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1561329055
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -21,45 +21,159 @@
import java.io.IOException;
import java.util.Map;
+import jline.internal.Log;
import org.apache.hadoop.hive.conf.HiveConf;
import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.metastore.Warehouse;
+import org.apache.hadoop.hive.metastore.api.MetaException;
import org.apache.hadoop.hive.ql.Context.RewritePolicy;
import org.apache.hadoop.hive.ql.DriverUtils;
import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.parse.AlterTableSnapshotRefSpec;
import org.apache.hadoop.hive.ql.session.SessionState;
import org.apache.hadoop.hive.ql.txn.compactor.CompactorContext;
import org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor;
import org.apache.hive.iceberg.org.apache.orc.storage.common.TableName;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.PartitionData;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter;
+import org.apache.iceberg.mr.hive.IcebergBranchExec;
+import org.apache.iceberg.mr.hive.IcebergTableUtil;
+import org.apache.iceberg.types.Type;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class IcebergMajorQueryCompactor extends QueryCompactor {
private static final Logger LOG = LoggerFactory.getLogger(IcebergMajorQueryCompactor.class.getName());
+ private static final String branchName = "compaction";
Review Comment:
done
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -21,45 +21,159 @@
import java.io.IOException;
import java.util.Map;
+import jline.internal.Log;
import org.apache.hadoop.hive.conf.HiveConf;
import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.metastore.Warehouse;
+import org.apache.hadoop.hive.metastore.api.MetaException;
import org.apache.hadoop.hive.ql.Context.RewritePolicy;
import org.apache.hadoop.hive.ql.DriverUtils;
import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.parse.AlterTableSnapshotRefSpec;
import org.apache.hadoop.hive.ql.session.SessionState;
import org.apache.hadoop.hive.ql.txn.compactor.CompactorContext;
import org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor;
import org.apache.hive.iceberg.org.apache.orc.storage.common.TableName;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.PartitionData;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter;
+import org.apache.iceberg.mr.hive.IcebergBranchExec;
+import org.apache.iceberg.mr.hive.IcebergTableUtil;
+import org.apache.iceberg.types.Type;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class IcebergMajorQueryCompactor extends QueryCompactor {
private static final Logger LOG = LoggerFactory.getLogger(IcebergMajorQueryCompactor.class.getName());
+ private static final String branchName = "compaction";
+ private String compactTableName;
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1526363279
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -2232,7 +2232,9 @@ public static enum ConfVars {
"If this is set to true the URI for auth will have the default location masked with DEFAULT_TABLE_LOCATION"),
HIVE_ICEBERG_ALLOW_DATAFILES_IN_TABLE_LOCATION_ONLY("hive.iceberg.allow.datafiles.in.table.location.only", false,
"If this is set to true, then all the data files being read should be withing the table location"),
-
+ HIVE_ICEBERG_COMPACTION_PARTITION("hive.iceberg.compaction.partition", "",
Review Comment:
Thanks, replaced with a constant in Context class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1525189830
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -2232,7 +2232,9 @@ public static enum ConfVars {
"If this is set to true the URI for auth will have the default location masked with DEFAULT_TABLE_LOCATION"),
HIVE_ICEBERG_ALLOW_DATAFILES_IN_TABLE_LOCATION_ONLY("hive.iceberg.allow.datafiles.in.table.location.only", false,
"If this is set to true, then all the data files being read should be withing the table location"),
-
+ HIVE_ICEBERG_COMPACTION_PARTITION("hive.iceberg.compaction.partition", "",
Review Comment:
why do we need HiveConf for that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1525205302
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -2021,4 +2022,34 @@ public List<Partition> getPartitionsByExpr(org.apache.hadoop.hive.ql.metadata.Ta
throw new SemanticException(String.format("Error while fetching the partitions due to: %s", e));
}
}
+
+ @Override
+ public List<Partition> getPartitions(org.apache.hadoop.hive.ql.metadata.Table hmsTable, List<String> partSpecs) {
+
+ return partSpecs
+ .stream()
+ .map(spec -> {
+ Map<String, String> partSpecsMap = Arrays.stream(spec.split("/"))
+ .collect(Collectors.toMap(
+ x -> x.split("=")[0],
+ x -> x.split("=")[1]
+ ));
+ validatePartitionName(hmsTable, partSpecsMap);
+ return spec;
+ })
+ .map(partName -> new DummyPartition(hmsTable, partName))
+ .collect(Collectors.toList());
+ }
+
+ private void validatePartitionName(org.apache.hadoop.hive.ql.metadata.Table hmsTable, Map<String, String> partSpec) {
+ try {
+ List<String> partNames = getPartitionNames(hmsTable, partSpec);
+
+ if (partNames.isEmpty()) {
+ throw new RuntimeException("Failed getting partitions from partition spec");
Review Comment:
do we have some more specific exceptions in Hive?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1532288946
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##########
@@ -544,14 +549,20 @@ private void commitWrite(Table table, String branchName, Long snapshotId, long s
* @param rewritePolicy The rewrite policy to use for the insert overwrite commit
*/
private void commitOverwrite(Table table, String branchName, long startTime, FilesForCommit results,
- RewritePolicy rewritePolicy) {
+ RewritePolicy rewritePolicy, String partitionSpec) {
Preconditions.checkArgument(results.deleteFiles().isEmpty(), "Can not handle deletes with overwrite");
if (!results.dataFiles().isEmpty()) {
Transaction transaction = table.newTransaction();
if (rewritePolicy == RewritePolicy.ALL_PARTITIONS) {
DeleteFiles delete = transaction.newDelete();
delete.deleteFromRowFilter(Expressions.alwaysTrue());
delete.commit();
+ } else if (rewritePolicy == RewritePolicy.SINGLE_PARTITION) {
Review Comment:
can't we use the DEFAULT? rewrite policy enum is either all or not all (custom set)
so you can simply use `else` without `if`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1532869151
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##########
@@ -462,7 +463,12 @@ private void commitTable(FileIO io, ExecutorService executor, OutputTable output
.map(x -> x.getJobConf().get(ConfVars.REWRITE_POLICY.varname))
.orElse(RewritePolicy.DEFAULT.name()));
- commitOverwrite(table, branchName, startTime, filesForCommit, rewritePolicy);
+ String partitionSpec = outputTable.jobContexts.stream()
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2019059227
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [20 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2053053734
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [14 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1571017320
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/compact/AlterTableCompactOperation.java:
##########
@@ -62,6 +65,12 @@ public AlterTableCompactOperation(DDLOperationContext context, AlterTableCompact
throw new HiveException(ErrorMsg.NONACID_COMPACTION_NOT_SUPPORTED, table.getDbName(), table.getTableName());
}
+ if (desc.getPartitionSpec() != null && DDLUtils.isIcebergTable(table) &&
Review Comment:
could we move this logic into AlterTableUtils?
````
public boolean isIdentityPartitionedTable(table) {
return table.getStorageHandler() != null && table.getStorageHandler().getPartitionTransformSpec(table).stream()
.map(TransformSpec::getTransformType)
.allMatch(type -> type == TransformSpec.TransformType.IDENTITY);
````
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1568961100
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +51,69 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ Table table = IcebergTableUtil.getTable(conf, context.getTable());
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select * from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "));
+ }
+
+ SessionState sessionState = setupQueryCompactionSession(conf, context.getCompactionInfo(), tblProperties);
+ sessionState.setConf(conf);
Review Comment:
check MajorQueryCompactor for example, it just clones the conf and passes that to session
````
HiveConf conf = new HiveConf(hiveConf);
/*
* For now, we will group splits on tez so that we end up with all bucket files,
* with same bucket number in one map task.
*/
conf.set(HiveConf.ConfVars.SPLIT_GROUPING_MODE.varname, CompactorUtil.COMPACTOR);
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1571017320
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/compact/AlterTableCompactOperation.java:
##########
@@ -62,6 +65,12 @@ public AlterTableCompactOperation(DDLOperationContext context, AlterTableCompact
throw new HiveException(ErrorMsg.NONACID_COMPACTION_NOT_SUPPORTED, table.getDbName(), table.getTableName());
}
+ if (desc.getPartitionSpec() != null && DDLUtils.isIcebergTable(table) &&
Review Comment:
could we move this logic into AlterTableUtils?
````
isIdentityPartitionSpec(table) {
return table.getStorageHandler() != null && table.getStorageHandler().getPartitionTransformSpec(table)
.stream().map(TransformSpec::getTransformType)
.allMatch(type -> type == TransformSpec.TransformType.IDENTITY);
````
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1570846660
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##########
@@ -8815,6 +8815,23 @@ These props are now enabled elsewhere (see commit diffs). It would be better in
conf.set(AcidUtils.CONF_ACID_KEY, "true");
SessionState.get().getConf().set(AcidUtils.CONF_ACID_KEY, "true");
}
+
+ private void validateColumnCounts(String dest, Table table, Map<String, String> partitionSpec,
+ int inColumnCnt, int outColumnCnt) throws SemanticException {
+
+ long staticPartitionColumnCnt = partitionSpec != null ?
+ partitionSpec.values().stream().filter(Objects::nonNull).count() : 0;
+
+ if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
Review Comment:
````
if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt
&& (!alwaysUnpartitioned || inColumnCnt != outColumnCnt) ) {
...
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1570846660
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##########
@@ -8815,6 +8815,23 @@ These props are now enabled elsewhere (see commit diffs). It would be better in
conf.set(AcidUtils.CONF_ACID_KEY, "true");
SessionState.get().getConf().set(AcidUtils.CONF_ACID_KEY, "true");
}
+
+ private void validateColumnCounts(String dest, Table table, Map<String, String> partitionSpec,
+ int inColumnCnt, int outColumnCnt) throws SemanticException {
+
+ long staticPartitionColumnCnt = partitionSpec != null ?
+ partitionSpec.values().stream().filter(Objects::nonNull).count() : 0;
+
+ if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
Review Comment:
````
if (inColumnCnt + (alwaysUnpartitioned ? 0 : staticPartitionColumnCnt) != outColumnCnt) {
,,,
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1571421601
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/compact/AlterTableCompactOperation.java:
##########
@@ -62,6 +65,12 @@ public AlterTableCompactOperation(DDLOperationContext context, AlterTableCompact
throw new HiveException(ErrorMsg.NONACID_COMPACTION_NOT_SUPPORTED, table.getDbName(), table.getTableName());
}
+ if (desc.getPartitionSpec() != null && DDLUtils.isIcebergTable(table) &&
Review Comment:
For the identity partition table, the suggested code works only for Iceberg tables, but not for native.
Modified to:
```
public static boolean isIdentityPartitionTable(Table table) {
return table.getStorageHandler() == null || table.getStorageHandler().getPartitionTransformSpec(table).stream()
.map(TransformSpec::getTransformType)
.allMatch(type -> type == TransformSpec.TransformType.IDENTITY);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1569000480
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +51,69 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ Table table = IcebergTableUtil.getTable(conf, context.getTable());
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select * from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "));
+ }
+
+ SessionState sessionState = setupQueryCompactionSession(conf, context.getCompactionInfo(), tblProperties);
+ sessionState.setConf(conf);
Review Comment:
Yes, I think we should. By looking the at code, I think it behaves the same for ACID compactions, meaning that conf is not updates when using the same session..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1569039181
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##########
@@ -8810,7 +8810,8 @@ private Operator genConversionSelectOperatorByAddPartition(String dest, QB qb, O
List<ExprNodeDesc> expressions = new ArrayList<>(outColumnCnt);
AtomicBoolean convert = new AtomicBoolean(false);
- if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt) {
+ if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
Review Comment:
we shouldn't overload existing code it's already complicated. is it possible to extract this column check in some validateColumns method?
cc @kasakrisz
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1571193903
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##########
@@ -8815,6 +8815,23 @@ These props are now enabled elsewhere (see commit diffs). It would be better in
conf.set(AcidUtils.CONF_ACID_KEY, "true");
SessionState.get().getConf().set(AcidUtils.CONF_ACID_KEY, "true");
}
+
+ private void validateColumnCounts(String dest, Table table, Map<String, String> partitionSpec,
+ int inColumnCnt, int outColumnCnt) throws SemanticException {
+
+ long staticPartitionColumnCnt = partitionSpec != null ?
+ partitionSpec.values().stream().filter(Objects::nonNull).count() : 0;
+
+ if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
Review Comment:
Extracted the code into a local var, but about the second part, unfortunately it won't work because I saw that for Iceberg in some cases (q-tests) it can be `inColumnCnt + staticPartitionColumnCnt == outColumnCnt` and in other cases ` inColumnCnt == outColumnCnt` and the suggested expression won't cover the first case.
Would you accept this instead?
```
if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
((alwaysUnpartitioned && inColumnCnt != outColumnCnt) || !alwaysUnpartitioned)) {
...
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1569023720
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/PartitionUtils.java:
##########
@@ -108,11 +122,17 @@ public static Partition getPartition(Hive db, Table table, Map<String, String> p
public static List<Partition> getPartitions(Hive db, Table table, Map<String, String> partitionSpec,
boolean throwException) throws SemanticException {
List<Partition> partitions;
- try {
- partitions = partitionSpec == null ? db.getPartitions(table) : db.getPartitions(table, partitionSpec);
- } catch (Exception e) {
- throw new SemanticException(toMessage(ErrorMsg.INVALID_PARTITION, partitionSpec), e);
- }
+ if (table.getStorageHandler() != null && table.getStorageHandler().alwaysUnpartitioned()) {
+ partitions = new ArrayList<>();
+ partitions.add(getPartition(db, table, partitionSpec, throwException));
Review Comment:
that is a diff API, it supposed to return all partitions matching spec, not just 1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1572011889
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##########
@@ -8815,6 +8815,23 @@ These props are now enabled elsewhere (see commit diffs). It would be better in
conf.set(AcidUtils.CONF_ACID_KEY, "true");
SessionState.get().getConf().set(AcidUtils.CONF_ACID_KEY, "true");
}
+
+ private void validateColumnCounts(String dest, Table table, Map<String, String> partitionSpec,
+ int inColumnCnt, int outColumnCnt) throws SemanticException {
+
+ long staticPartitionColumnCnt = partitionSpec != null ?
+ partitionSpec.values().stream().filter(Objects::nonNull).count() : 0;
+
+ if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
Review Comment:
i had the same code at the beginning, but it looks a bit weird if you split it
````
!alwaysUnpartitioned && inColumnCnt + staticPartitionColumnCnt != outColumnCnt
or
alwaysUnpartitioned && inColumnCnt + staticPartitionColumnCnt != outColumnCnt && inColumnCnt != outColumnCnt
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1572034783
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/compact/AlterTableCompactOperation.java:
##########
@@ -62,6 +65,17 @@ public AlterTableCompactOperation(DDLOperationContext context, AlterTableCompact
throw new HiveException(ErrorMsg.NONACID_COMPACTION_NOT_SUPPORTED, table.getDbName(), table.getTableName());
}
+ if (desc.getPartitionSpec() != null && !AlterTableUtils.isIdentityPartitionTable(table)) {
+ throw new HiveException("Compaction is not supported on the table " +
+ table.getDbName() + "." + table.getTableName() + " with non-identity partition spec");
+ }
+
+ if (desc.getPartitionSpec() != null && DDLUtils.isIcebergTable(table) &&
Review Comment:
how about we expose isEligibleForCompaction via StorageHandler
````
private boolean hasUndergonePartitionEvolution(Table table) {
// If it is a table which has undergone partition evolution, return true.
return table.currentSnapshot() != null &&
table.currentSnapshot().allManifests(table.io()).parallelStream()
.map(ManifestFile::partitionSpecId)
.anyMatch(id -> id < table.spec().specId());
}
private boolean isIdentityPartitionTable(Table table) {
return getPartitionTransformSpec(table).stream().map(TransformSpec::getTransformType)
.allMatch(type -> type == TransformSpec.TransformType.IDENTITY);
}
public boolean isEligibleForCompaction(Table table, Map<String, String> partitionSpec) throws HiveException {
if (partitionSpec != null) {
if (hasUndergonePartitionEvolution(table)) {
throw new HiveException("Compaction on partition level is not supported on the table " +
table.getDbName() + "." + table.getTableName() + " which has undergone partition evolution");
}
if (isIdentityPartitionTable(table)) {
throw new HiveException("Compaction is not supported on the table " +
table.getDbName() + "." + table.getTableName() + " with non-identity partition spec");
}
}
return true;
}
````
and call
````
if (table.getStorageHandler() != null) {
table.getStorageHandler().isEligibleForCompaction(table, partitionSpec);
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1572036781
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -750,6 +750,10 @@ default boolean canUseTruncate(org.apache.hadoop.hive.ql.metadata.Table hmsTable
return true;
}
+ default boolean hasUndergonePartitionEvolution(org.apache.hadoop.hive.ql.metadata.Table table) {
Review Comment:
let's expose `public boolean isEligibleForCompaction(Table table, Map<String, String> partitionSpec) throws HiveException {`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1577433921
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +56,52 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ org.apache.hadoop.hive.ql.metadata.Table table = Hive.get(conf).getTable(context.getTable().getDbName(),
+ context.getTable().getTableName());
+ Map<String, String> partSpecMap = new LinkedHashMap<>();
+ Warehouse.makeSpecFromName(partSpecMap, new Path(partSpec), null);
+
+ List<FieldSchema> partitionKeys = table.getStorageHandler().getPartitionKeys(table);
+ List<String> partValues = partitionKeys.stream().map(
+ fs -> String.join("=", HiveUtils.unparseIdentifier(fs.getName()),
+ TypeInfoUtils.convertStringToLiteralForSQL(partSpecMap.get(fs.getName()),
+ ((PrimitiveTypeInfo) TypeInfoUtils.getTypeInfoFromTypeString(fs.getType())).getPrimitiveCategory())
+ )
+ ).collect(Collectors.toList());
+
+ String queryFields = table.getCols().stream()
+ .map(FieldSchema::getName)
+ .filter(col -> !partSpecMap.containsKey(col))
+ .collect(Collectors.joining(","));
+
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select %4$s from %1$s where %3$s",
+ compactTableName,
+ StringUtils.join(partValues, ","),
+ StringUtils.join(partValues, " and "),
+ queryFields);
+ }
+
+ SessionState sessionState = setupQueryCompactionSession(conf, context.getCompactionInfo(), tblProperties);
+ String compactionTarget = "table " + compactTableName +
+ (Strings.nullToEmpty(partSpec).isEmpty() ? "" : ", partition " + partSpec);
Review Comment:
why no simply
````
"table " + HiveUtils.unparseIdentifier(compactTableName) +
(partSpec != null ? ", partition " + HiveUtils.unparseIdentifier(partSpec) : "")
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1575082688
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/compact/AlterTableCompactOperation.java:
##########
@@ -133,7 +138,7 @@ private List<Partition> getPartitions(Table table, AlterTableCompactDesc desc, D
}
} else {
Map<String, String> partitionSpec = desc.getPartitionSpec();
- partitions = context.getDb().getPartitions(table, partitionSpec);
+ partitions.addAll(PartitionUtils.getPartitions(context.getDb(), table, partitionSpec, false));
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1575794099
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -1915,6 +1917,46 @@ private boolean hasUndergonePartitionEvolution(Table table) {
.anyMatch(id -> id < table.spec().specId());
}
+ private boolean isIdentityPartitionTable(org.apache.hadoop.hive.ql.metadata.Table table) {
+ return getPartitionTransformSpec(table).stream().map(TransformSpec::getTransformType)
+ .allMatch(type -> type == TransformSpec.TransformType.IDENTITY);
+ }
+
+ @Override
+ public org.apache.commons.lang3.tuple.Pair<Boolean, ErrorMsg> isEligibleForCompaction(
+ org.apache.hadoop.hive.ql.metadata.Table table, Map<String, String> partitionSpec) {
+ if (partitionSpec != null) {
+ Table icebergTable = IcebergTableUtil.getTable(conf, table.getTTable());
+ if (hasUndergonePartitionEvolution(icebergTable)) {
+ return org.apache.commons.lang3.tuple.Pair.of(false, ErrorMsg.COMPACTION_PARTITION_EVOLUTION);
+ }
+ if (!isIdentityPartitionTable(table)) {
+ return org.apache.commons.lang3.tuple.Pair.of(false, ErrorMsg.COMPACTION_NON_IDENTITY_PARTITION_SPEC);
+ }
+ }
+ return org.apache.commons.lang3.tuple.Pair.of(true, null);
+ }
+
+ @Override
+ public List<Partition> getPartitions(org.apache.hadoop.hive.ql.metadata.Table table,
+ Map<String, String> partitionSpec, short limit) throws SemanticException {
+ return table.getStorageHandler().getPartitionNames(table, partitionSpec).stream()
+ .limit(limit > 0 ? limit : Long.MAX_VALUE)
+ .map(partName -> new DummyPartition(table, partName, partitionSpec)).collect(Collectors.toList());
+ }
+
+ @Override
+ public Partition getPartition(org.apache.hadoop.hive.ql.metadata.Table table,
+ Map<String, String> partitionSpec) throws SemanticException {
+ validatePartSpec(table, partitionSpec);
+ try {
+ String partName = Warehouse.makePartName(partitionSpec, false);
+ return new DummyPartition(table, partName, partitionSpec);
+ } catch (MetaException e) {
Review Comment:
can we use `makeDynamicPartNameNoTrailingSeperator` to avoid exception? should be ok since we validate before
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1575417125
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +55,71 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ Table table = IcebergTableUtil.getTable(conf, context.getTable());
+ List<String> partFields = table.spec().fields().stream().map(PartitionField::name).collect(Collectors.toList());
+ String queryFields = table.schema().columns().stream().map(Types.NestedField::name)
+ .filter(f -> !partFields.contains(f)).collect(Collectors.joining(","));
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select %4$s from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "), queryFields);
+ }
+
+ SessionState sessionState = setupQueryCompactionSession(conf, context.getCompactionInfo(), tblProperties);
- SessionState sessionState = setupQueryCompactionSession(context.getConf(),
- context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
try {
- DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
+ DriverUtils.runOnDriver(conf, sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
+ return true;
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
- throw new RuntimeException(e);
+ LOG.error("Failed compacting table {}", compactTableName, e);
+ throw e;
} finally {
sessionState.setCompaction(false);
}
+ }
+
+ private String partDataToSQL(PartitionData partitionData, String partSpec, String delimiter)
+ throws HiveException {
+
+ try {
+ Map<String, String> partSpecMap = Warehouse.makeSpecFromName(partSpec);
+ StringBuilder sb = new StringBuilder();
+
+ for (int i = 0; i < partitionData.size(); ++i) {
+ if (i > 0) {
+ sb.append(delimiter);
+ }
+
+ String quoteOpt = "";
+ if (partitionData.getType(i).typeId() == Type.TypeID.STRING ||
+ partitionData.getType(i).typeId() == Type.TypeID.DATE ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIME ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIMESTAMP ||
+ partitionData.getType(i).typeId() == Type.TypeID.BINARY) {
+ quoteOpt = "'";
+ }
- return true;
+ String fieldName = partitionData.getSchema().getFields().get(i).name();
+
+ sb.append(fieldName)
+ .append("=")
+ .append(quoteOpt)
+ .append(partSpecMap.get(fieldName))
+ .append(quoteOpt);
Review Comment:
`getTypeInfoFromTypeString()` didn't work with getting the type from partitionData, but I made it work a little differently just like in `TruncateTableAnalyzer.constructDeleteQuery()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1576401895
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -1915,6 +1917,46 @@ private boolean hasUndergonePartitionEvolution(Table table) {
.anyMatch(id -> id < table.spec().specId());
}
+ private boolean isIdentityPartitionTable(org.apache.hadoop.hive.ql.metadata.Table table) {
+ return getPartitionTransformSpec(table).stream().map(TransformSpec::getTransformType)
+ .allMatch(type -> type == TransformSpec.TransformType.IDENTITY);
+ }
+
+ @Override
+ public org.apache.commons.lang3.tuple.Pair<Boolean, ErrorMsg> isEligibleForCompaction(
+ org.apache.hadoop.hive.ql.metadata.Table table, Map<String, String> partitionSpec) {
+ if (partitionSpec != null) {
+ Table icebergTable = IcebergTableUtil.getTable(conf, table.getTTable());
+ if (hasUndergonePartitionEvolution(icebergTable)) {
+ return org.apache.commons.lang3.tuple.Pair.of(false, ErrorMsg.COMPACTION_PARTITION_EVOLUTION);
+ }
+ if (!isIdentityPartitionTable(table)) {
+ return org.apache.commons.lang3.tuple.Pair.of(false, ErrorMsg.COMPACTION_NON_IDENTITY_PARTITION_SPEC);
+ }
+ }
+ return org.apache.commons.lang3.tuple.Pair.of(true, null);
+ }
+
+ @Override
+ public List<Partition> getPartitions(org.apache.hadoop.hive.ql.metadata.Table table,
+ Map<String, String> partitionSpec, short limit) throws SemanticException {
Review Comment:
We don't need limit, fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1576859775
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +55,51 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ Hive db = Hive.get(conf);
+ org.apache.hadoop.hive.ql.metadata.Table table = db.getTable(context.getTable().getDbName(),
+ context.getTable().getTableName());
+ Map<String, String> partSpecMap = new LinkedHashMap<>();
+ Warehouse.makeSpecFromName(partSpecMap, new Path(partSpec), null);
+
+ List<FieldSchema> partitionKeys = table.getStorageHandler().getPartitionKeys(table);
+ List<String> partValues = partitionKeys.stream().map(
+ fs -> String.join("=", HiveUtils.unparseIdentifier(fs.getName()),
+ TypeInfoUtils.convertStringToLiteralForSQL(partSpecMap.get(fs.getName()),
+ ((PrimitiveTypeInfo) TypeInfoUtils.getTypeInfoFromTypeString(fs.getType())).getPrimitiveCategory())
+ )
+ ).collect(Collectors.toList());
+
+ String queryFields = table.getCols().stream()
+ .map(FieldSchema::getName)
+ .filter(col -> !partSpecMap.containsKey(col))
+ .collect(Collectors.joining(","));
+
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select %4$s from %1$s where %3$s",
+ compactTableName,
+ StringUtils.join(partValues, ","),
+ StringUtils.join(partValues, " and "),
+ queryFields);
+ }
+
+ SessionState sessionState = setupQueryCompactionSession(conf, context.getCompactionInfo(), tblProperties);
- SessionState sessionState = setupQueryCompactionSession(context.getConf(),
- context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
try {
- DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
+ DriverUtils.runOnDriver(conf, sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
+ return true;
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
- throw new RuntimeException(e);
+ LOG.error("Failed compacting table {}", compactTableName, e);
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1526459556
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##########
@@ -544,14 +549,20 @@ private void commitWrite(Table table, String branchName, Long snapshotId, long s
* @param rewritePolicy The rewrite policy to use for the insert overwrite commit
*/
private void commitOverwrite(Table table, String branchName, long startTime, FilesForCommit results,
- RewritePolicy rewritePolicy) {
+ RewritePolicy rewritePolicy, String partitionSpec) {
Preconditions.checkArgument(results.deleteFiles().isEmpty(), "Can not handle deletes with overwrite");
if (!results.dataFiles().isEmpty()) {
Transaction transaction = table.newTransaction();
if (rewritePolicy == RewritePolicy.ALL_PARTITIONS) {
DeleteFiles delete = transaction.newDelete();
delete.deleteFromRowFilter(Expressions.alwaysTrue());
delete.commit();
+ } else if (rewritePolicy == RewritePolicy.SINGLE_PARTITION) {
Review Comment:
If it were `rewritePolicy != RewritePolicy.ALL_PARTITIONS`, it would also be true for the case `rewritePolicy != RewritePolicy.DEFAULT`, but the code in the else block should only be executed for compacting a specific partition, not in general case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1565481530
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##########
@@ -569,14 +575,25 @@ private void commitWrite(Table table, String branchName, Long snapshotId, long s
* @param rewritePolicy The rewrite policy to use for the insert overwrite commit
*/
private void commitOverwrite(Table table, String branchName, long startTime, FilesForCommit results,
- RewritePolicy rewritePolicy) {
+ RewritePolicy rewritePolicy, String partitionSpec) {
Preconditions.checkArgument(results.deleteFiles().isEmpty(), "Can not handle deletes with overwrite");
if (!results.dataFiles().isEmpty()) {
Transaction transaction = table.newTransaction();
if (rewritePolicy == RewritePolicy.ALL_PARTITIONS) {
DeleteFiles delete = transaction.newDelete();
delete.deleteFromRowFilter(Expressions.alwaysTrue());
delete.commit();
+ LOG.info("Overwrite commit: truncated all partitions for table: {} with {} file(s)",
+ table, results.dataFiles().size());
+ } else if (!partitionSpec.isEmpty()) {
Review Comment:
why do we need this? that won't help with partition evolution in any case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2010731014
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [18 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2008589684
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [18 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1532280539
##########
ql/src/java/org/apache/hadoop/hive/ql/Context.java:
##########
@@ -117,6 +117,7 @@ public class Context {
protected String optimizedSql;
protected String calcitePlan;
protected String cmd = "";
+ public static String compactPartition = "hive.iceberg.compaction.partition";
Review Comment:
that shouldn't belong here, maybe AlterTableCompactOperation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1532252364
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java:
##########
@@ -640,7 +640,13 @@ public void preTruncateTable(org.apache.hadoop.hive.metastore.api.Table table, E
throws MetaException {
this.catalogProperties = getCatalogProperties(table);
this.icebergTable = Catalogs.loadTable(conf, catalogProperties);
- Map<String, PartitionField> partitionFieldMap = icebergTable.spec().fields().stream()
+ truncatePartitionBySpec(icebergTable, partNames, icebergTable.newDelete());
+ context.putToProperties("truncateSkipDataDeletion", "true");
Review Comment:
could we please use `TRUNCATE_SKIP_DATA_DELETION` constant here. btw, what does it do?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1531086273
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/PartitionIterable.java:
##########
@@ -99,7 +99,16 @@ private void getNextBatch() {
batchCounter++;
}
try {
- if (isAuthRequired) {
+ if (table.getStorageHandler() != null && table.getStorageHandler().alwaysUnpartitioned()) {
+ List<Partition> partitions = new ArrayList<>();
Review Comment:
I removed this change, it is not needed anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-1990014978
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [16 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1561222370
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -21,45 +21,159 @@
import java.io.IOException;
import java.util.Map;
+import jline.internal.Log;
import org.apache.hadoop.hive.conf.HiveConf;
import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.metastore.Warehouse;
+import org.apache.hadoop.hive.metastore.api.MetaException;
import org.apache.hadoop.hive.ql.Context.RewritePolicy;
import org.apache.hadoop.hive.ql.DriverUtils;
import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.parse.AlterTableSnapshotRefSpec;
import org.apache.hadoop.hive.ql.session.SessionState;
import org.apache.hadoop.hive.ql.txn.compactor.CompactorContext;
import org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor;
import org.apache.hive.iceberg.org.apache.orc.storage.common.TableName;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.PartitionData;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter;
+import org.apache.iceberg.mr.hive.IcebergBranchExec;
+import org.apache.iceberg.mr.hive.IcebergTableUtil;
+import org.apache.iceberg.types.Type;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class IcebergMajorQueryCompactor extends QueryCompactor {
private static final Logger LOG = LoggerFactory.getLogger(IcebergMajorQueryCompactor.class.getName());
+ private static final String branchName = "compaction";
+ private String compactTableName;
Review Comment:
it shouldn't have a state
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2046094834
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [13 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1561284855
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##########
@@ -485,7 +485,7 @@ private void commitTable(FileIO io, ExecutorService executor, OutputTable output
}
}
- private Long getSnapshotId(Table table, String branchName) {
+ public static Long getSnapshotId(Table table, String branchName) {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2053691139
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [17 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1533045609
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##########
@@ -8810,7 +8810,8 @@ private Operator genConversionSelectOperatorByAddPartition(String dest, QB qb, O
List<ExprNodeDesc> expressions = new ArrayList<>(outColumnCnt);
AtomicBoolean convert = new AtomicBoolean(false);
- if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt) {
+ if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
Review Comment:
I modified the check for Iceberg case as following, is it fine now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2073992773
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [11 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1575838382
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +55,71 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ Table table = IcebergTableUtil.getTable(conf, context.getTable());
+ List<String> partFields = table.spec().fields().stream().map(PartitionField::name).collect(Collectors.toList());
+ String queryFields = table.schema().columns().stream().map(Types.NestedField::name)
+ .filter(f -> !partFields.contains(f)).collect(Collectors.joining(","));
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select %4$s from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "), queryFields);
+ }
+
+ SessionState sessionState = setupQueryCompactionSession(conf, context.getCompactionInfo(), tblProperties);
- SessionState sessionState = setupQueryCompactionSession(context.getConf(),
- context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
try {
- DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
+ DriverUtils.runOnDriver(conf, sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
+ return true;
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
- throw new RuntimeException(e);
+ LOG.error("Failed compacting table {}", compactTableName, e);
+ throw e;
} finally {
sessionState.setCompaction(false);
}
+ }
+
+ private String partDataToSQL(PartitionData partitionData, String partSpec, String delimiter)
+ throws HiveException {
+
+ try {
+ Map<String, String> partSpecMap = Warehouse.makeSpecFromName(partSpec);
+ StringBuilder sb = new StringBuilder();
+
+ for (int i = 0; i < partitionData.size(); ++i) {
+ if (i > 0) {
+ sb.append(delimiter);
+ }
+
+ String quoteOpt = "";
+ if (partitionData.getType(i).typeId() == Type.TypeID.STRING ||
+ partitionData.getType(i).typeId() == Type.TypeID.DATE ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIME ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIMESTAMP ||
+ partitionData.getType(i).typeId() == Type.TypeID.BINARY) {
+ quoteOpt = "'";
+ }
- return true;
+ String fieldName = partitionData.getSchema().getFields().get(i).name();
+
+ sb.append(fieldName)
+ .append("=")
+ .append(quoteOpt)
+ .append(partSpecMap.get(fieldName))
+ .append(quoteOpt);
Review Comment:
how about
````
Map<String, String> partSpecMap = new LinkedHashMap<>();
Warehouse.makeSpecFromName(partSpecMap, new Path(partSpec), null);
List<FieldSchema> partitionKeys = hiveTable.getStorageHandler().getPartitionKeys(hiveTable);
List<String> partValues = partitionKeys.stream().map(
fs -> String.join("=", HiveUtils.unparseIdentifier(fs.getName()),
TypeInfoUtils.convertStringToLiteralForSQL(partSpecMap.get(fs.getName()),
((PrimitiveTypeInfo) TypeInfoUtils.getTypeInfoFromTypeString(fs.getType())).getPrimitiveCategory())
)
).collect(Collectors.toList());
String queryFields = hiveTable.getCols().stream().filter(col -> !partSpecMap.containsKey(col.getName()))
.map(FieldSchema::getName)
.collect(Collectors.joining(","));
compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select %4$s from %1$s where %3$s",
compactTableName,
StringUtils.join(partValues, ","),
StringUtils.join(partValues, " and "),
queryFields);
}
````
table.getStorageHandler().getColumnInfo looks expensive and unnecessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2073591787
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [12 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1575838382
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +55,71 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ Table table = IcebergTableUtil.getTable(conf, context.getTable());
+ List<String> partFields = table.spec().fields().stream().map(PartitionField::name).collect(Collectors.toList());
+ String queryFields = table.schema().columns().stream().map(Types.NestedField::name)
+ .filter(f -> !partFields.contains(f)).collect(Collectors.joining(","));
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select %4$s from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "), queryFields);
+ }
+
+ SessionState sessionState = setupQueryCompactionSession(conf, context.getCompactionInfo(), tblProperties);
- SessionState sessionState = setupQueryCompactionSession(context.getConf(),
- context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
try {
- DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
+ DriverUtils.runOnDriver(conf, sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
+ return true;
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
- throw new RuntimeException(e);
+ LOG.error("Failed compacting table {}", compactTableName, e);
+ throw e;
} finally {
sessionState.setCompaction(false);
}
+ }
+
+ private String partDataToSQL(PartitionData partitionData, String partSpec, String delimiter)
+ throws HiveException {
+
+ try {
+ Map<String, String> partSpecMap = Warehouse.makeSpecFromName(partSpec);
+ StringBuilder sb = new StringBuilder();
+
+ for (int i = 0; i < partitionData.size(); ++i) {
+ if (i > 0) {
+ sb.append(delimiter);
+ }
+
+ String quoteOpt = "";
+ if (partitionData.getType(i).typeId() == Type.TypeID.STRING ||
+ partitionData.getType(i).typeId() == Type.TypeID.DATE ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIME ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIMESTAMP ||
+ partitionData.getType(i).typeId() == Type.TypeID.BINARY) {
+ quoteOpt = "'";
+ }
- return true;
+ String fieldName = partitionData.getSchema().getFields().get(i).name();
+
+ sb.append(fieldName)
+ .append("=")
+ .append(quoteOpt)
+ .append(partSpecMap.get(fieldName))
+ .append(quoteOpt);
Review Comment:
how about
````
PrimitiveObjectInspector primitiveObjInspector = IcebergObjectInspector.primitive(partitionData.getType(i))
TypeInfoUtils.convertStringToLiteralForSQL(value, primitiveObjInspector.getPrimitiveCategory())
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1575838382
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +55,71 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ Table table = IcebergTableUtil.getTable(conf, context.getTable());
+ List<String> partFields = table.spec().fields().stream().map(PartitionField::name).collect(Collectors.toList());
+ String queryFields = table.schema().columns().stream().map(Types.NestedField::name)
+ .filter(f -> !partFields.contains(f)).collect(Collectors.joining(","));
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select %4$s from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "), queryFields);
+ }
+
+ SessionState sessionState = setupQueryCompactionSession(conf, context.getCompactionInfo(), tblProperties);
- SessionState sessionState = setupQueryCompactionSession(context.getConf(),
- context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
try {
- DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
+ DriverUtils.runOnDriver(conf, sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
+ return true;
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
- throw new RuntimeException(e);
+ LOG.error("Failed compacting table {}", compactTableName, e);
+ throw e;
} finally {
sessionState.setCompaction(false);
}
+ }
+
+ private String partDataToSQL(PartitionData partitionData, String partSpec, String delimiter)
+ throws HiveException {
+
+ try {
+ Map<String, String> partSpecMap = Warehouse.makeSpecFromName(partSpec);
+ StringBuilder sb = new StringBuilder();
+
+ for (int i = 0; i < partitionData.size(); ++i) {
+ if (i > 0) {
+ sb.append(delimiter);
+ }
+
+ String quoteOpt = "";
+ if (partitionData.getType(i).typeId() == Type.TypeID.STRING ||
+ partitionData.getType(i).typeId() == Type.TypeID.DATE ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIME ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIMESTAMP ||
+ partitionData.getType(i).typeId() == Type.TypeID.BINARY) {
+ quoteOpt = "'";
+ }
- return true;
+ String fieldName = partitionData.getSchema().getFields().get(i).name();
+
+ sb.append(fieldName)
+ .append("=")
+ .append(quoteOpt)
+ .append(partSpecMap.get(fieldName))
+ .append(quoteOpt);
Review Comment:
how about
````
Map<String, String> partSpecMap = new LinkedHashMap<>();
Warehouse.makeSpecFromName(partSpecMap, new Path(partSpec), null);
List<FieldSchema> partitionKeys = table.getStorageHandler().getPartitionKeys(table);
List<String> partValues = partitionKeys.stream().map(
fs -> String.join("=", HiveUtils.unparseIdentifier(fs.getName()),
TypeInfoUtils.convertStringToLiteralForSQL(partSpecMap.get(fs.getName()),
((PrimitiveTypeInfo) TypeInfoUtils.getTypeInfoFromTypeString(fs.getType())).getPrimitiveCategory())
)
).collect(Collectors.toList());
String queryFields = table.getCols().stream()
.map(FieldSchema::getName)
.filter(col -> !partSpecMap.containsKey(col))
.collect(Collectors.joining(","));
compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select %4$s from %1$s where %3$s",
compactTableName,
StringUtils.join(partValues, ","),
StringUtils.join(partValues, " and "),
queryFields);
````
table.getStorageHandler().getColumnInfo looks expensive and unnecessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1569012829
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +51,69 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ Table table = IcebergTableUtil.getTable(conf, context.getTable());
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select * from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "));
+ }
+
+ SessionState sessionState = setupQueryCompactionSession(conf, context.getCompactionInfo(), tblProperties);
+ sessionState.setConf(conf);
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1569215581
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/compact/AlterTableCompactOperation.java:
##########
@@ -133,7 +140,7 @@ private List<Partition> getPartitions(Table table, AlterTableCompactDesc desc, D
}
} else {
Map<String, String> partitionSpec = desc.getPartitionSpec();
- partitions = context.getDb().getPartitions(table, partitionSpec);
+ partitions.addAll(PartitionUtils.getPartitions(context.getDb(), table, partitionSpec, false));
Review Comment:
This is existing functionality which I didn't change.
I think it calls getPartitions on purpose to validate that only one partition is retrieved with the provided spec.
On the next lines it checks that only 1 partition was found for the partition spec and throws an exception otherwise.
```
if (partitions.size() > 1) {
throw new HiveException(ErrorMsg.TOO_MANY_COMPACTION_PARTITIONS);
} else if (partitions.isEmpty()) {
throw new HiveException(ErrorMsg.INVALID_PARTITION_SPEC);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1571017320
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/compact/AlterTableCompactOperation.java:
##########
@@ -62,6 +65,12 @@ public AlterTableCompactOperation(DDLOperationContext context, AlterTableCompact
throw new HiveException(ErrorMsg.NONACID_COMPACTION_NOT_SUPPORTED, table.getDbName(), table.getTableName());
}
+ if (desc.getPartitionSpec() != null && DDLUtils.isIcebergTable(table) &&
Review Comment:
could we move this logic into AlterTableUtils?
````
public boolean isIdentityPartitionSpec(table) {
return table.getStorageHandler() != null && table.getStorageHandler().getPartitionTransformSpec(table)
.stream().map(TransformSpec::getTransformType)
.allMatch(type -> type == TransformSpec.TransformType.IDENTITY);
````
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1573856606
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -750,6 +750,11 @@ default boolean canUseTruncate(org.apache.hadoop.hive.ql.metadata.Table hmsTable
return true;
}
+ default boolean isEligibilityForCompaction(org.apache.hadoop.hive.ql.metadata.Table table,
Review Comment:
typo: isEligibleForCompaction, default should be false
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1569209670
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/PartitionUtils.java:
##########
@@ -108,11 +122,17 @@ public static Partition getPartition(Hive db, Table table, Map<String, String> p
public static List<Partition> getPartitions(Hive db, Table table, Map<String, String> partitionSpec,
boolean throwException) throws SemanticException {
List<Partition> partitions;
- try {
- partitions = partitionSpec == null ? db.getPartitions(table) : db.getPartitions(table, partitionSpec);
- } catch (Exception e) {
- throw new SemanticException(toMessage(ErrorMsg.INVALID_PARTITION, partitionSpec), e);
- }
+ if (table.getStorageHandler() != null && table.getStorageHandler().alwaysUnpartitioned()) {
+ partitions = new ArrayList<>();
+ partitions.add(getPartition(db, table, partitionSpec, throwException));
Review Comment:
Yes, it is used in case of compactions.
It is called from AlterTableCompactOperation.
It is used to parallelize compaction of partitioned table - when partition spec is not provided it creates one compaction request per partition. For now it is not done for Iceberg, but will be useful in next PRs in which we will implement it for Iceberg too.
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -2127,4 +2127,26 @@ public List<FileStatus> getMergeTaskInputFiles(Properties properties) throws IOE
public MergeTaskProperties getMergeTaskProperties(Properties properties) {
return new IcebergMergeTaskProperties(properties);
}
+
+ @Override
+ public boolean isIdentityPartitionSpec(org.apache.hadoop.hive.metastore.api.Table table,
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1570846660
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##########
@@ -8815,6 +8815,23 @@ These props are now enabled elsewhere (see commit diffs). It would be better in
conf.set(AcidUtils.CONF_ACID_KEY, "true");
SessionState.get().getConf().set(AcidUtils.CONF_ACID_KEY, "true");
}
+
+ private void validateColumnCounts(String dest, Table table, Map<String, String> partitionSpec,
+ int inColumnCnt, int outColumnCnt) throws SemanticException {
+
+ long staticPartitionColumnCnt = partitionSpec != null ?
+ partitionSpec.values().stream().filter(Objects::nonNull).count() : 0;
+
+ if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
Review Comment:
````
if (!alwaysUnpartitioned ? inColumnCnt + staticPartitionColumnCnt != outColumnCnt
: inColumnCnt != outColumnCnt) {
,,,
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2075460681
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [10 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2060176311
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [10 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2060015039
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [10 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1565474973
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##########
@@ -569,14 +575,25 @@ private void commitWrite(Table table, String branchName, Long snapshotId, long s
* @param rewritePolicy The rewrite policy to use for the insert overwrite commit
*/
private void commitOverwrite(Table table, String branchName, long startTime, FilesForCommit results,
- RewritePolicy rewritePolicy) {
+ RewritePolicy rewritePolicy, String partitionSpec) {
Preconditions.checkArgument(results.deleteFiles().isEmpty(), "Can not handle deletes with overwrite");
if (!results.dataFiles().isEmpty()) {
Transaction transaction = table.newTransaction();
if (rewritePolicy == RewritePolicy.ALL_PARTITIONS) {
Review Comment:
do we need a rewrite policy, since we can check for `partitionSpec.isEmpty()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2062962153
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [10 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2061644070
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [10 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1569532221
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##########
@@ -8810,7 +8810,8 @@ private Operator genConversionSelectOperatorByAddPartition(String dest, QB qb, O
List<ExprNodeDesc> expressions = new ArrayList<>(outColumnCnt);
AtomicBoolean convert = new AtomicBoolean(false);
- if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt) {
+ if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
Review Comment:
Moved this validation to a separate method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2026305280
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [17 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1570827056
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/compact/AlterTableCompactOperation.java:
##########
@@ -133,7 +140,7 @@ private List<Partition> getPartitions(Table table, AlterTableCompactDesc desc, D
}
} else {
Map<String, String> partitionSpec = desc.getPartitionSpec();
- partitions = context.getDb().getPartitions(table, partitionSpec);
+ partitions.addAll(PartitionUtils.getPartitions(context.getDb(), table, partitionSpec, false));
Review Comment:
is that used for the scenarios like below, when we provide partial spec?
````
/country=Ukraine/city=Kharkiv
/country=Ukraine/city=Kyiv
````
alter table T partition(country=Ukraine) compact 'MAJOR'
how would we handle this in Iceberg ATM? Pick the first one or try to compact on country=Ukraine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1575084143
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/PartitionUtils.java:
##########
@@ -108,10 +122,18 @@ public static Partition getPartition(Hive db, Table table, Map<String, String> p
public static List<Partition> getPartitions(Hive db, Table table, Map<String, String> partitionSpec,
boolean throwException) throws SemanticException {
List<Partition> partitions;
- try {
- partitions = partitionSpec == null ? db.getPartitions(table) : db.getPartitions(table, partitionSpec);
- } catch (Exception e) {
- throw new SemanticException(toMessage(ErrorMsg.INVALID_PARTITION, partitionSpec), e);
+ if (table.getStorageHandler() != null && table.getStorageHandler().alwaysUnpartitioned()) {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1571421601
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/compact/AlterTableCompactOperation.java:
##########
@@ -62,6 +65,12 @@ public AlterTableCompactOperation(DDLOperationContext context, AlterTableCompact
throw new HiveException(ErrorMsg.NONACID_COMPACTION_NOT_SUPPORTED, table.getDbName(), table.getTableName());
}
+ if (desc.getPartitionSpec() != null && DDLUtils.isIcebergTable(table) &&
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1525195701
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##########
@@ -544,14 +549,20 @@ private void commitWrite(Table table, String branchName, Long snapshotId, long s
* @param rewritePolicy The rewrite policy to use for the insert overwrite commit
*/
private void commitOverwrite(Table table, String branchName, long startTime, FilesForCommit results,
- RewritePolicy rewritePolicy) {
+ RewritePolicy rewritePolicy, String partitionSpec) {
Preconditions.checkArgument(results.deleteFiles().isEmpty(), "Can not handle deletes with overwrite");
if (!results.dataFiles().isEmpty()) {
Transaction transaction = table.newTransaction();
if (rewritePolicy == RewritePolicy.ALL_PARTITIONS) {
DeleteFiles delete = transaction.newDelete();
delete.deleteFromRowFilter(Expressions.alwaysTrue());
delete.commit();
+ } else if (rewritePolicy == RewritePolicy.SINGLE_PARTITION) {
Review Comment:
should it be `rewritePolicy != RewritePolicy.ALL_PARTITIONS`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1530212442
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java:
##########
@@ -932,7 +934,9 @@ public void setDbName(String databaseName) {
}
public List<FieldSchema> getPartitionKeys() {
- return tTable.getPartitionKeys();
+ return getStorageHandler() != null && getStorageHandler().alwaysUnpartitioned() &&
+ SessionState.getSessionConf().get(Context.operationNameConf, StringUtils.EMPTY).equals(AlterTableType.COMPACT.getName()) ?
Review Comment:
why do you need to check operationType here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1532252364
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java:
##########
@@ -640,7 +640,13 @@ public void preTruncateTable(org.apache.hadoop.hive.metastore.api.Table table, E
throws MetaException {
this.catalogProperties = getCatalogProperties(table);
this.icebergTable = Catalogs.loadTable(conf, catalogProperties);
- Map<String, PartitionField> partitionFieldMap = icebergTable.spec().fields().stream()
+ truncatePartitionBySpec(icebergTable, partNames, icebergTable.newDelete());
+ context.putToProperties("truncateSkipDataDeletion", "true");
Review Comment:
could we please use `TRUNCATE_SKIP_DATA_DELETION` constant here. btw, is that used in case of Iceberg?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1532861769
##########
ql/src/java/org/apache/hadoop/hive/ql/Context.java:
##########
@@ -117,6 +117,7 @@ public class Context {
protected String optimizedSql;
protected String calcitePlan;
protected String cmd = "";
+ public static String compactPartition = "hive.iceberg.compaction.partition";
Review Comment:
Done, moved to AlterTableCompactOperation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1531086583
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java:
##########
@@ -932,7 +934,9 @@ public void setDbName(String databaseName) {
}
public List<FieldSchema> getPartitionKeys() {
- return tTable.getPartitionKeys();
+ return getStorageHandler() != null && getStorageHandler().alwaysUnpartitioned() &&
+ SessionState.getSessionConf().get(Context.operationNameConf, StringUtils.EMPTY).equals(AlterTableType.COMPACT.getName()) ?
Review Comment:
Removed this change, not needed anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1535400489
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##########
@@ -8810,7 +8810,8 @@ private Operator genConversionSelectOperatorByAddPartition(String dest, QB qb, O
List<ExprNodeDesc> expressions = new ArrayList<>(outColumnCnt);
AtomicBoolean convert = new AtomicBoolean(false);
- if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt) {
+ if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
Review Comment:
previously this check was done regardless of `alwaysUnpartitioned` flag and now we moved it inside the if?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1572036781
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -750,6 +750,10 @@ default boolean canUseTruncate(org.apache.hadoop.hive.ql.metadata.Table hmsTable
return true;
}
+ default boolean hasUndergonePartitionEvolution(org.apache.hadoop.hive.ql.metadata.Table table) {
Review Comment:
don't expose it via API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1575794738
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -1915,6 +1917,46 @@ private boolean hasUndergonePartitionEvolution(Table table) {
.anyMatch(id -> id < table.spec().specId());
}
+ private boolean isIdentityPartitionTable(org.apache.hadoop.hive.ql.metadata.Table table) {
+ return getPartitionTransformSpec(table).stream().map(TransformSpec::getTransformType)
+ .allMatch(type -> type == TransformSpec.TransformType.IDENTITY);
+ }
+
+ @Override
+ public org.apache.commons.lang3.tuple.Pair<Boolean, ErrorMsg> isEligibleForCompaction(
+ org.apache.hadoop.hive.ql.metadata.Table table, Map<String, String> partitionSpec) {
+ if (partitionSpec != null) {
+ Table icebergTable = IcebergTableUtil.getTable(conf, table.getTTable());
+ if (hasUndergonePartitionEvolution(icebergTable)) {
+ return org.apache.commons.lang3.tuple.Pair.of(false, ErrorMsg.COMPACTION_PARTITION_EVOLUTION);
+ }
+ if (!isIdentityPartitionTable(table)) {
+ return org.apache.commons.lang3.tuple.Pair.of(false, ErrorMsg.COMPACTION_NON_IDENTITY_PARTITION_SPEC);
+ }
+ }
+ return org.apache.commons.lang3.tuple.Pair.of(true, null);
+ }
+
+ @Override
+ public List<Partition> getPartitions(org.apache.hadoop.hive.ql.metadata.Table table,
+ Map<String, String> partitionSpec, short limit) throws SemanticException {
Review Comment:
why do we need limit here? if we need it, we should pass it to `getPartitionNames`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1577440548
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -777,6 +783,16 @@ default List<Partition> getPartitionsByExpr(org.apache.hadoop.hive.ql.metadata.T
"for a table.");
}
+ default List<Partition> getPartitions(org.apache.hadoop.hive.ql.metadata.Table table,
+ Map<String, String> partitionSpec) throws SemanticException {
+ throw new UnsupportedOperationException("Storage handler does not support getting partitions for a table.");
+ }
+
+ default Partition getPartition(org.apache.hadoop.hive.ql.metadata.Table table, Map<String, String> partSpec)
Review Comment:
`partitionSpec`
please keep param names consistent, that is an interface. also would be nice to add javadocs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1575794738
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -1915,6 +1917,46 @@ private boolean hasUndergonePartitionEvolution(Table table) {
.anyMatch(id -> id < table.spec().specId());
}
+ private boolean isIdentityPartitionTable(org.apache.hadoop.hive.ql.metadata.Table table) {
+ return getPartitionTransformSpec(table).stream().map(TransformSpec::getTransformType)
+ .allMatch(type -> type == TransformSpec.TransformType.IDENTITY);
+ }
+
+ @Override
+ public org.apache.commons.lang3.tuple.Pair<Boolean, ErrorMsg> isEligibleForCompaction(
+ org.apache.hadoop.hive.ql.metadata.Table table, Map<String, String> partitionSpec) {
+ if (partitionSpec != null) {
+ Table icebergTable = IcebergTableUtil.getTable(conf, table.getTTable());
+ if (hasUndergonePartitionEvolution(icebergTable)) {
+ return org.apache.commons.lang3.tuple.Pair.of(false, ErrorMsg.COMPACTION_PARTITION_EVOLUTION);
+ }
+ if (!isIdentityPartitionTable(table)) {
+ return org.apache.commons.lang3.tuple.Pair.of(false, ErrorMsg.COMPACTION_NON_IDENTITY_PARTITION_SPEC);
+ }
+ }
+ return org.apache.commons.lang3.tuple.Pair.of(true, null);
+ }
+
+ @Override
+ public List<Partition> getPartitions(org.apache.hadoop.hive.ql.metadata.Table table,
+ Map<String, String> partitionSpec, short limit) throws SemanticException {
Review Comment:
why do we need limit here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2071276284
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [12 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1573873316
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +55,71 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ Table table = IcebergTableUtil.getTable(conf, context.getTable());
+ List<String> partFields = table.spec().fields().stream().map(PartitionField::name).collect(Collectors.toList());
+ String queryFields = table.schema().columns().stream().map(Types.NestedField::name)
+ .filter(f -> !partFields.contains(f)).collect(Collectors.joining(","));
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select %4$s from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "), queryFields);
+ }
+
+ SessionState sessionState = setupQueryCompactionSession(conf, context.getCompactionInfo(), tblProperties);
- SessionState sessionState = setupQueryCompactionSession(context.getConf(),
- context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
try {
- DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
+ DriverUtils.runOnDriver(conf, sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
+ return true;
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
- throw new RuntimeException(e);
+ LOG.error("Failed compacting table {}", compactTableName, e);
+ throw e;
} finally {
sessionState.setCompaction(false);
}
+ }
+
+ private String partDataToSQL(PartitionData partitionData, String partSpec, String delimiter)
+ throws HiveException {
+
+ try {
+ Map<String, String> partSpecMap = Warehouse.makeSpecFromName(partSpec);
+ StringBuilder sb = new StringBuilder();
+
+ for (int i = 0; i < partitionData.size(); ++i) {
+ if (i > 0) {
+ sb.append(delimiter);
+ }
+
+ String quoteOpt = "";
+ if (partitionData.getType(i).typeId() == Type.TypeID.STRING ||
+ partitionData.getType(i).typeId() == Type.TypeID.DATE ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIME ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIMESTAMP ||
+ partitionData.getType(i).typeId() == Type.TypeID.BINARY) {
+ quoteOpt = "'";
+ }
- return true;
+ String fieldName = partitionData.getSchema().getFields().get(i).name();
+
+ sb.append(fieldName)
+ .append("=")
+ .append(quoteOpt)
+ .append(partSpecMap.get(fieldName))
+ .append(quoteOpt);
Review Comment:
nit: remove extra space
also, you could simplify the code with
````
HiveUtils.unparseIdentifier(fieldName)
.append("=")
.append(TypeInfoUtils.convertStringToLiteralForSQL(
((PrimitiveTypeInfo) TypeInfoUtils.getTypeInfoFromTypeString(partitionData.getType(i).typeId())).getPrimitiveCategory()
````
check `TruncateTableAnalyzer.constructDeleteQuery`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2048372907
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [23 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-1986187329
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [16 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1526549267
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -2021,4 +2022,34 @@ public List<Partition> getPartitionsByExpr(org.apache.hadoop.hive.ql.metadata.Ta
throw new SemanticException(String.format("Error while fetching the partitions due to: %s", e));
}
}
+
+ @Override
+ public List<Partition> getPartitions(org.apache.hadoop.hive.ql.metadata.Table hmsTable, List<String> partSpecs) {
+
+ return partSpecs
+ .stream()
+ .map(spec -> {
+ Map<String, String> partSpecsMap = Arrays.stream(spec.split("/"))
+ .collect(Collectors.toMap(
+ x -> x.split("=")[0],
+ x -> x.split("=")[1]
+ ));
+ validatePartitionName(hmsTable, partSpecsMap);
+ return spec;
+ })
+ .map(partName -> new DummyPartition(hmsTable, partName))
+ .collect(Collectors.toList());
+ }
+
+ private void validatePartitionName(org.apache.hadoop.hive.ql.metadata.Table hmsTable, Map<String, String> partSpec) {
+ try {
+ List<String> partNames = getPartitionNames(hmsTable, partSpec);
+
+ if (partNames.isEmpty()) {
+ throw new RuntimeException("Failed getting partitions from partition spec");
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1525205302
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -2021,4 +2022,34 @@ public List<Partition> getPartitionsByExpr(org.apache.hadoop.hive.ql.metadata.Ta
throw new SemanticException(String.format("Error while fetching the partitions due to: %s", e));
}
}
+
+ @Override
+ public List<Partition> getPartitions(org.apache.hadoop.hive.ql.metadata.Table hmsTable, List<String> partSpecs) {
+
+ return partSpecs
+ .stream()
+ .map(spec -> {
+ Map<String, String> partSpecsMap = Arrays.stream(spec.split("/"))
+ .collect(Collectors.toMap(
+ x -> x.split("=")[0],
+ x -> x.split("=")[1]
+ ));
+ validatePartitionName(hmsTable, partSpecsMap);
+ return spec;
+ })
+ .map(partName -> new DummyPartition(hmsTable, partName))
+ .collect(Collectors.toList());
+ }
+
+ private void validatePartitionName(org.apache.hadoop.hive.ql.metadata.Table hmsTable, Map<String, String> partSpec) {
+ try {
+ List<String> partNames = getPartitionNames(hmsTable, partSpec);
+
+ if (partNames.isEmpty()) {
+ throw new RuntimeException("Failed getting partitions from partition spec");
Review Comment:
do we have some more specific exceptions in Hive? also why do we need this name validation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2008196909
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [18 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1570827056
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/compact/AlterTableCompactOperation.java:
##########
@@ -133,7 +140,7 @@ private List<Partition> getPartitions(Table table, AlterTableCompactDesc desc, D
}
} else {
Map<String, String> partitionSpec = desc.getPartitionSpec();
- partitions = context.getDb().getPartitions(table, partitionSpec);
+ partitions.addAll(PartitionUtils.getPartitions(context.getDb(), table, partitionSpec, false));
Review Comment:
is that used for the scenarios like below, when we provide partial spec?
````
/country=Ukraine/city=Kharkiv
/country=Ukraine/city=Kyiv
````
alter table T partition(country=Ukraine) compact 'MAJOR'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1572912840
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##########
@@ -8815,6 +8815,23 @@ These props are now enabled elsewhere (see commit diffs). It would be better in
conf.set(AcidUtils.CONF_ACID_KEY, "true");
SessionState.get().getConf().set(AcidUtils.CONF_ACID_KEY, "true");
}
+
+ private void validateColumnCounts(String dest, Table table, Map<String, String> partitionSpec,
+ int inColumnCnt, int outColumnCnt) throws SemanticException {
+
+ long staticPartitionColumnCnt = partitionSpec != null ?
+ partitionSpec.values().stream().filter(Objects::nonNull).count() : 0;
+
+ if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
Review Comment:
Changed IOW query in Iceberg compactor class and to not include partition keys in the select query and now this change in SemanticAnalyzer is not needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1573866315
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/PartitionUtils.java:
##########
@@ -94,10 +98,20 @@ public static ExprNodeGenericFuncDesc makeUnaryPredicate(String fn, ExprNodeDesc
public static Partition getPartition(Hive db, Table table, Map<String, String> partitionSpec, boolean throwException)
throws SemanticException {
Partition partition;
- try {
- partition = db.getPartition(table, partitionSpec, false);
- } catch (Exception e) {
- throw new SemanticException(toMessage(ErrorMsg.INVALID_PARTITION, partitionSpec), e);
+ if (table.getStorageHandler() != null && table.getStorageHandler().alwaysUnpartitioned()) {
Review Comment:
that should be done in `Hive.getPartition()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1573858388
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/compact/AlterTableCompactOperation.java:
##########
@@ -133,7 +138,7 @@ private List<Partition> getPartitions(Table table, AlterTableCompactDesc desc, D
}
} else {
Map<String, String> partitionSpec = desc.getPartitionSpec();
- partitions = context.getDb().getPartitions(table, partitionSpec);
+ partitions.addAll(PartitionUtils.getPartitions(context.getDb(), table, partitionSpec, false));
Review Comment:
you should amend `Hive.getPartitions`, similar to `Hive.getPartitionsByExpr`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1571421601
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/compact/AlterTableCompactOperation.java:
##########
@@ -62,6 +65,12 @@ public AlterTableCompactOperation(DDLOperationContext context, AlterTableCompact
throw new HiveException(ErrorMsg.NONACID_COMPACTION_NOT_SUPPORTED, table.getDbName(), table.getTableName());
}
+ if (desc.getPartitionSpec() != null && DDLUtils.isIcebergTable(table) &&
Review Comment:
For the identity partition table validation, the suggested code works only for Iceberg tables, but not for native.
Modified to:
```
public static boolean isIdentityPartitionTable(Table table) {
return table.getStorageHandler() == null || table.getStorageHandler().getPartitionTransformSpec(table).stream()
.map(TransformSpec::getTransformType)
.allMatch(type -> type == TransformSpec.TransformType.IDENTITY);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1573084638
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/compact/AlterTableCompactOperation.java:
##########
@@ -62,6 +65,17 @@ public AlterTableCompactOperation(DDLOperationContext context, AlterTableCompact
throw new HiveException(ErrorMsg.NONACID_COMPACTION_NOT_SUPPORTED, table.getDbName(), table.getTableName());
}
+ if (desc.getPartitionSpec() != null && !AlterTableUtils.isIdentityPartitionTable(table)) {
+ throw new HiveException("Compaction is not supported on the table " +
+ table.getDbName() + "." + table.getTableName() + " with non-identity partition spec");
+ }
+
+ if (desc.getPartitionSpec() != null && DDLUtils.isIcebergTable(table) &&
Review Comment:
Done
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -750,6 +750,10 @@ default boolean canUseTruncate(org.apache.hadoop.hive.ql.metadata.Table hmsTable
return true;
}
+ default boolean hasUndergonePartitionEvolution(org.apache.hadoop.hive.ql.metadata.Table table) {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1573866442
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/PartitionUtils.java:
##########
@@ -108,10 +122,18 @@ public static Partition getPartition(Hive db, Table table, Map<String, String> p
public static List<Partition> getPartitions(Hive db, Table table, Map<String, String> partitionSpec,
boolean throwException) throws SemanticException {
List<Partition> partitions;
- try {
- partitions = partitionSpec == null ? db.getPartitions(table) : db.getPartitions(table, partitionSpec);
- } catch (Exception e) {
- throw new SemanticException(toMessage(ErrorMsg.INVALID_PARTITION, partitionSpec), e);
+ if (table.getStorageHandler() != null && table.getStorageHandler().alwaysUnpartitioned()) {
Review Comment:
same here, `Hive.getPartitions()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1570846660
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##########
@@ -8815,6 +8815,23 @@ These props are now enabled elsewhere (see commit diffs). It would be better in
conf.set(AcidUtils.CONF_ACID_KEY, "true");
SessionState.get().getConf().set(AcidUtils.CONF_ACID_KEY, "true");
}
+
+ private void validateColumnCounts(String dest, Table table, Map<String, String> partitionSpec,
+ int inColumnCnt, int outColumnCnt) throws SemanticException {
+
+ long staticPartitionColumnCnt = partitionSpec != null ?
+ partitionSpec.values().stream().filter(Objects::nonNull).count() : 0;
+
+ if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
Review Comment:
````
if (!alwaysUnpartitioned ? inColumnCnt + staticPartitionColumnCnt != outColumnCnt
: inColumnCnt != outColumnCnt) {
,,,
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1568956730
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +51,69 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ Table table = IcebergTableUtil.getTable(conf, context.getTable());
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select * from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "));
+ }
+
+ SessionState sessionState = setupQueryCompactionSession(conf, context.getCompactionInfo(), tblProperties);
+ sessionState.setConf(conf);
Review Comment:
won't that override the conf setup in setupQueryCompactionSession
````
Util.disableLlapCaching(conf);
conf.set(HiveConf.ConfVars.HIVE_QUOTEDID_SUPPORT.varname, "column");
conf.setBoolVar(HiveConf.ConfVars.HIVE_SERVER2_ENABLE_DOAS, true);
conf.setBoolVar(HiveConf.ConfVars.HIVE_HDFS_ENCRYPTION_SHIM_CACHE_ON, false);
Util.overrideConfProps(conf, compactionInfo, tblProperties);
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1561329346
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -21,45 +21,159 @@
import java.io.IOException;
import java.util.Map;
+import jline.internal.Log;
import org.apache.hadoop.hive.conf.HiveConf;
import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.metastore.Warehouse;
+import org.apache.hadoop.hive.metastore.api.MetaException;
import org.apache.hadoop.hive.ql.Context.RewritePolicy;
import org.apache.hadoop.hive.ql.DriverUtils;
import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.parse.AlterTableSnapshotRefSpec;
import org.apache.hadoop.hive.ql.session.SessionState;
import org.apache.hadoop.hive.ql.txn.compactor.CompactorContext;
import org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor;
import org.apache.hive.iceberg.org.apache.orc.storage.common.TableName;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.PartitionData;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter;
+import org.apache.iceberg.mr.hive.IcebergBranchExec;
+import org.apache.iceberg.mr.hive.IcebergTableUtil;
+import org.apache.iceberg.types.Type;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class IcebergMajorQueryCompactor extends QueryCompactor {
private static final Logger LOG = LoggerFactory.getLogger(IcebergMajorQueryCompactor.class.getName());
+ private static final String branchName = "compaction";
+ private String compactTableName;
@Override
public boolean run(CompactorContext context) throws IOException, HiveException, InterruptedException {
- String compactTableName = TableName.getDbTable(context.getTable().getDbName(), context.getTable().getTableName());
+ compactTableName = TableName.getDbTable(context.getTable().getDbName(), context.getTable().getTableName());
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
SessionState sessionState = setupQueryCompactionSession(context.getConf(),
context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+
try {
- DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
- LOG.info("Completed compaction for table {}", compactTableName);
- } catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
- throw new RuntimeException(e);
+ if (partSpec == null) {
+ HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+
+ executeHiveQuery(context, compactionQuery, sessionState,
+ String.format("compacting table %s", compactTableName));
+ } else {
+ Table table = IcebergTableUtil.getTable(context.getConf(), context.getTable());
+ AlterTableSnapshotRefSpec.DropSnapshotRefSpec dropBranchSpec =
+ new AlterTableSnapshotRefSpec.DropSnapshotRefSpec(branchName, true);
+ IcebergBranchExec.dropBranch(table, dropBranchSpec);
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1565469401
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java:
##########
@@ -640,7 +640,13 @@ public void preTruncateTable(org.apache.hadoop.hive.metastore.api.Table table, E
throws MetaException {
this.catalogProperties = getCatalogProperties(table);
this.icebergTable = Catalogs.loadTable(conf, catalogProperties);
- Map<String, PartitionField> partitionFieldMap = icebergTable.spec().fields().stream()
+ truncatePartitionBySpec(icebergTable, partNames, icebergTable.newDelete());
+ context.putToProperties("truncateSkipDataDeletion", "true");
Review Comment:
i don't see constant used here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1565694331
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +52,68 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.DEFAULT.name());
+ Table table = IcebergTableUtil.getTable(context.getConf(), context.getTable());
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ context.getConf().set(Context.compactPartitionSpec, partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select * from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "));
+ }
SessionState sessionState = setupQueryCompactionSession(context.getConf(),
context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+
try {
DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
+ return true;
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
- throw new RuntimeException(e);
+ LOG.error("Failed compacting table {}", compactTableName, e);
+ throw e;
} finally {
sessionState.setCompaction(false);
}
+ }
+
+ private String partDataToSQL(PartitionData partitionData, String partSpec, String delimiter)
Review Comment:
you should use logic from `truncatePartitionBySpec` as you can't override partition with non identity spec
````
if (!TransformSpec.TransformType.IDENTITY.equals(transformType)) {
throw new MetaException(
String.format("Partition transforms are not supported in compact operation: %s", entry.getKey()));
}
````
but should be done before we serialize the request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1565823941
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##########
@@ -569,14 +575,25 @@ private void commitWrite(Table table, String branchName, Long snapshotId, long s
* @param rewritePolicy The rewrite policy to use for the insert overwrite commit
*/
private void commitOverwrite(Table table, String branchName, long startTime, FilesForCommit results,
- RewritePolicy rewritePolicy) {
+ RewritePolicy rewritePolicy, String partitionSpec) {
Preconditions.checkArgument(results.deleteFiles().isEmpty(), "Can not handle deletes with overwrite");
if (!results.dataFiles().isEmpty()) {
Transaction transaction = table.newTransaction();
if (rewritePolicy == RewritePolicy.ALL_PARTITIONS) {
DeleteFiles delete = transaction.newDelete();
delete.deleteFromRowFilter(Expressions.alwaysTrue());
delete.commit();
+ LOG.info("Overwrite commit: truncated all partitions for table: {} with {} file(s)",
+ table, results.dataFiles().size());
+ } else if (!partitionSpec.isEmpty()) {
Review Comment:
This is not for partition evolution case. This is for compaction on partition level without evolution. The partition needs to be truncated before adding compacted files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1576332462
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -1915,6 +1917,46 @@ private boolean hasUndergonePartitionEvolution(Table table) {
.anyMatch(id -> id < table.spec().specId());
}
+ private boolean isIdentityPartitionTable(org.apache.hadoop.hive.ql.metadata.Table table) {
+ return getPartitionTransformSpec(table).stream().map(TransformSpec::getTransformType)
+ .allMatch(type -> type == TransformSpec.TransformType.IDENTITY);
+ }
+
+ @Override
+ public org.apache.commons.lang3.tuple.Pair<Boolean, ErrorMsg> isEligibleForCompaction(
+ org.apache.hadoop.hive.ql.metadata.Table table, Map<String, String> partitionSpec) {
+ if (partitionSpec != null) {
+ Table icebergTable = IcebergTableUtil.getTable(conf, table.getTTable());
+ if (hasUndergonePartitionEvolution(icebergTable)) {
+ return org.apache.commons.lang3.tuple.Pair.of(false, ErrorMsg.COMPACTION_PARTITION_EVOLUTION);
+ }
+ if (!isIdentityPartitionTable(table)) {
+ return org.apache.commons.lang3.tuple.Pair.of(false, ErrorMsg.COMPACTION_NON_IDENTITY_PARTITION_SPEC);
+ }
+ }
+ return org.apache.commons.lang3.tuple.Pair.of(true, null);
+ }
+
+ @Override
+ public List<Partition> getPartitions(org.apache.hadoop.hive.ql.metadata.Table table,
+ Map<String, String> partitionSpec, short limit) throws SemanticException {
+ return table.getStorageHandler().getPartitionNames(table, partitionSpec).stream()
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1576572891
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -1915,6 +1917,46 @@ private boolean hasUndergonePartitionEvolution(Table table) {
.anyMatch(id -> id < table.spec().specId());
}
+ private boolean isIdentityPartitionTable(org.apache.hadoop.hive.ql.metadata.Table table) {
+ return getPartitionTransformSpec(table).stream().map(TransformSpec::getTransformType)
+ .allMatch(type -> type == TransformSpec.TransformType.IDENTITY);
+ }
+
+ @Override
+ public org.apache.commons.lang3.tuple.Pair<Boolean, ErrorMsg> isEligibleForCompaction(
+ org.apache.hadoop.hive.ql.metadata.Table table, Map<String, String> partitionSpec) {
+ if (partitionSpec != null) {
+ Table icebergTable = IcebergTableUtil.getTable(conf, table.getTTable());
+ if (hasUndergonePartitionEvolution(icebergTable)) {
+ return org.apache.commons.lang3.tuple.Pair.of(false, ErrorMsg.COMPACTION_PARTITION_EVOLUTION);
+ }
+ if (!isIdentityPartitionTable(table)) {
+ return org.apache.commons.lang3.tuple.Pair.of(false, ErrorMsg.COMPACTION_NON_IDENTITY_PARTITION_SPEC);
+ }
+ }
+ return org.apache.commons.lang3.tuple.Pair.of(true, null);
+ }
+
+ @Override
+ public List<Partition> getPartitions(org.apache.hadoop.hive.ql.metadata.Table table,
+ Map<String, String> partitionSpec, short limit) throws SemanticException {
+ return table.getStorageHandler().getPartitionNames(table, partitionSpec).stream()
+ .limit(limit > 0 ? limit : Long.MAX_VALUE)
+ .map(partName -> new DummyPartition(table, partName, partitionSpec)).collect(Collectors.toList());
+ }
+
+ @Override
+ public Partition getPartition(org.apache.hadoop.hive.ql.metadata.Table table,
+ Map<String, String> partitionSpec) throws SemanticException {
+ validatePartSpec(table, partitionSpec);
+ try {
+ String partName = Warehouse.makePartName(partitionSpec, false);
+ return new DummyPartition(table, partName, partitionSpec);
+ } catch (MetaException e) {
Review Comment:
The line above also throws SemanticException (as discussed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1576860570
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +55,51 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ Hive db = Hive.get(conf);
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1577964336
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +56,52 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ org.apache.hadoop.hive.ql.metadata.Table table = Hive.get(conf).getTable(context.getTable().getDbName(),
+ context.getTable().getTableName());
+ Map<String, String> partSpecMap = new LinkedHashMap<>();
+ Warehouse.makeSpecFromName(partSpecMap, new Path(partSpec), null);
+
+ List<FieldSchema> partitionKeys = table.getStorageHandler().getPartitionKeys(table);
+ List<String> partValues = partitionKeys.stream().map(
+ fs -> String.join("=", HiveUtils.unparseIdentifier(fs.getName()),
+ TypeInfoUtils.convertStringToLiteralForSQL(partSpecMap.get(fs.getName()),
+ ((PrimitiveTypeInfo) TypeInfoUtils.getTypeInfoFromTypeString(fs.getType())).getPrimitiveCategory())
+ )
+ ).collect(Collectors.toList());
+
+ String queryFields = table.getCols().stream()
+ .map(FieldSchema::getName)
+ .filter(col -> !partSpecMap.containsKey(col))
+ .collect(Collectors.joining(","));
+
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select %4$s from %1$s where %3$s",
+ compactTableName,
+ StringUtils.join(partValues, ","),
+ StringUtils.join(partValues, " and "),
+ queryFields);
+ }
+
+ SessionState sessionState = setupQueryCompactionSession(conf, context.getCompactionInfo(), tblProperties);
+ String compactionTarget = "table " + compactTableName +
+ (Strings.nullToEmpty(partSpec).isEmpty() ? "" : ", partition " + partSpec);
Review Comment:
Done
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -777,6 +783,16 @@ default List<Partition> getPartitionsByExpr(org.apache.hadoop.hive.ql.metadata.T
"for a table.");
}
+ default List<Partition> getPartitions(org.apache.hadoop.hive.ql.metadata.Table table,
+ Map<String, String> partitionSpec) throws SemanticException {
+ throw new UnsupportedOperationException("Storage handler does not support getting partitions for a table.");
+ }
+
+ default Partition getPartition(org.apache.hadoop.hive.ql.metadata.Table table, Map<String, String> partSpec)
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1532252364
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java:
##########
@@ -640,7 +640,13 @@ public void preTruncateTable(org.apache.hadoop.hive.metastore.api.Table table, E
throws MetaException {
this.catalogProperties = getCatalogProperties(table);
this.icebergTable = Catalogs.loadTable(conf, catalogProperties);
- Map<String, PartitionField> partitionFieldMap = icebergTable.spec().fields().stream()
+ truncatePartitionBySpec(icebergTable, partNames, icebergTable.newDelete());
+ context.putToProperties("truncateSkipDataDeletion", "true");
Review Comment:
could we please use `TRUNCATE_SKIP_DATA_DELETION` constant here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1532360882
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##########
@@ -8810,7 +8810,8 @@ private Operator genConversionSelectOperatorByAddPartition(String dest, QB qb, O
List<ExprNodeDesc> expressions = new ArrayList<>(outColumnCnt);
AtomicBoolean convert = new AtomicBoolean(false);
- if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt) {
+ if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
Review Comment:
i don't think we should just ignore that check for iceberg, can we update/fix the `staticPartitionColumnCnt`?
cc @kasakrisz
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1532925990
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/AbstractBaseAlterTableAnalyzer.java:
##########
@@ -95,11 +98,16 @@ protected void addInputsOutputsAlterTable(TableName tableName, Map<String, Strin
// ReadEntity as no lock.
re.noLockNeeded();
inputs.add(re);
+ Partition partition;
if (AlterTableUtils.isFullPartitionSpec(table, partitionSpec)) {
// Fully specified partition spec
- Partition part = PartitionUtils.getPartition(db, table, partitionSpec, true);
- outputs.add(new WriteEntity(part, writeType));
+ if (table.getStorageHandler() != null && table.getStorageHandler().alwaysUnpartitioned()) {
Review Comment:
Done
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +53,68 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+ RewritePolicy rewritePolicy;
+
+ if (partSpec == null) {
+ compactionQuery = String.format("insert overwrite table %s select * from %<s",
+ compactTableName);
+ rewritePolicy = RewritePolicy.ALL_PARTITIONS;
+ } else {
+ Table table = IcebergTableUtil.getTable(context.getConf(), context.getTable());
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ context.getConf().set(Context.compactPartition, partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select * from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "));
+ rewritePolicy = RewritePolicy.SINGLE_PARTITION;
+ }
SessionState sessionState = setupQueryCompactionSession(context.getConf(),
context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, rewritePolicy.name());
try {
DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
+ LOG.error("Error doing query based {} compaction", rewritePolicy.name(), e);
throw new RuntimeException(e);
} finally {
sessionState.setCompaction(false);
}
return true;
}
+
+ private String partDataToSQL(PartitionData partitionData, String partSpec, String delimiter) {
+ StringBuilder sb = new StringBuilder();
+ List<String> values = Arrays
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2048827938
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [23 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2049201402
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [24 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1561226378
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -21,45 +21,159 @@
import java.io.IOException;
import java.util.Map;
+import jline.internal.Log;
import org.apache.hadoop.hive.conf.HiveConf;
import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.metastore.Warehouse;
+import org.apache.hadoop.hive.metastore.api.MetaException;
import org.apache.hadoop.hive.ql.Context.RewritePolicy;
import org.apache.hadoop.hive.ql.DriverUtils;
import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.parse.AlterTableSnapshotRefSpec;
import org.apache.hadoop.hive.ql.session.SessionState;
import org.apache.hadoop.hive.ql.txn.compactor.CompactorContext;
import org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor;
import org.apache.hive.iceberg.org.apache.orc.storage.common.TableName;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.PartitionData;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter;
+import org.apache.iceberg.mr.hive.IcebergBranchExec;
+import org.apache.iceberg.mr.hive.IcebergTableUtil;
+import org.apache.iceberg.types.Type;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class IcebergMajorQueryCompactor extends QueryCompactor {
private static final Logger LOG = LoggerFactory.getLogger(IcebergMajorQueryCompactor.class.getName());
+ private static final String branchName = "compaction";
+ private String compactTableName;
@Override
public boolean run(CompactorContext context) throws IOException, HiveException, InterruptedException {
- String compactTableName = TableName.getDbTable(context.getTable().getDbName(), context.getTable().getTableName());
+ compactTableName = TableName.getDbTable(context.getTable().getDbName(), context.getTable().getTableName());
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
SessionState sessionState = setupQueryCompactionSession(context.getConf(),
context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+
try {
- DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
- LOG.info("Completed compaction for table {}", compactTableName);
- } catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
- throw new RuntimeException(e);
+ if (partSpec == null) {
+ HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+
+ executeHiveQuery(context, compactionQuery, sessionState,
+ String.format("compacting table %s", compactTableName));
+ } else {
+ Table table = IcebergTableUtil.getTable(context.getConf(), context.getTable());
+ AlterTableSnapshotRefSpec.DropSnapshotRefSpec dropBranchSpec =
+ new AlterTableSnapshotRefSpec.DropSnapshotRefSpec(branchName, true);
+ IcebergBranchExec.dropBranch(table, dropBranchSpec);
Review Comment:
branch should be unique, you should be able execute number of compaction requests in parallel
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2043221329
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [15 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1565744058
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +52,68 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.DEFAULT.name());
+ Table table = IcebergTableUtil.getTable(context.getConf(), context.getTable());
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ context.getConf().set(Context.compactPartitionSpec, partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select * from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "));
+ }
SessionState sessionState = setupQueryCompactionSession(context.getConf(),
context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+
try {
DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
+ return true;
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
- throw new RuntimeException(e);
+ LOG.error("Failed compacting table {}", compactTableName, e);
+ throw e;
} finally {
sessionState.setCompaction(false);
}
+ }
+
+ private String partDataToSQL(PartitionData partitionData, String partSpec, String delimiter)
+ throws HiveException {
- return true;
+ try {
+ Map<String, String> partSpecMap = Warehouse.makeSpecFromName(partSpec);
+ StringBuilder sb = new StringBuilder();
+
+ for (int i = 0; i < partitionData.size(); ++i) {
+ if (i > 0) {
+ sb.append(delimiter);
+ }
+
+ String quoteOpt = "";
+ if (partitionData.getType(i).typeId() == Type.TypeID.STRING ||
+ partitionData.getType(i).typeId() == Type.TypeID.DATE ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIME ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIMESTAMP ||
+ partitionData.getType(i).typeId() == Type.TypeID.BINARY) {
+ quoteOpt = "'";
+ }
+
+ sb.append(partitionData.getSchema().getFields().get(i).name())
+ .append("=")
+ .append(quoteOpt)
+ .append(partSpecMap.get(partitionData.getPartitionType().fields().get(i).name()))
Review Comment:
should we use `partitionData.getSchema().getFields()` here as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1565694331
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +52,68 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.DEFAULT.name());
+ Table table = IcebergTableUtil.getTable(context.getConf(), context.getTable());
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ context.getConf().set(Context.compactPartitionSpec, partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select * from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "));
+ }
SessionState sessionState = setupQueryCompactionSession(context.getConf(),
context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+
try {
DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
+ return true;
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
- throw new RuntimeException(e);
+ LOG.error("Failed compacting table {}", compactTableName, e);
+ throw e;
} finally {
sessionState.setCompaction(false);
}
+ }
+
+ private String partDataToSQL(PartitionData partitionData, String partSpec, String delimiter)
Review Comment:
you should use logic from `truncatePartitionBySpec` as you can't override partition with non identity spec
````
if (!TransformSpec.TransformType.IDENTITY.equals(transformType)) {
throw new MetaException(
String.format("Partition transforms are not supported in compact operation: %s", entry.getKey()));
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2062798424
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [10 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1572011889
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##########
@@ -8815,6 +8815,23 @@ These props are now enabled elsewhere (see commit diffs). It would be better in
conf.set(AcidUtils.CONF_ACID_KEY, "true");
SessionState.get().getConf().set(AcidUtils.CONF_ACID_KEY, "true");
}
+
+ private void validateColumnCounts(String dest, Table table, Map<String, String> partitionSpec,
+ int inColumnCnt, int outColumnCnt) throws SemanticException {
+
+ long staticPartitionColumnCnt = partitionSpec != null ?
+ partitionSpec.values().stream().filter(Objects::nonNull).count() : 0;
+
+ if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
Review Comment:
i had the same code at the beginning, but it looks a bit weird if you split it
````
!alwaysUnpartitioned && inColumnCnt + staticPartitionColumnCnt != outColumnCnt
or
alwaysUnpartitioned && inColumnCnt + staticPartitionColumnCnt != outColumnCnt && inColumnCnt != outColumnCnt
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1568977615
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -2127,4 +2127,26 @@ public List<FileStatus> getMergeTaskInputFiles(Properties properties) throws IOE
public MergeTaskProperties getMergeTaskProperties(Properties properties) {
return new IcebergMergeTaskProperties(properties);
}
+
+ @Override
+ public boolean isIdentityPartitionSpec(org.apache.hadoop.hive.metastore.api.Table table,
Review Comment:
why should it be in API, could we move this to TableUtils?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1571193903
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##########
@@ -8815,6 +8815,23 @@ These props are now enabled elsewhere (see commit diffs). It would be better in
conf.set(AcidUtils.CONF_ACID_KEY, "true");
SessionState.get().getConf().set(AcidUtils.CONF_ACID_KEY, "true");
}
+
+ private void validateColumnCounts(String dest, Table table, Map<String, String> partitionSpec,
+ int inColumnCnt, int outColumnCnt) throws SemanticException {
+
+ long staticPartitionColumnCnt = partitionSpec != null ?
+ partitionSpec.values().stream().filter(Objects::nonNull).count() : 0;
+
+ if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
Review Comment:
Extracted the code into a local var, but about the second part, unfortunately it won't work because I saw that for Iceberg in some cases (q-tests) it can be `inColumnCnt + staticPartitionColumnCnt == outColumnCnt` where staticPartitionColumnCnt is > 1 and in other cases it can be ` inColumnCnt == outColumnCnt` and the suggested expression won't cover the first case.
Would you accept this instead?
```
if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
((alwaysUnpartitioned && inColumnCnt != outColumnCnt) || !alwaysUnpartitioned)) {
...
}
```
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##########
@@ -8815,6 +8815,23 @@ These props are now enabled elsewhere (see commit diffs). It would be better in
conf.set(AcidUtils.CONF_ACID_KEY, "true");
SessionState.get().getConf().set(AcidUtils.CONF_ACID_KEY, "true");
}
+
+ private void validateColumnCounts(String dest, Table table, Map<String, String> partitionSpec,
+ int inColumnCnt, int outColumnCnt) throws SemanticException {
+
+ long staticPartitionColumnCnt = partitionSpec != null ?
+ partitionSpec.values().stream().filter(Objects::nonNull).count() : 0;
+
+ if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
Review Comment:
Extracted the code into a local var, but about the second part, unfortunately it won't work because I saw that for Iceberg in some cases (q-tests) it can be `inColumnCnt + staticPartitionColumnCnt == outColumnCnt` where staticPartitionColumnCnt is > 0 and in other cases it can be ` inColumnCnt == outColumnCnt` and the suggested expression won't cover the first case.
Would you accept this instead?
```
if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
((alwaysUnpartitioned && inColumnCnt != outColumnCnt) || !alwaysUnpartitioned)) {
...
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1575779868
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -1915,6 +1917,46 @@ private boolean hasUndergonePartitionEvolution(Table table) {
.anyMatch(id -> id < table.spec().specId());
}
+ private boolean isIdentityPartitionTable(org.apache.hadoop.hive.ql.metadata.Table table) {
+ return getPartitionTransformSpec(table).stream().map(TransformSpec::getTransformType)
+ .allMatch(type -> type == TransformSpec.TransformType.IDENTITY);
+ }
+
+ @Override
+ public org.apache.commons.lang3.tuple.Pair<Boolean, ErrorMsg> isEligibleForCompaction(
+ org.apache.hadoop.hive.ql.metadata.Table table, Map<String, String> partitionSpec) {
+ if (partitionSpec != null) {
+ Table icebergTable = IcebergTableUtil.getTable(conf, table.getTTable());
+ if (hasUndergonePartitionEvolution(icebergTable)) {
+ return org.apache.commons.lang3.tuple.Pair.of(false, ErrorMsg.COMPACTION_PARTITION_EVOLUTION);
+ }
+ if (!isIdentityPartitionTable(table)) {
+ return org.apache.commons.lang3.tuple.Pair.of(false, ErrorMsg.COMPACTION_NON_IDENTITY_PARTITION_SPEC);
+ }
+ }
+ return org.apache.commons.lang3.tuple.Pair.of(true, null);
+ }
+
+ @Override
+ public List<Partition> getPartitions(org.apache.hadoop.hive.ql.metadata.Table table,
+ Map<String, String> partitionSpec, short limit) throws SemanticException {
+ return table.getStorageHandler().getPartitionNames(table, partitionSpec).stream()
+ .limit(limit > 0 ? limit : Long.MAX_VALUE)
Review Comment:
`Math.min(limit, Long.MAX_VALUE)` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1573856606
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -750,6 +750,11 @@ default boolean canUseTruncate(org.apache.hadoop.hive.ql.metadata.Table hmsTable
return true;
}
+ default boolean isEligibilityForCompaction(org.apache.hadoop.hive.ql.metadata.Table table,
Review Comment:
typo: isEligibleForCompaction
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1573884289
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -1915,6 +1915,28 @@ private boolean hasUndergonePartitionEvolution(Table table) {
.anyMatch(id -> id < table.spec().specId());
}
+ private boolean isIdentityPartitionTable(org.apache.hadoop.hive.ql.metadata.Table table) {
+ return getPartitionTransformSpec(table).stream().map(TransformSpec::getTransformType)
+ .allMatch(type -> type == TransformSpec.TransformType.IDENTITY);
+ }
+
+ @Override
+ public boolean isEligibilityForCompaction(org.apache.hadoop.hive.ql.metadata.Table table,
+ Map<String, String> partitionSpec) throws HiveException {
+ if (partitionSpec != null) {
+ Table icebergTable = IcebergTableUtil.getTable(conf, table.getTTable());
+ if (hasUndergonePartitionEvolution(icebergTable)) {
+ throw new HiveException("Compaction on partition level is not supported on the table " +
+ table.getDbName() + "." + table.getTableName() + " which has undergone partition evolution");
+ }
+ if (!isIdentityPartitionTable(table)) {
Review Comment:
only primitiveTypes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1530212442
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java:
##########
@@ -932,7 +934,9 @@ public void setDbName(String databaseName) {
}
public List<FieldSchema> getPartitionKeys() {
- return tTable.getPartitionKeys();
+ return getStorageHandler() != null && getStorageHandler().alwaysUnpartitioned() &&
+ SessionState.getSessionConf().get(Context.operationNameConf, StringUtils.EMPTY).equals(AlterTableType.COMPACT.getName()) ?
Review Comment:
why do you need to check operationType?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1532347495
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/AbstractBaseAlterTableAnalyzer.java:
##########
@@ -95,11 +98,16 @@ protected void addInputsOutputsAlterTable(TableName tableName, Map<String, Strin
// ReadEntity as no lock.
re.noLockNeeded();
inputs.add(re);
+ Partition partition;
if (AlterTableUtils.isFullPartitionSpec(table, partitionSpec)) {
// Fully specified partition spec
- Partition part = PartitionUtils.getPartition(db, table, partitionSpec, true);
- outputs.add(new WriteEntity(part, writeType));
+ if (table.getStorageHandler() != null && table.getStorageHandler().alwaysUnpartitioned()) {
Review Comment:
why not to make `PartitionUtils.getPartition(db, table, partitionSpec, true)` generic. it should return partitions based on table type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1532338093
##########
ql/src/java/org/apache/hadoop/hive/ql/Context.java:
##########
@@ -256,6 +257,7 @@ public String toString() {
public enum RewritePolicy {
DEFAULT,
+ SINGLE_PARTITION,
Review Comment:
why can't we use default?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1532931650
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +53,68 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+ RewritePolicy rewritePolicy;
+
+ if (partSpec == null) {
+ compactionQuery = String.format("insert overwrite table %s select * from %<s",
+ compactTableName);
+ rewritePolicy = RewritePolicy.ALL_PARTITIONS;
+ } else {
+ Table table = IcebergTableUtil.getTable(context.getConf(), context.getTable());
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ context.getConf().set(Context.compactPartition, partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select * from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "));
+ rewritePolicy = RewritePolicy.SINGLE_PARTITION;
+ }
SessionState sessionState = setupQueryCompactionSession(context.getConf(),
context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, rewritePolicy.name());
try {
DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
+ LOG.error("Error doing query based {} compaction", rewritePolicy.name(), e);
throw new RuntimeException(e);
} finally {
sessionState.setCompaction(false);
}
return true;
}
+
+ private String partDataToSQL(PartitionData partitionData, String partSpec, String delimiter) {
+ StringBuilder sb = new StringBuilder();
+ List<String> values = Arrays
+ .stream(partSpec.split("/"))
+ .map(x -> x.split("=")[1])
+ .collect(Collectors.toList());
+
+ for (int i = 0; i < partitionData.size(); ++i) {
Review Comment:
Replaced the stream operation with a call to `Warehouse#makeSpecFromName(partName)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1532325915
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +53,68 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+ RewritePolicy rewritePolicy;
+
+ if (partSpec == null) {
+ compactionQuery = String.format("insert overwrite table %s select * from %<s",
+ compactTableName);
+ rewritePolicy = RewritePolicy.ALL_PARTITIONS;
+ } else {
+ Table table = IcebergTableUtil.getTable(context.getConf(), context.getTable());
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ context.getConf().set(Context.compactPartition, partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select * from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "));
+ rewritePolicy = RewritePolicy.SINGLE_PARTITION;
+ }
SessionState sessionState = setupQueryCompactionSession(context.getConf(),
context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, rewritePolicy.name());
try {
DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
+ LOG.error("Error doing query based {} compaction", rewritePolicy.name(), e);
throw new RuntimeException(e);
} finally {
sessionState.setCompaction(false);
}
return true;
}
+
+ private String partDataToSQL(PartitionData partitionData, String partSpec, String delimiter) {
+ StringBuilder sb = new StringBuilder();
+ List<String> values = Arrays
Review Comment:
should we use `Warehouse#makeSpecFromName(partName)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1575838382
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +55,71 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ Table table = IcebergTableUtil.getTable(conf, context.getTable());
+ List<String> partFields = table.spec().fields().stream().map(PartitionField::name).collect(Collectors.toList());
+ String queryFields = table.schema().columns().stream().map(Types.NestedField::name)
+ .filter(f -> !partFields.contains(f)).collect(Collectors.joining(","));
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select %4$s from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "), queryFields);
+ }
+
+ SessionState sessionState = setupQueryCompactionSession(conf, context.getCompactionInfo(), tblProperties);
- SessionState sessionState = setupQueryCompactionSession(context.getConf(),
- context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
try {
- DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
+ DriverUtils.runOnDriver(conf, sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
+ return true;
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
- throw new RuntimeException(e);
+ LOG.error("Failed compacting table {}", compactTableName, e);
+ throw e;
} finally {
sessionState.setCompaction(false);
}
+ }
+
+ private String partDataToSQL(PartitionData partitionData, String partSpec, String delimiter)
+ throws HiveException {
+
+ try {
+ Map<String, String> partSpecMap = Warehouse.makeSpecFromName(partSpec);
+ StringBuilder sb = new StringBuilder();
+
+ for (int i = 0; i < partitionData.size(); ++i) {
+ if (i > 0) {
+ sb.append(delimiter);
+ }
+
+ String quoteOpt = "";
+ if (partitionData.getType(i).typeId() == Type.TypeID.STRING ||
+ partitionData.getType(i).typeId() == Type.TypeID.DATE ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIME ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIMESTAMP ||
+ partitionData.getType(i).typeId() == Type.TypeID.BINARY) {
+ quoteOpt = "'";
+ }
- return true;
+ String fieldName = partitionData.getSchema().getFields().get(i).name();
+
+ sb.append(fieldName)
+ .append("=")
+ .append(quoteOpt)
+ .append(partSpecMap.get(fieldName))
+ .append(quoteOpt);
Review Comment:
how about
````
if (partitionData.getType(i).isPrimitive()) {
PrimitiveObjectInspector primitiveObjInspector = IcebergObjectInspector.primitive(partitionData.getType(i))
TypeInfoUtils.convertStringToLiteralForSQL(value, primitiveObjInspector.getPrimitiveCategory())
} else {
throw new ...
}
````
table.getStorageHandler().getColumnInfo looks expensive and unnecessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1575794099
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -1915,6 +1917,46 @@ private boolean hasUndergonePartitionEvolution(Table table) {
.anyMatch(id -> id < table.spec().specId());
}
+ private boolean isIdentityPartitionTable(org.apache.hadoop.hive.ql.metadata.Table table) {
+ return getPartitionTransformSpec(table).stream().map(TransformSpec::getTransformType)
+ .allMatch(type -> type == TransformSpec.TransformType.IDENTITY);
+ }
+
+ @Override
+ public org.apache.commons.lang3.tuple.Pair<Boolean, ErrorMsg> isEligibleForCompaction(
+ org.apache.hadoop.hive.ql.metadata.Table table, Map<String, String> partitionSpec) {
+ if (partitionSpec != null) {
+ Table icebergTable = IcebergTableUtil.getTable(conf, table.getTTable());
+ if (hasUndergonePartitionEvolution(icebergTable)) {
+ return org.apache.commons.lang3.tuple.Pair.of(false, ErrorMsg.COMPACTION_PARTITION_EVOLUTION);
+ }
+ if (!isIdentityPartitionTable(table)) {
+ return org.apache.commons.lang3.tuple.Pair.of(false, ErrorMsg.COMPACTION_NON_IDENTITY_PARTITION_SPEC);
+ }
+ }
+ return org.apache.commons.lang3.tuple.Pair.of(true, null);
+ }
+
+ @Override
+ public List<Partition> getPartitions(org.apache.hadoop.hive.ql.metadata.Table table,
+ Map<String, String> partitionSpec, short limit) throws SemanticException {
+ return table.getStorageHandler().getPartitionNames(table, partitionSpec).stream()
+ .limit(limit > 0 ? limit : Long.MAX_VALUE)
+ .map(partName -> new DummyPartition(table, partName, partitionSpec)).collect(Collectors.toList());
+ }
+
+ @Override
+ public Partition getPartition(org.apache.hadoop.hive.ql.metadata.Table table,
+ Map<String, String> partitionSpec) throws SemanticException {
+ validatePartSpec(table, partitionSpec);
+ try {
+ String partName = Warehouse.makePartName(partitionSpec, false);
+ return new DummyPartition(table, partName, partitionSpec);
+ } catch (MetaException e) {
Review Comment:
can we use makeDynamicPartName to avoid exception? should be ok since we validate before
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1575838382
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +55,71 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ Table table = IcebergTableUtil.getTable(conf, context.getTable());
+ List<String> partFields = table.spec().fields().stream().map(PartitionField::name).collect(Collectors.toList());
+ String queryFields = table.schema().columns().stream().map(Types.NestedField::name)
+ .filter(f -> !partFields.contains(f)).collect(Collectors.joining(","));
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select %4$s from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "), queryFields);
+ }
+
+ SessionState sessionState = setupQueryCompactionSession(conf, context.getCompactionInfo(), tblProperties);
- SessionState sessionState = setupQueryCompactionSession(context.getConf(),
- context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
try {
- DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
+ DriverUtils.runOnDriver(conf, sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
+ return true;
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
- throw new RuntimeException(e);
+ LOG.error("Failed compacting table {}", compactTableName, e);
+ throw e;
} finally {
sessionState.setCompaction(false);
}
+ }
+
+ private String partDataToSQL(PartitionData partitionData, String partSpec, String delimiter)
+ throws HiveException {
+
+ try {
+ Map<String, String> partSpecMap = Warehouse.makeSpecFromName(partSpec);
+ StringBuilder sb = new StringBuilder();
+
+ for (int i = 0; i < partitionData.size(); ++i) {
+ if (i > 0) {
+ sb.append(delimiter);
+ }
+
+ String quoteOpt = "";
+ if (partitionData.getType(i).typeId() == Type.TypeID.STRING ||
+ partitionData.getType(i).typeId() == Type.TypeID.DATE ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIME ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIMESTAMP ||
+ partitionData.getType(i).typeId() == Type.TypeID.BINARY) {
+ quoteOpt = "'";
+ }
- return true;
+ String fieldName = partitionData.getSchema().getFields().get(i).name();
+
+ sb.append(fieldName)
+ .append("=")
+ .append(quoteOpt)
+ .append(partSpecMap.get(fieldName))
+ .append(quoteOpt);
Review Comment:
how about
````
Map<String, String> partSpecMap = new LinkedHashMap<>();
Warehouse.makeSpecFromName(partSpecMap, new Path(partSpec), null);
List<FieldSchema> partitionKeys = table.getStorageHandler().getPartitionKeys(table);
List<String> partValues = partitionKeys.stream().map(
fs -> String.join("=", HiveUtils.unparseIdentifier(fs.getName()),
TypeInfoUtils.convertStringToLiteralForSQL(partSpecMap.get(fs.getName()),
((PrimitiveTypeInfo) TypeInfoUtils.getTypeInfoFromTypeString(fs.getType())).getPrimitiveCategory())
)
).collect(Collectors.toList());
String queryFields = table.getCols().stream()
.map(FieldSchema::getName)
.filter(col -> !partSpecMap.containsKey(col))
.collect(Collectors.joining(","));
compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select %4$s from %1$s where %3$s",
compactTableName,
StringUtils.join(partValues, ","),
StringUtils.join(partValues, " and "),
queryFields);
````
table.getStorageHandler().getColumnInfo looks expensive and unnecessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1576709834
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +55,51 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ Hive db = Hive.get(conf);
+ org.apache.hadoop.hive.ql.metadata.Table table = db.getTable(context.getTable().getDbName(),
+ context.getTable().getTableName());
+ Map<String, String> partSpecMap = new LinkedHashMap<>();
+ Warehouse.makeSpecFromName(partSpecMap, new Path(partSpec), null);
+
+ List<FieldSchema> partitionKeys = table.getStorageHandler().getPartitionKeys(table);
+ List<String> partValues = partitionKeys.stream().map(
+ fs -> String.join("=", HiveUtils.unparseIdentifier(fs.getName()),
+ TypeInfoUtils.convertStringToLiteralForSQL(partSpecMap.get(fs.getName()),
+ ((PrimitiveTypeInfo) TypeInfoUtils.getTypeInfoFromTypeString(fs.getType())).getPrimitiveCategory())
+ )
+ ).collect(Collectors.toList());
+
+ String queryFields = table.getCols().stream()
+ .map(FieldSchema::getName)
+ .filter(col -> !partSpecMap.containsKey(col))
+ .collect(Collectors.joining(","));
+
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select %4$s from %1$s where %3$s",
+ compactTableName,
+ StringUtils.join(partValues, ","),
+ StringUtils.join(partValues, " and "),
+ queryFields);
+ }
+
+ SessionState sessionState = setupQueryCompactionSession(conf, context.getCompactionInfo(), tblProperties);
- SessionState sessionState = setupQueryCompactionSession(context.getConf(),
- context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
try {
- DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
+ DriverUtils.runOnDriver(conf, sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
+ return true;
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
- throw new RuntimeException(e);
+ LOG.error("Failed compacting table {}", compactTableName, e);
Review Comment:
"Failed to compact table {}, partition {} "
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ merged PR #5123:
URL: https://github.com/apache/hive/pull/5123
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-1998803291
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [9 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1525198364
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##########
@@ -544,14 +549,20 @@ private void commitWrite(Table table, String branchName, Long snapshotId, long s
* @param rewritePolicy The rewrite policy to use for the insert overwrite commit
*/
private void commitOverwrite(Table table, String branchName, long startTime, FilesForCommit results,
- RewritePolicy rewritePolicy) {
+ RewritePolicy rewritePolicy, String partitionSpec) {
Preconditions.checkArgument(results.deleteFiles().isEmpty(), "Can not handle deletes with overwrite");
if (!results.dataFiles().isEmpty()) {
Transaction transaction = table.newTransaction();
if (rewritePolicy == RewritePolicy.ALL_PARTITIONS) {
DeleteFiles delete = transaction.newDelete();
delete.deleteFromRowFilter(Expressions.alwaysTrue());
delete.commit();
+ } else if (rewritePolicy == RewritePolicy.SINGLE_PARTITION) {
+ try {
+ HiveIcebergMetaHook.preTruncateTable(table, Arrays.asList(partitionSpec), transaction.newDelete());
Review Comment:
should we rename to `truncatePartitionBySpec`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1525224491
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/PartitionIterable.java:
##########
@@ -99,7 +99,16 @@ private void getNextBatch() {
batchCounter++;
}
try {
- if (isAuthRequired) {
+ if (table.getStorageHandler() != null && table.getStorageHandler().alwaysUnpartitioned()) {
+ List<Partition> partitions = new ArrayList<>();
Review Comment:
why do we need new List var?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1526463319
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##########
@@ -544,14 +549,20 @@ private void commitWrite(Table table, String branchName, Long snapshotId, long s
* @param rewritePolicy The rewrite policy to use for the insert overwrite commit
*/
private void commitOverwrite(Table table, String branchName, long startTime, FilesForCommit results,
- RewritePolicy rewritePolicy) {
+ RewritePolicy rewritePolicy, String partitionSpec) {
Preconditions.checkArgument(results.deleteFiles().isEmpty(), "Can not handle deletes with overwrite");
if (!results.dataFiles().isEmpty()) {
Transaction transaction = table.newTransaction();
if (rewritePolicy == RewritePolicy.ALL_PARTITIONS) {
DeleteFiles delete = transaction.newDelete();
delete.deleteFromRowFilter(Expressions.alwaysTrue());
delete.commit();
+ } else if (rewritePolicy == RewritePolicy.SINGLE_PARTITION) {
+ try {
+ HiveIcebergMetaHook.preTruncateTable(table, Arrays.asList(partitionSpec), transaction.newDelete());
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1526550105
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +52,68 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+ RewritePolicy rewritePolicy;
+
+ if (partSpec == null) {
+ compactionQuery = String.format("insert overwrite table %s select * from %<s",
+ compactTableName);
+ rewritePolicy = RewritePolicy.ALL_PARTITIONS;
+ } else {
+ Table table = IcebergTableUtil.getTable(context.getConf(), context.getTable());
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ HiveConf.setVar(context.getConf(), ConfVars.HIVE_ICEBERG_COMPACTION_PARTITION, partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select * from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "));
+ rewritePolicy = RewritePolicy.SINGLE_PARTITION;
+ }
SessionState sessionState = setupQueryCompactionSession(context.getConf(),
context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, rewritePolicy.name());
try {
DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
+ LOG.error("Error doing query based {} compaction", rewritePolicy.name(), e);
throw new RuntimeException(e);
} finally {
sessionState.setCompaction(false);
}
return true;
}
+
+ public String partDataToSQL(PartitionData partitionData, String partSpec, String delimiter) {
Review Comment:
Changed to private.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-1986945071
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [16 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1525198364
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##########
@@ -544,14 +549,20 @@ private void commitWrite(Table table, String branchName, Long snapshotId, long s
* @param rewritePolicy The rewrite policy to use for the insert overwrite commit
*/
private void commitOverwrite(Table table, String branchName, long startTime, FilesForCommit results,
- RewritePolicy rewritePolicy) {
+ RewritePolicy rewritePolicy, String partitionSpec) {
Preconditions.checkArgument(results.deleteFiles().isEmpty(), "Can not handle deletes with overwrite");
if (!results.dataFiles().isEmpty()) {
Transaction transaction = table.newTransaction();
if (rewritePolicy == RewritePolicy.ALL_PARTITIONS) {
DeleteFiles delete = transaction.newDelete();
delete.deleteFromRowFilter(Expressions.alwaysTrue());
delete.commit();
+ } else if (rewritePolicy == RewritePolicy.SINGLE_PARTITION) {
+ try {
+ HiveIcebergMetaHook.preTruncateTable(table, Arrays.asList(partitionSpec), transaction.newDelete());
Review Comment:
should we rename to `truncateTable`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1525212033
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +52,68 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+ RewritePolicy rewritePolicy;
+
+ if (partSpec == null) {
+ compactionQuery = String.format("insert overwrite table %s select * from %<s",
+ compactTableName);
+ rewritePolicy = RewritePolicy.ALL_PARTITIONS;
+ } else {
+ Table table = IcebergTableUtil.getTable(context.getConf(), context.getTable());
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ HiveConf.setVar(context.getConf(), ConfVars.HIVE_ICEBERG_COMPACTION_PARTITION, partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select * from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "));
+ rewritePolicy = RewritePolicy.SINGLE_PARTITION;
+ }
SessionState sessionState = setupQueryCompactionSession(context.getConf(),
context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, rewritePolicy.name());
try {
DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
+ LOG.error("Error doing query based {} compaction", rewritePolicy.name(), e);
throw new RuntimeException(e);
} finally {
sessionState.setCompaction(false);
}
return true;
}
+
+ public String partDataToSQL(PartitionData partitionData, String partSpec, String delimiter) {
Review Comment:
should it be public?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-1992559857
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [17 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1526459556
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##########
@@ -544,14 +549,20 @@ private void commitWrite(Table table, String branchName, Long snapshotId, long s
* @param rewritePolicy The rewrite policy to use for the insert overwrite commit
*/
private void commitOverwrite(Table table, String branchName, long startTime, FilesForCommit results,
- RewritePolicy rewritePolicy) {
+ RewritePolicy rewritePolicy, String partitionSpec) {
Preconditions.checkArgument(results.deleteFiles().isEmpty(), "Can not handle deletes with overwrite");
if (!results.dataFiles().isEmpty()) {
Transaction transaction = table.newTransaction();
if (rewritePolicy == RewritePolicy.ALL_PARTITIONS) {
DeleteFiles delete = transaction.newDelete();
delete.deleteFromRowFilter(Expressions.alwaysTrue());
delete.commit();
+ } else if (rewritePolicy == RewritePolicy.SINGLE_PARTITION) {
Review Comment:
If it were `rewritePolicy != RewritePolicy.ALL_PARTITIONS`, it would also be true for the case when `rewritePolicy == RewritePolicy.DEFAULT`, but the code in the else block should only be executed for compacting a specific partition, not in general case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1532297215
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##########
@@ -462,7 +463,12 @@ private void commitTable(FileIO io, ExecutorService executor, OutputTable output
.map(x -> x.getJobConf().get(ConfVars.REWRITE_POLICY.varname))
.orElse(RewritePolicy.DEFAULT.name()));
- commitOverwrite(table, branchName, startTime, filesForCommit, rewritePolicy);
+ String partitionSpec = outputTable.jobContexts.stream()
Review Comment:
that should be a List here instead of mapping in commitOverwrite
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1531084836
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/AbstractBaseAlterTableAnalyzer.java:
##########
@@ -98,6 +100,7 @@ protected void addInputsOutputsAlterTable(TableName tableName, Map<String, Strin
if (AlterTableUtils.isFullPartitionSpec(table, partitionSpec)) {
// Fully specified partition spec
+ SessionState.getSessionConf().set(Context.operationNameConf, op.getName());
Review Comment:
Removed checking of operation type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1532360882
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##########
@@ -8810,7 +8810,8 @@ private Operator genConversionSelectOperatorByAddPartition(String dest, QB qb, O
List<ExprNodeDesc> expressions = new ArrayList<>(outColumnCnt);
AtomicBoolean convert = new AtomicBoolean(false);
- if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt) {
+ if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
Review Comment:
i don't think we should just ignore that check for iceberg, can we update the `staticPartitionColumnCnt`?
cc @kasakrisz
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1531084174
##########
ql/src/java/org/apache/hadoop/hive/ql/Compiler.java:
##########
@@ -100,6 +100,7 @@ public QueryPlan compile(String rawCommand, boolean deferClose) throws CommandPr
boolean parsed = false;
QueryPlan plan = null;
try {
+ SessionState.getSessionConf().set(Context.operationNameConf, org.apache.commons.lang3.StringUtils.EMPTY);
Review Comment:
Done, removed checking of operation type as part of compact operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1532337396
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +53,68 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+ RewritePolicy rewritePolicy;
+
+ if (partSpec == null) {
+ compactionQuery = String.format("insert overwrite table %s select * from %<s",
+ compactTableName);
+ rewritePolicy = RewritePolicy.ALL_PARTITIONS;
+ } else {
+ Table table = IcebergTableUtil.getTable(context.getConf(), context.getTable());
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ context.getConf().set(Context.compactPartition, partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select * from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "));
+ rewritePolicy = RewritePolicy.SINGLE_PARTITION;
+ }
SessionState sessionState = setupQueryCompactionSession(context.getConf(),
context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, rewritePolicy.name());
try {
DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
+ LOG.error("Error doing query based {} compaction", rewritePolicy.name(), e);
throw new RuntimeException(e);
} finally {
sessionState.setCompaction(false);
}
return true;
}
+
+ private String partDataToSQL(PartitionData partitionData, String partSpec, String delimiter) {
+ StringBuilder sb = new StringBuilder();
+ List<String> values = Arrays
+ .stream(partSpec.split("/"))
+ .map(x -> x.split("=")[1])
+ .collect(Collectors.toList());
+
+ for (int i = 0; i < partitionData.size(); ++i) {
Review Comment:
why not use `Stream::collect(Collectors.joining())`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1532854461
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java:
##########
@@ -640,7 +640,13 @@ public void preTruncateTable(org.apache.hadoop.hive.metastore.api.Table table, E
throws MetaException {
this.catalogProperties = getCatalogProperties(table);
this.icebergTable = Catalogs.loadTable(conf, catalogProperties);
- Map<String, PartitionField> partitionFieldMap = icebergTable.spec().fields().stream()
+ truncatePartitionBySpec(icebergTable, partNames, icebergTable.newDelete());
+ context.putToProperties("truncateSkipDataDeletion", "true");
Review Comment:
Changed to the constant. I will check if that is used in case of Iceberg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1575794099
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -1915,6 +1917,46 @@ private boolean hasUndergonePartitionEvolution(Table table) {
.anyMatch(id -> id < table.spec().specId());
}
+ private boolean isIdentityPartitionTable(org.apache.hadoop.hive.ql.metadata.Table table) {
+ return getPartitionTransformSpec(table).stream().map(TransformSpec::getTransformType)
+ .allMatch(type -> type == TransformSpec.TransformType.IDENTITY);
+ }
+
+ @Override
+ public org.apache.commons.lang3.tuple.Pair<Boolean, ErrorMsg> isEligibleForCompaction(
+ org.apache.hadoop.hive.ql.metadata.Table table, Map<String, String> partitionSpec) {
+ if (partitionSpec != null) {
+ Table icebergTable = IcebergTableUtil.getTable(conf, table.getTTable());
+ if (hasUndergonePartitionEvolution(icebergTable)) {
+ return org.apache.commons.lang3.tuple.Pair.of(false, ErrorMsg.COMPACTION_PARTITION_EVOLUTION);
+ }
+ if (!isIdentityPartitionTable(table)) {
+ return org.apache.commons.lang3.tuple.Pair.of(false, ErrorMsg.COMPACTION_NON_IDENTITY_PARTITION_SPEC);
+ }
+ }
+ return org.apache.commons.lang3.tuple.Pair.of(true, null);
+ }
+
+ @Override
+ public List<Partition> getPartitions(org.apache.hadoop.hive.ql.metadata.Table table,
+ Map<String, String> partitionSpec, short limit) throws SemanticException {
+ return table.getStorageHandler().getPartitionNames(table, partitionSpec).stream()
+ .limit(limit > 0 ? limit : Long.MAX_VALUE)
+ .map(partName -> new DummyPartition(table, partName, partitionSpec)).collect(Collectors.toList());
+ }
+
+ @Override
+ public Partition getPartition(org.apache.hadoop.hive.ql.metadata.Table table,
+ Map<String, String> partitionSpec) throws SemanticException {
+ validatePartSpec(table, partitionSpec);
+ try {
+ String partName = Warehouse.makePartName(partitionSpec, false);
+ return new DummyPartition(table, partName, partitionSpec);
+ } catch (MetaException e) {
Review Comment:
you should throw `MetaException` and catch it in the orig place - Analyzer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1575838382
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +55,71 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ Table table = IcebergTableUtil.getTable(conf, context.getTable());
+ List<String> partFields = table.spec().fields().stream().map(PartitionField::name).collect(Collectors.toList());
+ String queryFields = table.schema().columns().stream().map(Types.NestedField::name)
+ .filter(f -> !partFields.contains(f)).collect(Collectors.joining(","));
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select %4$s from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "), queryFields);
+ }
+
+ SessionState sessionState = setupQueryCompactionSession(conf, context.getCompactionInfo(), tblProperties);
- SessionState sessionState = setupQueryCompactionSession(context.getConf(),
- context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
try {
- DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
+ DriverUtils.runOnDriver(conf, sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
+ return true;
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
- throw new RuntimeException(e);
+ LOG.error("Failed compacting table {}", compactTableName, e);
+ throw e;
} finally {
sessionState.setCompaction(false);
}
+ }
+
+ private String partDataToSQL(PartitionData partitionData, String partSpec, String delimiter)
+ throws HiveException {
+
+ try {
+ Map<String, String> partSpecMap = Warehouse.makeSpecFromName(partSpec);
+ StringBuilder sb = new StringBuilder();
+
+ for (int i = 0; i < partitionData.size(); ++i) {
+ if (i > 0) {
+ sb.append(delimiter);
+ }
+
+ String quoteOpt = "";
+ if (partitionData.getType(i).typeId() == Type.TypeID.STRING ||
+ partitionData.getType(i).typeId() == Type.TypeID.DATE ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIME ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIMESTAMP ||
+ partitionData.getType(i).typeId() == Type.TypeID.BINARY) {
+ quoteOpt = "'";
+ }
- return true;
+ String fieldName = partitionData.getSchema().getFields().get(i).name();
+
+ sb.append(fieldName)
+ .append("=")
+ .append(quoteOpt)
+ .append(partSpecMap.get(fieldName))
+ .append(quoteOpt);
Review Comment:
how about
````
Map<String, String> partSpecMap = new LinkedHashMap<>();
Warehouse.makeSpecFromName(partSpecMap, new Path(partSpec), null);
List<FieldSchema> partitionKeys = hiveTable.getStorageHandler().getPartitionKeys(hiveTable);
List<String> partValues = partitionKeys.stream().map(
fs -> String.join("=", HiveUtils.unparseIdentifier(fs.getName()),
TypeInfoUtils.convertStringToLiteralForSQL(partSpecMap.get(fs.getName()),
((PrimitiveTypeInfo) TypeInfoUtils.getTypeInfoFromTypeString(fs.getType())).getPrimitiveCategory())
)
).collect(Collectors.toList());
String queryFields = hiveTable.getCols().stream().map(FieldSchema::getName)
.filter(col -> !partSpecMap.containsKey(col))
.collect(Collectors.joining(","));
compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select %4$s from %1$s where %3$s",
compactTableName,
StringUtils.join(partValues, ","),
StringUtils.join(partValues, " and "),
queryFields);
}
````
table.getStorageHandler().getColumnInfo looks expensive and unnecessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1576703781
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +55,51 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ Hive db = Hive.get(conf);
Review Comment:
can we inline where it's used
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1573873316
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +55,71 @@ public boolean run(CompactorContext context) throws IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select * from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from %<s", compactTableName);
+ } else {
+ Table table = IcebergTableUtil.getTable(conf, context.getTable());
+ List<String> partFields = table.spec().fields().stream().map(PartitionField::name).collect(Collectors.toList());
+ String queryFields = table.schema().columns().stream().map(Types.NestedField::name)
+ .filter(f -> !partFields.contains(f)).collect(Collectors.joining(","));
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s partition(%2$s) select %4$s from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "), queryFields);
+ }
+
+ SessionState sessionState = setupQueryCompactionSession(conf, context.getCompactionInfo(), tblProperties);
- SessionState sessionState = setupQueryCompactionSession(context.getConf(),
- context.getCompactionInfo(), tblProperties);
- HiveConf.setVar(context.getConf(), ConfVars.REWRITE_POLICY, RewritePolicy.ALL_PARTITIONS.name());
try {
- DriverUtils.runOnDriver(context.getConf(), sessionState, compactionQuery);
+ DriverUtils.runOnDriver(conf, sessionState, compactionQuery);
LOG.info("Completed compaction for table {}", compactTableName);
+ return true;
} catch (HiveException e) {
- LOG.error("Error doing query based {} compaction", RewritePolicy.ALL_PARTITIONS.name(), e);
- throw new RuntimeException(e);
+ LOG.error("Failed compacting table {}", compactTableName, e);
+ throw e;
} finally {
sessionState.setCompaction(false);
}
+ }
+
+ private String partDataToSQL(PartitionData partitionData, String partSpec, String delimiter)
+ throws HiveException {
+
+ try {
+ Map<String, String> partSpecMap = Warehouse.makeSpecFromName(partSpec);
+ StringBuilder sb = new StringBuilder();
+
+ for (int i = 0; i < partitionData.size(); ++i) {
+ if (i > 0) {
+ sb.append(delimiter);
+ }
+
+ String quoteOpt = "";
+ if (partitionData.getType(i).typeId() == Type.TypeID.STRING ||
+ partitionData.getType(i).typeId() == Type.TypeID.DATE ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIME ||
+ partitionData.getType(i).typeId() == Type.TypeID.TIMESTAMP ||
+ partitionData.getType(i).typeId() == Type.TypeID.BINARY) {
+ quoteOpt = "'";
+ }
- return true;
+ String fieldName = partitionData.getSchema().getFields().get(i).name();
+
+ sb.append(fieldName)
+ .append("=")
+ .append(quoteOpt)
+ .append(partSpecMap.get(fieldName))
+ .append(quoteOpt);
Review Comment:
nit: remove extra space
also, you could simplify the code with
````
HiveUtils.unparseIdentifier(fieldName)
.append("=")
.append(TypeInfoUtils.convertStringToLiteralForSQL(
partSpecMap.get(fieldName),
((PrimitiveTypeInfo) TypeInfoUtils.getTypeInfoFromTypeString(partitionData.getType(i).typeId())).getPrimitiveCategory()
````
check `DDLPlanUtils.getPartitionActualName`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1573884289
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -1915,6 +1915,28 @@ private boolean hasUndergonePartitionEvolution(Table table) {
.anyMatch(id -> id < table.spec().specId());
}
+ private boolean isIdentityPartitionTable(org.apache.hadoop.hive.ql.metadata.Table table) {
+ return getPartitionTransformSpec(table).stream().map(TransformSpec::getTransformType)
+ .allMatch(type -> type == TransformSpec.TransformType.IDENTITY);
+ }
+
+ @Override
+ public boolean isEligibilityForCompaction(org.apache.hadoop.hive.ql.metadata.Table table,
+ Map<String, String> partitionSpec) throws HiveException {
+ if (partitionSpec != null) {
+ Table icebergTable = IcebergTableUtil.getTable(conf, table.getTTable());
+ if (hasUndergonePartitionEvolution(icebergTable)) {
+ throw new HiveException("Compaction on partition level is not supported on the table " +
+ table.getDbName() + "." + table.getTableName() + " which has undergone partition evolution");
+ }
+ if (!isIdentityPartitionTable(table)) {
Review Comment:
only primitiveTypes: `partitionData.getType(i).isPrimitiveType()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1573881457
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java:
##########
@@ -782,7 +782,7 @@ private void setCommonHmsTablePropertiesForIceberg(org.apache.hadoop.hive.metast
* @param hmsTable Table for which we are calculating the properties
* @return The properties we can provide for Iceberg functions, like {@link Catalogs}
*/
- private static Properties getCatalogProperties(org.apache.hadoop.hive.metastore.api.Table hmsTable) {
+ public static Properties getCatalogProperties(org.apache.hadoop.hive.metastore.api.Table hmsTable) {
Review Comment:
should it be public?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1569023720
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/PartitionUtils.java:
##########
@@ -108,11 +122,17 @@ public static Partition getPartition(Hive db, Table table, Map<String, String> p
public static List<Partition> getPartitions(Hive db, Table table, Map<String, String> partitionSpec,
boolean throwException) throws SemanticException {
List<Partition> partitions;
- try {
- partitions = partitionSpec == null ? db.getPartitions(table) : db.getPartitions(table, partitionSpec);
- } catch (Exception e) {
- throw new SemanticException(toMessage(ErrorMsg.INVALID_PARTITION, partitionSpec), e);
- }
+ if (table.getStorageHandler() != null && table.getStorageHandler().alwaysUnpartitioned()) {
+ partitions = new ArrayList<>();
+ partitions.add(getPartition(db, table, partitionSpec, throwException));
Review Comment:
that is a diff API, it's supposed to return all partitions matching spec, not just 1
are we using this in case of compaction?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1569035919
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/compact/AlterTableCompactOperation.java:
##########
@@ -133,7 +140,7 @@ private List<Partition> getPartitions(Table table, AlterTableCompactDesc desc, D
}
} else {
Map<String, String> partitionSpec = desc.getPartitionSpec();
- partitions = context.getDb().getPartitions(table, partitionSpec);
+ partitions.addAll(PartitionUtils.getPartitions(context.getDb(), table, partitionSpec, false));
Review Comment:
why don't we use `partitionUtils.getPartition()` - it should return just 1 partition
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1569351921
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:
##########
@@ -1187,6 +1187,11 @@ private List<Partition> getPartitionsByQuery(String catName, String dbName, Stri
MetastoreDirectSqlUtils.timingTrace(doTrace, queryText, start, queryTime);
}
String partIds = getIdListForIn(partitions.keySet());
+
+ if (partIds.isEmpty()) {
Review Comment:
Yes, because the issue with Metastore Direct SQL was encountered after I had added Iceberg partition list functionality in PartitionUtils and also I think PartitionUtils is not used by this metastore code because it is found in hive-ql and it's not a dependency for the metastore modules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2050407048
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [17 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2062147149
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123) **Quality Gate passed**
Issues
 [10 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5123&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5123&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5123)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1570834961
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:
##########
@@ -1187,6 +1187,11 @@ private List<Partition> getPartitionsByQuery(String catName, String dbName, Stri
MetastoreDirectSqlUtils.timingTrace(doTrace, queryText, start, queryTime);
}
String partIds = getIdListForIn(partitions.keySet());
+
+ if (partIds.isEmpty()) {
Review Comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1570841410
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##########
@@ -8815,6 +8815,23 @@ These props are now enabled elsewhere (see commit diffs). It would be better in
conf.set(AcidUtils.CONF_ACID_KEY, "true");
SessionState.get().getConf().set(AcidUtils.CONF_ACID_KEY, "true");
}
+
+ private void validateColumnCounts(String dest, Table table, Map<String, String> partitionSpec,
+ int inColumnCnt, int outColumnCnt) throws SemanticException {
+
+ long staticPartitionColumnCnt = partitionSpec != null ?
+ partitionSpec.values().stream().filter(Objects::nonNull).count() : 0;
+
+ if (inColumnCnt + staticPartitionColumnCnt != outColumnCnt &&
Review Comment:
could we extract `table.getStorageHandler() != null && table.getStorageHandler().alwaysUnpartitioned()` into local var `alwaysUnpartitioned`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1571430376
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/compact/AlterTableCompactOperation.java:
##########
@@ -133,7 +140,7 @@ private List<Partition> getPartitions(Table table, AlterTableCompactDesc desc, D
}
} else {
Map<String, String> partitionSpec = desc.getPartitionSpec();
- partitions = context.getDb().getPartitions(table, partitionSpec);
+ partitions.addAll(PartitionUtils.getPartitions(context.getDb(), table, partitionSpec, false));
Review Comment:
Yes, this is used for scenarios like you said.
It is already supported for Iceberg with the change I did yesterday to PartitionUtils.getPartitions() method.
I verified it with a q-test and it throws the exception `TOO_MANY_COMPACTION_PARTITIONS` as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org