You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "kuczoram (via GitHub)" <gi...@apache.org> on 2023/12/15 13:03:22 UTC
[PR] HIVE-27956: Query based compactor implementation separation [hive]
kuczoram opened a new pull request, #4952:
URL: https://github.com/apache/hive/pull/4952
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### Is the change a dependency upgrade?
<!--
If yes, please attach a file with output from mvn dependency:tree to validate a complete upgrade of dependency.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27956: Query based compactor implementation separation [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4952:
URL: https://github.com/apache/hive/pull/4952#issuecomment-1857963853
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4952) **Quality Gate passed**
The SonarCloud Quality Gate passed, but some issues were introduced.
[17 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4952&resolved=false&inNewCodePeriod=true)
[0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4952&resolved=false&inNewCodePeriod=true)
No data about Coverage
No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4952)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27956: Query based compactor implementation separation [hive]
Posted by "kuczoram (via GitHub)" <gi...@apache.org>.
kuczoram commented on PR #4952:
URL: https://github.com/apache/hive/pull/4952#issuecomment-2049562100
Since this PR is very far from the master now, it is easier to open a new PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27956: Query based compactor implementation separation [hive]
Posted by "kuczoram (via GitHub)" <gi...@apache.org>.
kuczoram closed pull request #4952: HIVE-27956: Query based compactor implementation separation
URL: https://github.com/apache/hive/pull/4952
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27956: Query based compactor implementation separation [hive]
Posted by "kuczoram (via GitHub)" <gi...@apache.org>.
kuczoram commented on code in PR #4952:
URL: https://github.com/apache/hive/pull/4952#discussion_r1431062052
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactionQueryBuilderForInsertOnly.java:
##########
@@ -0,0 +1,313 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.exec.DDLPlanUtils;
+import org.apache.hadoop.hive.ql.util.DirectionUtils;
+import org.apache.hive.common.util.HiveStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Modifier;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Builds query strings that help with query-based compaction of insert-only tables.
+ */
+class CompactionQueryBuilderForInsertOnly extends CompactionQueryBuilder {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CompactionQueryBuilderForInsertOnly.class.getName());
+
+ private StorageDescriptor storageDescriptor; // for Create in insert-only
+
+ /**
+ * Construct a CompactionQueryBuilderForInsertOnly with required params.
+ *
+ * @param compactionType major or minor or rebalance, e.g. CompactionType.MAJOR.
+ * Cannot be null.
+ * @param operation query's Operation e.g. Operation.CREATE.
+ * @throws IllegalArgumentException if compactionType is null or the compaction type is REBALANCE
+ */
+ CompactionQueryBuilderForInsertOnly(CompactionType compactionType, Operation operation, String resultTableName) {
+ super(compactionType, operation, true, resultTableName);
+ if (CompactionType.REBALANCE.equals(compactionType)) {
+ throw new IllegalArgumentException("Rebalance compaction is supported only on full ACID tables!");
Review Comment:
Thanks a lot @aturoczy for the review!!
Sure, I can add some unit tests, it would totally make sense.
I can change this message, no problem. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27956: Query based compactor implementation separation [hive]
Posted by "aturoczy (via GitHub)" <gi...@apache.org>.
aturoczy commented on code in PR #4952:
URL: https://github.com/apache/hive/pull/4952#discussion_r1430389294
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactionQueryBuilderForInsertOnly.java:
##########
@@ -0,0 +1,313 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.exec.DDLPlanUtils;
+import org.apache.hadoop.hive.ql.util.DirectionUtils;
+import org.apache.hive.common.util.HiveStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Modifier;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Builds query strings that help with query-based compaction of insert-only tables.
+ */
+class CompactionQueryBuilderForInsertOnly extends CompactionQueryBuilder {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CompactionQueryBuilderForInsertOnly.class.getName());
+
+ private StorageDescriptor storageDescriptor; // for Create in insert-only
+
+ /**
+ * Construct a CompactionQueryBuilderForInsertOnly with required params.
+ *
+ * @param compactionType major or minor or rebalance, e.g. CompactionType.MAJOR.
+ * Cannot be null.
+ * @param operation query's Operation e.g. Operation.CREATE.
+ * @throws IllegalArgumentException if compactionType is null or the compaction type is REBALANCE
+ */
+ CompactionQueryBuilderForInsertOnly(CompactionType compactionType, Operation operation, String resultTableName) {
+ super(compactionType, operation, true, resultTableName);
+ if (CompactionType.REBALANCE.equals(compactionType)) {
+ throw new IllegalArgumentException("Rebalance compaction is supported only on full ACID tables!");
Review Comment:
I would use the following exception message: Rebalance compaction can only be supported on full ACID tables.
But you can ignore this request. It is just style change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27956: Query based compactor implementation separation [hive]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #4952: HIVE-27956: Query based compactor implementation separation
URL: https://github.com/apache/hive/pull/4952
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27956: Query based compactor implementation separation [hive]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #4952:
URL: https://github.com/apache/hive/pull/4952#discussion_r1431774350
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactionQueryBuilderForInsertOnly.java:
##########
@@ -0,0 +1,313 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.exec.DDLPlanUtils;
+import org.apache.hadoop.hive.ql.util.DirectionUtils;
+import org.apache.hive.common.util.HiveStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Modifier;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Builds query strings that help with query-based compaction of insert-only tables.
+ */
+class CompactionQueryBuilderForInsertOnly extends CompactionQueryBuilder {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CompactionQueryBuilderForInsertOnly.class.getName());
+
+ private StorageDescriptor storageDescriptor; // for Create in insert-only
+
+ /**
+ * Construct a CompactionQueryBuilderForInsertOnly with required params.
+ *
+ * @param compactionType major or minor or rebalance, e.g. CompactionType.MAJOR.
+ * Cannot be null.
+ * @param operation query's Operation e.g. Operation.CREATE.
+ * @throws IllegalArgumentException if compactionType is null or the compaction type is REBALANCE
+ */
+ CompactionQueryBuilderForInsertOnly(CompactionType compactionType, Operation operation, String resultTableName) {
+ super(compactionType, operation, true, resultTableName);
+ if (CompactionType.REBALANCE.equals(compactionType)) {
+ throw new IllegalArgumentException("Rebalance compaction is supported only on full ACID tables!");
+ }
+ }
+
+ /**
+ * Set the StorageDescriptor of the table or partition to compact.
+ * Required for Create operations in insert-only compaction.
+ *
+ * @param storageDescriptor StorageDescriptor of the table or partition to compact, not null
+ */
+ CompactionQueryBuilder setStorageDescriptor(StorageDescriptor storageDescriptor) {
+ this.storageDescriptor = storageDescriptor;
+ return this;
+ }
+
+ protected void buildSelectClauseForInsert(StringBuilder query) {
Review Comment:
Missing ``Override ``
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactionQueryBuilderForInsertOnly.java:
##########
@@ -0,0 +1,313 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.exec.DDLPlanUtils;
+import org.apache.hadoop.hive.ql.util.DirectionUtils;
+import org.apache.hive.common.util.HiveStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Modifier;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Builds query strings that help with query-based compaction of insert-only tables.
+ */
+class CompactionQueryBuilderForInsertOnly extends CompactionQueryBuilder {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CompactionQueryBuilderForInsertOnly.class.getName());
+
+ private StorageDescriptor storageDescriptor; // for Create in insert-only
+
+ /**
+ * Construct a CompactionQueryBuilderForInsertOnly with required params.
+ *
+ * @param compactionType major or minor or rebalance, e.g. CompactionType.MAJOR.
+ * Cannot be null.
+ * @param operation query's Operation e.g. Operation.CREATE.
+ * @throws IllegalArgumentException if compactionType is null or the compaction type is REBALANCE
+ */
+ CompactionQueryBuilderForInsertOnly(CompactionType compactionType, Operation operation, String resultTableName) {
+ super(compactionType, operation, true, resultTableName);
+ if (CompactionType.REBALANCE.equals(compactionType)) {
+ throw new IllegalArgumentException("Rebalance compaction is supported only on full ACID tables!");
+ }
+ }
+
+ /**
+ * Set the StorageDescriptor of the table or partition to compact.
+ * Required for Create operations in insert-only compaction.
+ *
+ * @param storageDescriptor StorageDescriptor of the table or partition to compact, not null
+ */
+ CompactionQueryBuilder setStorageDescriptor(StorageDescriptor storageDescriptor) {
+ this.storageDescriptor = storageDescriptor;
+ return this;
+ }
+
+ protected void buildSelectClauseForInsert(StringBuilder query) {
+ // Need list of columns for major crud, mmmajor partitioned, mmminor
+ List<FieldSchema> cols;
+ if (CompactionType.MAJOR.equals(compactionType) && sourcePartition != null || CompactionType.MINOR.equals(
+ compactionType)) {
+ if (sourceTab == null) {
+ return; // avoid NPEs, don't throw an exception but skip this part of the query
+ }
+ cols = sourceTab.getSd().getCols();
+ } else {
+ cols = null;
+ }
+ switch (compactionType) {
+ case MAJOR: {
+ if (sourcePartition != null) { //mmmajor and partitioned
+ appendColumns(query, cols, false);
+ } else { // mmmajor and unpartitioned
+ query.append("*");
+ }
+ break;
+ }
+ case MINOR: {
+ appendColumns(query, cols, false);
+ }
+ }
+ }
+
+ protected void getSourceForInsert(StringBuilder query) {
Review Comment:
missing Override
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactionQueryBuilderForInsertOnly.java:
##########
@@ -0,0 +1,313 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.exec.DDLPlanUtils;
+import org.apache.hadoop.hive.ql.util.DirectionUtils;
+import org.apache.hive.common.util.HiveStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Modifier;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Builds query strings that help with query-based compaction of insert-only tables.
+ */
+class CompactionQueryBuilderForInsertOnly extends CompactionQueryBuilder {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CompactionQueryBuilderForInsertOnly.class.getName());
+
+ private StorageDescriptor storageDescriptor; // for Create in insert-only
+
+ /**
+ * Construct a CompactionQueryBuilderForInsertOnly with required params.
+ *
+ * @param compactionType major or minor or rebalance, e.g. CompactionType.MAJOR.
+ * Cannot be null.
+ * @param operation query's Operation e.g. Operation.CREATE.
+ * @throws IllegalArgumentException if compactionType is null or the compaction type is REBALANCE
+ */
+ CompactionQueryBuilderForInsertOnly(CompactionType compactionType, Operation operation, String resultTableName) {
+ super(compactionType, operation, true, resultTableName);
+ if (CompactionType.REBALANCE.equals(compactionType)) {
+ throw new IllegalArgumentException("Rebalance compaction is supported only on full ACID tables!");
+ }
+ }
+
+ /**
+ * Set the StorageDescriptor of the table or partition to compact.
+ * Required for Create operations in insert-only compaction.
+ *
+ * @param storageDescriptor StorageDescriptor of the table or partition to compact, not null
+ */
+ CompactionQueryBuilder setStorageDescriptor(StorageDescriptor storageDescriptor) {
+ this.storageDescriptor = storageDescriptor;
+ return this;
+ }
+
+ protected void buildSelectClauseForInsert(StringBuilder query) {
+ // Need list of columns for major crud, mmmajor partitioned, mmminor
+ List<FieldSchema> cols;
+ if (CompactionType.MAJOR.equals(compactionType) && sourcePartition != null || CompactionType.MINOR.equals(
+ compactionType)) {
+ if (sourceTab == null) {
+ return; // avoid NPEs, don't throw an exception but skip this part of the query
+ }
+ cols = sourceTab.getSd().getCols();
+ } else {
+ cols = null;
+ }
+ switch (compactionType) {
+ case MAJOR: {
+ if (sourcePartition != null) { //mmmajor and partitioned
+ appendColumns(query, cols, false);
+ } else { // mmmajor and unpartitioned
+ query.append("*");
+ }
+ break;
+ }
+ case MINOR: {
+ appendColumns(query, cols, false);
+ }
+ }
+ }
+
+ protected void getSourceForInsert(StringBuilder query) {
+ if (sourceTabForInsert != null) {
+ query.append(sourceTabForInsert);
+ } else {
+ query.append(sourceTab.getDbName()).append(".").append(sourceTab.getTableName());
+ }
+ query.append(" ");
+ if (CompactionType.MAJOR.equals(compactionType) && StringUtils.isNotBlank(orderByClause)) {
+ query.append(orderByClause);
+ }
+ }
+
+ protected void buildWhereClauseForInsert(StringBuilder query) {
+ if (CompactionType.MAJOR.equals(compactionType) && sourcePartition != null && sourceTab != null) {
+ List<String> vals = sourcePartition.getValues();
+ List<FieldSchema> keys = sourceTab.getPartitionKeys();
+ if (keys.size() != vals.size()) {
+ throw new IllegalStateException("source partition values (" + Arrays.toString(
+ vals.toArray()) + ") do not match source table values (" + Arrays.toString(
+ keys.toArray()) + "). Failing compaction.");
+ }
+
+ query.append(" where ");
+ for (int i = 0; i < keys.size(); ++i) {
+ FieldSchema keySchema = keys.get(i);
+ query.append(i == 0 ? "`" : " and `").append(keySchema.getName()).append("`=");
+ if (!keySchema.getType().equalsIgnoreCase(ColumnType.BOOLEAN_TYPE_NAME)) {
+ query.append("'").append(vals.get(i)).append("'");
+ } else {
+ query.append(vals.get(i));
+ }
+ }
+ }
+ }
Review Comment:
This code is dupe in both classes here & CompactionQueryBuilderForMajor, refactor and put in one place only
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactionQueryBuilderForInsertOnly.java:
##########
@@ -0,0 +1,313 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.exec.DDLPlanUtils;
+import org.apache.hadoop.hive.ql.util.DirectionUtils;
+import org.apache.hive.common.util.HiveStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Modifier;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Builds query strings that help with query-based compaction of insert-only tables.
+ */
+class CompactionQueryBuilderForInsertOnly extends CompactionQueryBuilder {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CompactionQueryBuilderForInsertOnly.class.getName());
+
+ private StorageDescriptor storageDescriptor; // for Create in insert-only
+
+ /**
+ * Construct a CompactionQueryBuilderForInsertOnly with required params.
+ *
+ * @param compactionType major or minor or rebalance, e.g. CompactionType.MAJOR.
+ * Cannot be null.
+ * @param operation query's Operation e.g. Operation.CREATE.
+ * @throws IllegalArgumentException if compactionType is null or the compaction type is REBALANCE
+ */
+ CompactionQueryBuilderForInsertOnly(CompactionType compactionType, Operation operation, String resultTableName) {
+ super(compactionType, operation, true, resultTableName);
+ if (CompactionType.REBALANCE.equals(compactionType)) {
+ throw new IllegalArgumentException("Rebalance compaction is supported only on full ACID tables!");
+ }
+ }
+
+ /**
+ * Set the StorageDescriptor of the table or partition to compact.
+ * Required for Create operations in insert-only compaction.
+ *
+ * @param storageDescriptor StorageDescriptor of the table or partition to compact, not null
+ */
+ CompactionQueryBuilder setStorageDescriptor(StorageDescriptor storageDescriptor) {
+ this.storageDescriptor = storageDescriptor;
+ return this;
+ }
+
+ protected void buildSelectClauseForInsert(StringBuilder query) {
+ // Need list of columns for major crud, mmmajor partitioned, mmminor
+ List<FieldSchema> cols;
+ if (CompactionType.MAJOR.equals(compactionType) && sourcePartition != null || CompactionType.MINOR.equals(
+ compactionType)) {
+ if (sourceTab == null) {
+ return; // avoid NPEs, don't throw an exception but skip this part of the query
+ }
+ cols = sourceTab.getSd().getCols();
+ } else {
+ cols = null;
+ }
+ switch (compactionType) {
+ case MAJOR: {
+ if (sourcePartition != null) { //mmmajor and partitioned
+ appendColumns(query, cols, false);
+ } else { // mmmajor and unpartitioned
+ query.append("*");
+ }
+ break;
+ }
+ case MINOR: {
+ appendColumns(query, cols, false);
+ }
+ }
+ }
+
+ protected void getSourceForInsert(StringBuilder query) {
+ if (sourceTabForInsert != null) {
+ query.append(sourceTabForInsert);
+ } else {
+ query.append(sourceTab.getDbName()).append(".").append(sourceTab.getTableName());
+ }
+ query.append(" ");
+ if (CompactionType.MAJOR.equals(compactionType) && StringUtils.isNotBlank(orderByClause)) {
+ query.append(orderByClause);
+ }
+ }
+
+ protected void buildWhereClauseForInsert(StringBuilder query) {
+ if (CompactionType.MAJOR.equals(compactionType) && sourcePartition != null && sourceTab != null) {
+ List<String> vals = sourcePartition.getValues();
+ List<FieldSchema> keys = sourceTab.getPartitionKeys();
+ if (keys.size() != vals.size()) {
+ throw new IllegalStateException("source partition values (" + Arrays.toString(
+ vals.toArray()) + ") do not match source table values (" + Arrays.toString(
+ keys.toArray()) + "). Failing compaction.");
+ }
+
+ query.append(" where ");
+ for (int i = 0; i < keys.size(); ++i) {
+ FieldSchema keySchema = keys.get(i);
+ query.append(i == 0 ? "`" : " and `").append(keySchema.getName()).append("`=");
+ if (!keySchema.getType().equalsIgnoreCase(ColumnType.BOOLEAN_TYPE_NAME)) {
+ query.append("'").append(vals.get(i)).append("'");
+ } else {
+ query.append(vals.get(i));
+ }
+ }
+ }
+ }
+
+ protected void getDdlForCreate(StringBuilder query) {
+ defineColumns(query);
+
+ // PARTITIONED BY. Used for parts of minor compaction.
+ if (isPartitioned) {
+ query.append(" PARTITIONED BY (`file_name` STRING) ");
+ }
+
+ // CLUSTERED BY. (bucketing)
+ getMmBucketing(query);
+
+ // SKEWED BY
+ getSkewedByClause(query);
+
+ // STORED AS / ROW FORMAT SERDE + INPUTFORMAT + OUTPUTFORMAT
+ copySerdeFromSourceTable(query);
+
+ // LOCATION
+ if (location != null) {
+ query.append(" LOCATION '").append(HiveStringUtils.escapeHiveCommand(location)).append("'");
+ }
+
+ // TBLPROPERTIES
+ addTblProperties(query);
+ }
+
+ /**
+ * Define columns of the create query.
+ */
+ private void defineColumns(StringBuilder query) {
+ if (sourceTab == null) {
+ return; // avoid NPEs, don't throw an exception but skip this part of the query
+ }
+ query.append("(");
+ List<String> columnDescs = getColumnDescs();
+ query.append(StringUtils.join(columnDescs, ','));
+ query.append(") ");
+ }
+
+ /**
+ * Part of Create operation. Copy source table bucketing for insert-only compaction.
+ */
+ private void getMmBucketing(StringBuilder query) {
+ if (sourceTab == null) {
+ return; // avoid NPEs, don't throw an exception but skip this part of the query
+ }
+ boolean isFirst;
+ List<String> buckCols = sourceTab.getSd().getBucketCols();
+ if (buckCols.size() > 0) {
+ query.append("CLUSTERED BY (").append(StringUtils.join(buckCols, ",")).append(") ");
+ List<Order> sortCols = sourceTab.getSd().getSortCols();
+ if (sortCols.size() > 0) {
Review Comment:
change to `` if (!sortCols.isEmpty()) {``
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactionQueryBuilderForInsertOnly.java:
##########
@@ -0,0 +1,313 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.exec.DDLPlanUtils;
+import org.apache.hadoop.hive.ql.util.DirectionUtils;
+import org.apache.hive.common.util.HiveStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Modifier;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Builds query strings that help with query-based compaction of insert-only tables.
+ */
+class CompactionQueryBuilderForInsertOnly extends CompactionQueryBuilder {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CompactionQueryBuilderForInsertOnly.class.getName());
+
+ private StorageDescriptor storageDescriptor; // for Create in insert-only
+
+ /**
+ * Construct a CompactionQueryBuilderForInsertOnly with required params.
+ *
+ * @param compactionType major or minor or rebalance, e.g. CompactionType.MAJOR.
+ * Cannot be null.
+ * @param operation query's Operation e.g. Operation.CREATE.
+ * @throws IllegalArgumentException if compactionType is null or the compaction type is REBALANCE
+ */
+ CompactionQueryBuilderForInsertOnly(CompactionType compactionType, Operation operation, String resultTableName) {
+ super(compactionType, operation, true, resultTableName);
+ if (CompactionType.REBALANCE.equals(compactionType)) {
+ throw new IllegalArgumentException("Rebalance compaction is supported only on full ACID tables!");
+ }
+ }
+
+ /**
+ * Set the StorageDescriptor of the table or partition to compact.
+ * Required for Create operations in insert-only compaction.
+ *
+ * @param storageDescriptor StorageDescriptor of the table or partition to compact, not null
+ */
+ CompactionQueryBuilder setStorageDescriptor(StorageDescriptor storageDescriptor) {
+ this.storageDescriptor = storageDescriptor;
+ return this;
+ }
+
+ protected void buildSelectClauseForInsert(StringBuilder query) {
+ // Need list of columns for major crud, mmmajor partitioned, mmminor
+ List<FieldSchema> cols;
+ if (CompactionType.MAJOR.equals(compactionType) && sourcePartition != null || CompactionType.MINOR.equals(
+ compactionType)) {
+ if (sourceTab == null) {
+ return; // avoid NPEs, don't throw an exception but skip this part of the query
+ }
+ cols = sourceTab.getSd().getCols();
+ } else {
+ cols = null;
+ }
Review Comment:
Can't we adjust this same logic inside the switch case? Otherwise this if check logic looks like redundant
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactionQueryBuilderForInsertOnly.java:
##########
@@ -0,0 +1,313 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.exec.DDLPlanUtils;
+import org.apache.hadoop.hive.ql.util.DirectionUtils;
+import org.apache.hive.common.util.HiveStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Modifier;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Builds query strings that help with query-based compaction of insert-only tables.
+ */
+class CompactionQueryBuilderForInsertOnly extends CompactionQueryBuilder {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CompactionQueryBuilderForInsertOnly.class.getName());
+
+ private StorageDescriptor storageDescriptor; // for Create in insert-only
+
+ /**
+ * Construct a CompactionQueryBuilderForInsertOnly with required params.
+ *
+ * @param compactionType major or minor or rebalance, e.g. CompactionType.MAJOR.
+ * Cannot be null.
+ * @param operation query's Operation e.g. Operation.CREATE.
+ * @throws IllegalArgumentException if compactionType is null or the compaction type is REBALANCE
+ */
+ CompactionQueryBuilderForInsertOnly(CompactionType compactionType, Operation operation, String resultTableName) {
+ super(compactionType, operation, true, resultTableName);
+ if (CompactionType.REBALANCE.equals(compactionType)) {
+ throw new IllegalArgumentException("Rebalance compaction is supported only on full ACID tables!");
+ }
+ }
+
+ /**
+ * Set the StorageDescriptor of the table or partition to compact.
+ * Required for Create operations in insert-only compaction.
+ *
+ * @param storageDescriptor StorageDescriptor of the table or partition to compact, not null
+ */
+ CompactionQueryBuilder setStorageDescriptor(StorageDescriptor storageDescriptor) {
+ this.storageDescriptor = storageDescriptor;
+ return this;
+ }
+
+ protected void buildSelectClauseForInsert(StringBuilder query) {
+ // Need list of columns for major crud, mmmajor partitioned, mmminor
+ List<FieldSchema> cols;
+ if (CompactionType.MAJOR.equals(compactionType) && sourcePartition != null || CompactionType.MINOR.equals(
+ compactionType)) {
+ if (sourceTab == null) {
+ return; // avoid NPEs, don't throw an exception but skip this part of the query
+ }
+ cols = sourceTab.getSd().getCols();
+ } else {
+ cols = null;
+ }
+ switch (compactionType) {
+ case MAJOR: {
+ if (sourcePartition != null) { //mmmajor and partitioned
+ appendColumns(query, cols, false);
+ } else { // mmmajor and unpartitioned
+ query.append("*");
+ }
+ break;
+ }
+ case MINOR: {
+ appendColumns(query, cols, false);
+ }
+ }
+ }
+
+ protected void getSourceForInsert(StringBuilder query) {
+ if (sourceTabForInsert != null) {
+ query.append(sourceTabForInsert);
+ } else {
+ query.append(sourceTab.getDbName()).append(".").append(sourceTab.getTableName());
+ }
+ query.append(" ");
+ if (CompactionType.MAJOR.equals(compactionType) && StringUtils.isNotBlank(orderByClause)) {
+ query.append(orderByClause);
+ }
+ }
+
+ protected void buildWhereClauseForInsert(StringBuilder query) {
+ if (CompactionType.MAJOR.equals(compactionType) && sourcePartition != null && sourceTab != null) {
+ List<String> vals = sourcePartition.getValues();
+ List<FieldSchema> keys = sourceTab.getPartitionKeys();
+ if (keys.size() != vals.size()) {
+ throw new IllegalStateException("source partition values (" + Arrays.toString(
+ vals.toArray()) + ") do not match source table values (" + Arrays.toString(
+ keys.toArray()) + "). Failing compaction.");
+ }
+
+ query.append(" where ");
+ for (int i = 0; i < keys.size(); ++i) {
+ FieldSchema keySchema = keys.get(i);
+ query.append(i == 0 ? "`" : " and `").append(keySchema.getName()).append("`=");
+ if (!keySchema.getType().equalsIgnoreCase(ColumnType.BOOLEAN_TYPE_NAME)) {
+ query.append("'").append(vals.get(i)).append("'");
+ } else {
+ query.append(vals.get(i));
+ }
+ }
+ }
+ }
+
+ protected void getDdlForCreate(StringBuilder query) {
+ defineColumns(query);
+
+ // PARTITIONED BY. Used for parts of minor compaction.
+ if (isPartitioned) {
+ query.append(" PARTITIONED BY (`file_name` STRING) ");
+ }
+
+ // CLUSTERED BY. (bucketing)
+ getMmBucketing(query);
+
+ // SKEWED BY
+ getSkewedByClause(query);
+
+ // STORED AS / ROW FORMAT SERDE + INPUTFORMAT + OUTPUTFORMAT
+ copySerdeFromSourceTable(query);
+
+ // LOCATION
+ if (location != null) {
+ query.append(" LOCATION '").append(HiveStringUtils.escapeHiveCommand(location)).append("'");
+ }
+
+ // TBLPROPERTIES
+ addTblProperties(query);
+ }
+
+ /**
+ * Define columns of the create query.
+ */
+ private void defineColumns(StringBuilder query) {
+ if (sourceTab == null) {
+ return; // avoid NPEs, don't throw an exception but skip this part of the query
+ }
+ query.append("(");
+ List<String> columnDescs = getColumnDescs();
+ query.append(StringUtils.join(columnDescs, ','));
+ query.append(") ");
+ }
+
+ /**
+ * Part of Create operation. Copy source table bucketing for insert-only compaction.
+ */
+ private void getMmBucketing(StringBuilder query) {
+ if (sourceTab == null) {
+ return; // avoid NPEs, don't throw an exception but skip this part of the query
+ }
+ boolean isFirst;
+ List<String> buckCols = sourceTab.getSd().getBucketCols();
+ if (buckCols.size() > 0) {
Review Comment:
Change to `` if (!buckCols.isEmpty()) {``
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactionQueryBuilderForInsertOnly.java:
##########
@@ -0,0 +1,313 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.exec.DDLPlanUtils;
+import org.apache.hadoop.hive.ql.util.DirectionUtils;
+import org.apache.hive.common.util.HiveStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Modifier;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Builds query strings that help with query-based compaction of insert-only tables.
+ */
+class CompactionQueryBuilderForInsertOnly extends CompactionQueryBuilder {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CompactionQueryBuilderForInsertOnly.class.getName());
+
+ private StorageDescriptor storageDescriptor; // for Create in insert-only
+
+ /**
+ * Construct a CompactionQueryBuilderForInsertOnly with required params.
+ *
+ * @param compactionType major or minor or rebalance, e.g. CompactionType.MAJOR.
+ * Cannot be null.
+ * @param operation query's Operation e.g. Operation.CREATE.
+ * @throws IllegalArgumentException if compactionType is null or the compaction type is REBALANCE
+ */
+ CompactionQueryBuilderForInsertOnly(CompactionType compactionType, Operation operation, String resultTableName) {
+ super(compactionType, operation, true, resultTableName);
+ if (CompactionType.REBALANCE.equals(compactionType)) {
+ throw new IllegalArgumentException("Rebalance compaction is supported only on full ACID tables!");
+ }
+ }
+
+ /**
+ * Set the StorageDescriptor of the table or partition to compact.
+ * Required for Create operations in insert-only compaction.
+ *
+ * @param storageDescriptor StorageDescriptor of the table or partition to compact, not null
+ */
+ CompactionQueryBuilder setStorageDescriptor(StorageDescriptor storageDescriptor) {
+ this.storageDescriptor = storageDescriptor;
+ return this;
+ }
+
+ protected void buildSelectClauseForInsert(StringBuilder query) {
+ // Need list of columns for major crud, mmmajor partitioned, mmminor
+ List<FieldSchema> cols;
+ if (CompactionType.MAJOR.equals(compactionType) && sourcePartition != null || CompactionType.MINOR.equals(
+ compactionType)) {
+ if (sourceTab == null) {
+ return; // avoid NPEs, don't throw an exception but skip this part of the query
+ }
+ cols = sourceTab.getSd().getCols();
+ } else {
+ cols = null;
+ }
+ switch (compactionType) {
+ case MAJOR: {
+ if (sourcePartition != null) { //mmmajor and partitioned
+ appendColumns(query, cols, false);
+ } else { // mmmajor and unpartitioned
+ query.append("*");
+ }
+ break;
+ }
+ case MINOR: {
+ appendColumns(query, cols, false);
+ }
+ }
+ }
+
+ protected void getSourceForInsert(StringBuilder query) {
+ if (sourceTabForInsert != null) {
+ query.append(sourceTabForInsert);
+ } else {
+ query.append(sourceTab.getDbName()).append(".").append(sourceTab.getTableName());
+ }
+ query.append(" ");
+ if (CompactionType.MAJOR.equals(compactionType) && StringUtils.isNotBlank(orderByClause)) {
+ query.append(orderByClause);
+ }
+ }
+
+ protected void buildWhereClauseForInsert(StringBuilder query) {
+ if (CompactionType.MAJOR.equals(compactionType) && sourcePartition != null && sourceTab != null) {
+ List<String> vals = sourcePartition.getValues();
+ List<FieldSchema> keys = sourceTab.getPartitionKeys();
+ if (keys.size() != vals.size()) {
+ throw new IllegalStateException("source partition values (" + Arrays.toString(
+ vals.toArray()) + ") do not match source table values (" + Arrays.toString(
+ keys.toArray()) + "). Failing compaction.");
+ }
+
+ query.append(" where ");
+ for (int i = 0; i < keys.size(); ++i) {
Review Comment:
keys.size() is being computed twice, here & above
```
if (keys.size() != vals.size()) {
```
can refactor into a variable and use at both places
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactionQueryBuilderForInsertOnly.java:
##########
@@ -0,0 +1,313 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.exec.DDLPlanUtils;
+import org.apache.hadoop.hive.ql.util.DirectionUtils;
+import org.apache.hive.common.util.HiveStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Modifier;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Builds query strings that help with query-based compaction of insert-only tables.
+ */
+class CompactionQueryBuilderForInsertOnly extends CompactionQueryBuilder {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CompactionQueryBuilderForInsertOnly.class.getName());
+
+ private StorageDescriptor storageDescriptor; // for Create in insert-only
+
+ /**
+ * Construct a CompactionQueryBuilderForInsertOnly with required params.
+ *
+ * @param compactionType major or minor or rebalance, e.g. CompactionType.MAJOR.
+ * Cannot be null.
+ * @param operation query's Operation e.g. Operation.CREATE.
+ * @throws IllegalArgumentException if compactionType is null or the compaction type is REBALANCE
+ */
+ CompactionQueryBuilderForInsertOnly(CompactionType compactionType, Operation operation, String resultTableName) {
+ super(compactionType, operation, true, resultTableName);
+ if (CompactionType.REBALANCE.equals(compactionType)) {
+ throw new IllegalArgumentException("Rebalance compaction is supported only on full ACID tables!");
+ }
+ }
+
+ /**
+ * Set the StorageDescriptor of the table or partition to compact.
+ * Required for Create operations in insert-only compaction.
+ *
+ * @param storageDescriptor StorageDescriptor of the table or partition to compact, not null
+ */
+ CompactionQueryBuilder setStorageDescriptor(StorageDescriptor storageDescriptor) {
+ this.storageDescriptor = storageDescriptor;
+ return this;
+ }
+
+ protected void buildSelectClauseForInsert(StringBuilder query) {
+ // Need list of columns for major crud, mmmajor partitioned, mmminor
+ List<FieldSchema> cols;
+ if (CompactionType.MAJOR.equals(compactionType) && sourcePartition != null || CompactionType.MINOR.equals(
+ compactionType)) {
+ if (sourceTab == null) {
+ return; // avoid NPEs, don't throw an exception but skip this part of the query
+ }
+ cols = sourceTab.getSd().getCols();
+ } else {
+ cols = null;
+ }
+ switch (compactionType) {
+ case MAJOR: {
+ if (sourcePartition != null) { //mmmajor and partitioned
+ appendColumns(query, cols, false);
+ } else { // mmmajor and unpartitioned
+ query.append("*");
+ }
+ break;
+ }
+ case MINOR: {
+ appendColumns(query, cols, false);
+ }
+ }
+ }
+
+ protected void getSourceForInsert(StringBuilder query) {
+ if (sourceTabForInsert != null) {
+ query.append(sourceTabForInsert);
+ } else {
+ query.append(sourceTab.getDbName()).append(".").append(sourceTab.getTableName());
+ }
+ query.append(" ");
+ if (CompactionType.MAJOR.equals(compactionType) && StringUtils.isNotBlank(orderByClause)) {
+ query.append(orderByClause);
+ }
+ }
+
+ protected void buildWhereClauseForInsert(StringBuilder query) {
Review Comment:
Missing override
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactionQueryBuilderForMinor.java:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.ql.io.AcidUtils;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hive.common.util.HiveStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Builds query strings that help with query-based MINOR compaction of CRUD.
+ */
+class CompactionQueryBuilderForMinor extends CompactionQueryBuilder {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CompactionQueryBuilderForMinor.class.getName());
+
+ /**
+ * Construct a CompactionQueryBuilderMinor with required params.
+ *
+ * @param compactionType major or minor or rebalance, e.g. CompactionType.MAJOR.
+ * Cannot be null.
+ * @param operation query's Operation e.g. Operation.CREATE.
+ * @throws IllegalArgumentException if compactionType is null
+ */
+ CompactionQueryBuilderForMinor(CompactionType compactionType, Operation operation, String resultTableName) {
+ super(compactionType, operation, false, resultTableName);
+ }
+
+ protected void buildSelectClauseForInsert(StringBuilder query) {
+ // Need list of columns for major crud, mmmajor partitioned, mmminor
+ query.append("`operation`, `originalTransaction`, `bucket`, `rowId`, `currentTransaction`, `row`");
+ }
+
+ protected void getSourceForInsert(StringBuilder query) {
+ if (sourceTabForInsert != null) {
+ query.append(sourceTabForInsert);
+ } else {
+ query.append(sourceTab.getDbName()).append(".").append(sourceTab.getTableName());
+ }
+ query.append(" ");
+ }
+
+ protected void buildWhereClauseForInsert(StringBuilder query) {
+ if (validWriteIdList != null) {
+ if (validWriteIdList != null) {
Review Comment:
Why it is there twice?
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactionQueryBuilderForMinor.java:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.ql.io.AcidUtils;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hive.common.util.HiveStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Builds query strings that help with query-based MINOR compaction of CRUD.
+ */
+class CompactionQueryBuilderForMinor extends CompactionQueryBuilder {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CompactionQueryBuilderForMinor.class.getName());
+
+ /**
+ * Construct a CompactionQueryBuilderMinor with required params.
+ *
+ * @param compactionType major or minor or rebalance, e.g. CompactionType.MAJOR.
+ * Cannot be null.
+ * @param operation query's Operation e.g. Operation.CREATE.
+ * @throws IllegalArgumentException if compactionType is null
+ */
+ CompactionQueryBuilderForMinor(CompactionType compactionType, Operation operation, String resultTableName) {
+ super(compactionType, operation, false, resultTableName);
+ }
+
+ protected void buildSelectClauseForInsert(StringBuilder query) {
Review Comment:
missing Override annotation here & other, please add them for all other overriden methods as well
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactionQueryBuilderForInsertOnly.java:
##########
@@ -0,0 +1,313 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.exec.DDLPlanUtils;
+import org.apache.hadoop.hive.ql.util.DirectionUtils;
+import org.apache.hive.common.util.HiveStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Modifier;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Builds query strings that help with query-based compaction of insert-only tables.
+ */
+class CompactionQueryBuilderForInsertOnly extends CompactionQueryBuilder {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CompactionQueryBuilderForInsertOnly.class.getName());
+
+ private StorageDescriptor storageDescriptor; // for Create in insert-only
+
+ /**
+ * Construct a CompactionQueryBuilderForInsertOnly with required params.
+ *
+ * @param compactionType major or minor or rebalance, e.g. CompactionType.MAJOR.
+ * Cannot be null.
+ * @param operation query's Operation e.g. Operation.CREATE.
+ * @throws IllegalArgumentException if compactionType is null or the compaction type is REBALANCE
+ */
+ CompactionQueryBuilderForInsertOnly(CompactionType compactionType, Operation operation, String resultTableName) {
+ super(compactionType, operation, true, resultTableName);
+ if (CompactionType.REBALANCE.equals(compactionType)) {
+ throw new IllegalArgumentException("Rebalance compaction is supported only on full ACID tables!");
+ }
+ }
+
+ /**
+ * Set the StorageDescriptor of the table or partition to compact.
+ * Required for Create operations in insert-only compaction.
+ *
+ * @param storageDescriptor StorageDescriptor of the table or partition to compact, not null
+ */
+ CompactionQueryBuilder setStorageDescriptor(StorageDescriptor storageDescriptor) {
+ this.storageDescriptor = storageDescriptor;
+ return this;
+ }
+
+ protected void buildSelectClauseForInsert(StringBuilder query) {
+ // Need list of columns for major crud, mmmajor partitioned, mmminor
+ List<FieldSchema> cols;
+ if (CompactionType.MAJOR.equals(compactionType) && sourcePartition != null || CompactionType.MINOR.equals(
+ compactionType)) {
+ if (sourceTab == null) {
+ return; // avoid NPEs, don't throw an exception but skip this part of the query
+ }
+ cols = sourceTab.getSd().getCols();
+ } else {
+ cols = null;
+ }
+ switch (compactionType) {
+ case MAJOR: {
+ if (sourcePartition != null) { //mmmajor and partitioned
+ appendColumns(query, cols, false);
+ } else { // mmmajor and unpartitioned
+ query.append("*");
+ }
+ break;
+ }
+ case MINOR: {
+ appendColumns(query, cols, false);
+ }
+ }
+ }
+
+ protected void getSourceForInsert(StringBuilder query) {
+ if (sourceTabForInsert != null) {
+ query.append(sourceTabForInsert);
+ } else {
+ query.append(sourceTab.getDbName()).append(".").append(sourceTab.getTableName());
+ }
+ query.append(" ");
+ if (CompactionType.MAJOR.equals(compactionType) && StringUtils.isNotBlank(orderByClause)) {
+ query.append(orderByClause);
+ }
+ }
+
+ protected void buildWhereClauseForInsert(StringBuilder query) {
+ if (CompactionType.MAJOR.equals(compactionType) && sourcePartition != null && sourceTab != null) {
+ List<String> vals = sourcePartition.getValues();
+ List<FieldSchema> keys = sourceTab.getPartitionKeys();
+ if (keys.size() != vals.size()) {
+ throw new IllegalStateException("source partition values (" + Arrays.toString(
+ vals.toArray()) + ") do not match source table values (" + Arrays.toString(
+ keys.toArray()) + "). Failing compaction.");
+ }
+
+ query.append(" where ");
+ for (int i = 0; i < keys.size(); ++i) {
+ FieldSchema keySchema = keys.get(i);
+ query.append(i == 0 ? "`" : " and `").append(keySchema.getName()).append("`=");
+ if (!keySchema.getType().equalsIgnoreCase(ColumnType.BOOLEAN_TYPE_NAME)) {
+ query.append("'").append(vals.get(i)).append("'");
+ } else {
+ query.append(vals.get(i));
+ }
+ }
+ }
+ }
+
+ protected void getDdlForCreate(StringBuilder query) {
Review Comment:
add override annotation
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactionQueryBuilderForMajor.java:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.ql.io.AcidUtils;
+import org.apache.hive.common.util.HiveStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Builds query strings that help with query-based MAJOR compaction of CRUD.
+ */
+class CompactionQueryBuilderForMajor extends CompactionQueryBuilder {
Review Comment:
+1
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactionQueryBuilderForMinor.java:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.ql.io.AcidUtils;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hive.common.util.HiveStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Builds query strings that help with query-based MINOR compaction of CRUD.
+ */
+class CompactionQueryBuilderForMinor extends CompactionQueryBuilder {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CompactionQueryBuilderForMinor.class.getName());
+
+ /**
+ * Construct a CompactionQueryBuilderMinor with required params.
+ *
+ * @param compactionType major or minor or rebalance, e.g. CompactionType.MAJOR.
+ * Cannot be null.
+ * @param operation query's Operation e.g. Operation.CREATE.
+ * @throws IllegalArgumentException if compactionType is null
+ */
+ CompactionQueryBuilderForMinor(CompactionType compactionType, Operation operation, String resultTableName) {
+ super(compactionType, operation, false, resultTableName);
+ }
+
+ protected void buildSelectClauseForInsert(StringBuilder query) {
+ // Need list of columns for major crud, mmmajor partitioned, mmminor
+ query.append("`operation`, `originalTransaction`, `bucket`, `rowId`, `currentTransaction`, `row`");
+ }
+
+ protected void getSourceForInsert(StringBuilder query) {
+ if (sourceTabForInsert != null) {
+ query.append(sourceTabForInsert);
+ } else {
+ query.append(sourceTab.getDbName()).append(".").append(sourceTab.getTableName());
+ }
+ query.append(" ");
+ }
+
+ protected void buildWhereClauseForInsert(StringBuilder query) {
+ if (validWriteIdList != null) {
+ if (validWriteIdList != null) {
+ long[] invalidWriteIds = validWriteIdList.getInvalidWriteIds();
+ if (invalidWriteIds.length > 0) {
+ query.append(" where `originalTransaction` not in (")
+ .append(StringUtils.join(ArrayUtils.toObject(invalidWriteIds), ",")).append(")");
+ }
+ }
+ }
+ }
+
+ protected void getDdlForCreate(StringBuilder query) {
+ defineColumns(query);
+
+ // PARTITIONED BY. Used for parts of minor compaction.
+ if (isPartitioned) {
+ query.append(" PARTITIONED BY (`file_name` STRING) ");
+ }
+
+ // CLUSTERED BY. (bucketing)
+ int bucketingVersion = 0;
+ if (CompactionType.MINOR.equals(compactionType)) {
+ bucketingVersion = getMinorCrudBucketing(query, bucketingVersion);
+ }
+
+ // STORED AS / ROW FORMAT SERDE + INPUTFORMAT + OUTPUTFORMAT
+ query.append(" stored as orc");
+
+ // LOCATION
+ if (location != null) {
+ query.append(" LOCATION '").append(HiveStringUtils.escapeHiveCommand(location)).append("'");
+ }
+
+ // TBLPROPERTIES
+ addTblProperties(query, bucketingVersion);
+ }
+
+ /**
+ * Define columns of the create query.
+ */
+ private void defineColumns(StringBuilder query) {
+ if (sourceTab == null) {
Review Comment:
Why are doing this null & return here and even other places can't we wrap our logic within if(sourceTab!=null)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27956: Query based compactor implementation separation [hive]
Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4952:
URL: https://github.com/apache/hive/pull/4952#discussion_r1431236510
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactionQueryBuilderForInsertOnly.java:
##########
@@ -0,0 +1,313 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.exec.DDLPlanUtils;
+import org.apache.hadoop.hive.ql.util.DirectionUtils;
+import org.apache.hive.common.util.HiveStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Modifier;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Builds query strings that help with query-based compaction of insert-only tables.
+ */
+class CompactionQueryBuilderForInsertOnly extends CompactionQueryBuilder {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CompactionQueryBuilderForInsertOnly.class.getName());
Review Comment:
not used
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactionQueryBuilderForMajor.java:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.ql.io.AcidUtils;
+import org.apache.hive.common.util.HiveStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Builds query strings that help with query-based MAJOR compaction of CRUD.
+ */
+class CompactionQueryBuilderForMajor extends CompactionQueryBuilder {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CompactionQueryBuilderForMajor.class.getName());
+
+ /**
+ * Construct a CompactionQueryBuilderForMajor with required params.
+ *
+ * @param compactionType major or minor or rebalance, e.g. CompactionType.MAJOR.
+ * Cannot be null.
+ * @param operation query's Operation e.g. Operation.CREATE.
+ * @throws IllegalArgumentException if compactionType is null
+ */
+ CompactionQueryBuilderForMajor(CompactionType compactionType, Operation operation, String resultTableName) {
+ super(compactionType, operation, false, resultTableName);
+ }
Review Comment:
`CompactionType.Major` could be passed directly into super call (hardcoded) instead of taking it as a parameter. Passing other values can cause this class to output incorrect SQL. The same applies for the other implementations.
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactionQueryBuilderForInsertOnly.java:
##########
@@ -0,0 +1,313 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.exec.DDLPlanUtils;
+import org.apache.hadoop.hive.ql.util.DirectionUtils;
+import org.apache.hive.common.util.HiveStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Modifier;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Builds query strings that help with query-based compaction of insert-only tables.
+ */
+class CompactionQueryBuilderForInsertOnly extends CompactionQueryBuilder {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CompactionQueryBuilderForInsertOnly.class.getName());
+
+ private StorageDescriptor storageDescriptor; // for Create in insert-only
+
+ /**
+ * Construct a CompactionQueryBuilderForInsertOnly with required params.
+ *
+ * @param compactionType major or minor or rebalance, e.g. CompactionType.MAJOR.
+ * Cannot be null.
+ * @param operation query's Operation e.g. Operation.CREATE.
+ * @throws IllegalArgumentException if compactionType is null or the compaction type is REBALANCE
+ */
+ CompactionQueryBuilderForInsertOnly(CompactionType compactionType, Operation operation, String resultTableName) {
+ super(compactionType, operation, true, resultTableName);
+ if (CompactionType.REBALANCE.equals(compactionType)) {
+ throw new IllegalArgumentException("Rebalance compaction is supported only on full ACID tables!");
+ }
+ }
+
+ /**
+ * Set the StorageDescriptor of the table or partition to compact.
+ * Required for Create operations in insert-only compaction.
+ *
+ * @param storageDescriptor StorageDescriptor of the table or partition to compact, not null
+ */
+ CompactionQueryBuilder setStorageDescriptor(StorageDescriptor storageDescriptor) {
+ this.storageDescriptor = storageDescriptor;
+ return this;
+ }
+
+ protected void buildSelectClauseForInsert(StringBuilder query) {
+ // Need list of columns for major crud, mmmajor partitioned, mmminor
+ List<FieldSchema> cols;
+ if (CompactionType.MAJOR.equals(compactionType) && sourcePartition != null || CompactionType.MINOR.equals(
+ compactionType)) {
+ if (sourceTab == null) {
+ return; // avoid NPEs, don't throw an exception but skip this part of the query
+ }
+ cols = sourceTab.getSd().getCols();
+ } else {
+ cols = null;
+ }
+ switch (compactionType) {
+ case MAJOR: {
+ if (sourcePartition != null) { //mmmajor and partitioned
+ appendColumns(query, cols, false);
+ } else { // mmmajor and unpartitioned
+ query.append("*");
+ }
+ break;
+ }
+ case MINOR: {
+ appendColumns(query, cols, false);
+ }
+ }
+ }
+
+ protected void getSourceForInsert(StringBuilder query) {
+ if (sourceTabForInsert != null) {
+ query.append(sourceTabForInsert);
+ } else {
+ query.append(sourceTab.getDbName()).append(".").append(sourceTab.getTableName());
+ }
+ query.append(" ");
+ if (CompactionType.MAJOR.equals(compactionType) && StringUtils.isNotBlank(orderByClause)) {
+ query.append(orderByClause);
+ }
+ }
+
+ protected void buildWhereClauseForInsert(StringBuilder query) {
+ if (CompactionType.MAJOR.equals(compactionType) && sourcePartition != null && sourceTab != null) {
+ List<String> vals = sourcePartition.getValues();
+ List<FieldSchema> keys = sourceTab.getPartitionKeys();
+ if (keys.size() != vals.size()) {
+ throw new IllegalStateException("source partition values (" + Arrays.toString(
+ vals.toArray()) + ") do not match source table values (" + Arrays.toString(
+ keys.toArray()) + "). Failing compaction.");
+ }
+
+ query.append(" where ");
+ for (int i = 0; i < keys.size(); ++i) {
+ FieldSchema keySchema = keys.get(i);
+ query.append(i == 0 ? "`" : " and `").append(keySchema.getName()).append("`=");
+ if (!keySchema.getType().equalsIgnoreCase(ColumnType.BOOLEAN_TYPE_NAME)) {
+ query.append("'").append(vals.get(i)).append("'");
+ } else {
+ query.append(vals.get(i));
+ }
+ }
+ }
+ }
+
+ protected void getDdlForCreate(StringBuilder query) {
+ defineColumns(query);
+
+ // PARTITIONED BY. Used for parts of minor compaction.
+ if (isPartitioned) {
+ query.append(" PARTITIONED BY (`file_name` STRING) ");
+ }
+
+ // CLUSTERED BY. (bucketing)
+ getMmBucketing(query);
+
+ // SKEWED BY
+ getSkewedByClause(query);
+
+ // STORED AS / ROW FORMAT SERDE + INPUTFORMAT + OUTPUTFORMAT
+ copySerdeFromSourceTable(query);
+
+ // LOCATION
+ if (location != null) {
+ query.append(" LOCATION '").append(HiveStringUtils.escapeHiveCommand(location)).append("'");
+ }
+
+ // TBLPROPERTIES
+ addTblProperties(query);
+ }
+
+ /**
+ * Define columns of the create query.
+ */
+ private void defineColumns(StringBuilder query) {
+ if (sourceTab == null) {
+ return; // avoid NPEs, don't throw an exception but skip this part of the query
+ }
+ query.append("(");
+ List<String> columnDescs = getColumnDescs();
+ query.append(StringUtils.join(columnDescs, ','));
+ query.append(") ");
+ }
+
+ /**
+ * Part of Create operation. Copy source table bucketing for insert-only compaction.
+ */
+ private void getMmBucketing(StringBuilder query) {
+ if (sourceTab == null) {
+ return; // avoid NPEs, don't throw an exception but skip this part of the query
+ }
+ boolean isFirst;
+ List<String> buckCols = sourceTab.getSd().getBucketCols();
+ if (buckCols.size() > 0) {
+ query.append("CLUSTERED BY (").append(StringUtils.join(buckCols, ",")).append(") ");
+ List<Order> sortCols = sourceTab.getSd().getSortCols();
+ if (sortCols.size() > 0) {
Review Comment:
Maybe `!buckCols.isEmpty()`
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactionQueryBuilderForMajor.java:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.ql.io.AcidUtils;
+import org.apache.hive.common.util.HiveStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Builds query strings that help with query-based MAJOR compaction of CRUD.
+ */
+class CompactionQueryBuilderForMajor extends CompactionQueryBuilder {
Review Comment:
getDdlForCreate(), defineColumns(), addTblProperties() are exactly the same as in the Rebalance implementation. You may check if the duplications can be removed without over engineering it.
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactionQueryBuilderForRebalance.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.ql.io.AcidUtils;
+import org.apache.hive.common.util.HiveStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.*;
+
+/**
+ * Builds query strings that help with REBALANCE compaction of CRUD.
+ */
+class CompactionQueryBuilderForRebalance extends CompactionQueryBuilder {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CompactionQueryBuilderForRebalance.class.getName());
Review Comment:
Not used
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactionQueryBuilderForMajor.java:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.ql.io.AcidUtils;
+import org.apache.hive.common.util.HiveStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Builds query strings that help with query-based MAJOR compaction of CRUD.
+ */
+class CompactionQueryBuilderForMajor extends CompactionQueryBuilder {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CompactionQueryBuilderForMajor.class.getName());
Review Comment:
Not used
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MajorQueryCompactor.java:
##########
@@ -72,10 +72,9 @@ public boolean run(CompactorContext context) throws IOException {
* See {@link org.apache.hadoop.hive.conf.HiveConf.ConfVars#SPLIT_GROUPING_MODE} for the config description.
*/
private List<String> getCreateQueries(String fullName, Table t, String tmpTableLocation) {
- return Lists.newArrayList(new CompactionQueryBuilder(
+ return Lists.newArrayList(new CompactionQueryBuilderForMajor(
Review Comment:
You may create a factory class for returning the specific `CompactionQueryBuilder` implementations. Using this approach have some benefit:
- the decisive logic can be hidden in the factory method
- the implementation classes can be hidden from the compactors (the factory returns the base class), especially if you move the builders into a sub-package. better decoupling between compactors and query builders.
- the `getCreateQueries(), getCompactionQueries(), getDropQueries() `method can be moved to `QueryCompactor`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27956: Query based compactor implementation separation [hive]
Posted by "kuczoram (via GitHub)" <gi...@apache.org>.
kuczoram commented on code in PR #4952:
URL: https://github.com/apache/hive/pull/4952#discussion_r1560957112
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactionQueryBuilderForInsertOnly.java:
##########
@@ -0,0 +1,313 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.exec.DDLPlanUtils;
+import org.apache.hadoop.hive.ql.util.DirectionUtils;
+import org.apache.hive.common.util.HiveStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Modifier;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Builds query strings that help with query-based compaction of insert-only tables.
+ */
+class CompactionQueryBuilderForInsertOnly extends CompactionQueryBuilder {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CompactionQueryBuilderForInsertOnly.class.getName());
+
+ private StorageDescriptor storageDescriptor; // for Create in insert-only
+
+ /**
+ * Construct a CompactionQueryBuilderForInsertOnly with required params.
+ *
+ * @param compactionType major or minor or rebalance, e.g. CompactionType.MAJOR.
+ * Cannot be null.
+ * @param operation query's Operation e.g. Operation.CREATE.
+ * @throws IllegalArgumentException if compactionType is null or the compaction type is REBALANCE
+ */
+ CompactionQueryBuilderForInsertOnly(CompactionType compactionType, Operation operation, String resultTableName) {
+ super(compactionType, operation, true, resultTableName);
+ if (CompactionType.REBALANCE.equals(compactionType)) {
+ throw new IllegalArgumentException("Rebalance compaction is supported only on full ACID tables!");
+ }
+ }
+
+ /**
+ * Set the StorageDescriptor of the table or partition to compact.
+ * Required for Create operations in insert-only compaction.
+ *
+ * @param storageDescriptor StorageDescriptor of the table or partition to compact, not null
+ */
+ CompactionQueryBuilder setStorageDescriptor(StorageDescriptor storageDescriptor) {
+ this.storageDescriptor = storageDescriptor;
+ return this;
+ }
+
+ protected void buildSelectClauseForInsert(StringBuilder query) {
Review Comment:
Hi @ayushtkn ,
I am so sorry, I completely missed to response here. Huge thanks for the review! I went through all of your comments and fixed the issues. I just somehow forgot to answer here. As this PR became very old, I opened a new one: https://github.com/apache/hive/pull/5192. That one contains the original changes and the adjustments based on the review comments and also a unit test for the query generation. If you have some time, I would be very happy if you could review it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org