You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/03/25 14:35:57 UTC
[GitHub] [incubator-doris] morningman opened a new pull request #3201:
[Bug][Refactor] Fix the conflict of temp partition and dynamic partition
operations
morningman opened a new pull request #3201: [Bug][Refactor] Fix the conflict of temp partition and dynamic partition operations
URL: https://github.com/apache/incubator-doris/pull/3201
The bug is described in issue: #3200.
This CL solve the problem by:
1. Refactor the alter operation conflict checking logic by introducing new classes `AlterOperations` and `AlterOpType`.
2. Allow add/drop temporary partition when dynamic partition feature is enabled.
3. Allow modifying table's property when there is temporary partition in table.
4. Make the properties `dynamic_partition.enable` optional, and default is true.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request
#3201: [Bug][Refactor] Fix the conflict of temp partition and dynamic
partition operations
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on a change in pull request #3201: [Bug][Refactor] Fix the conflict of temp partition and dynamic partition operations
URL: https://github.com/apache/incubator-doris/pull/3201#discussion_r398564708
##########
File path: fe/src/main/java/org/apache/doris/alter/AlterOpType.java
##########
@@ -0,0 +1,62 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.alter;
+
+public enum AlterOpType {
+ // rollup
+ ADD_ROLLUP,
+ DROP_ROLLUP,
+ // schema change
+ SCHEMA_CHANGE,
+ // partition
+ ADD_PARTITION,
+ DROP_PARTITION,
+ REPLACE_PARTITION,
+ MODIFY_PARTITION,
+ // rename
+ RENAME,
+ // table property
+ MODIFY_TABLE_PROPERTY,
+ MODIFY_TABLE_PROPERTY_SYNC, // Some operations are performed synchronously, so we distinguish them by suffix _SYNC
+ // others operation, such as add/drop backend. currently we do not care about them
+ ALTER_OTHER,
+
+ INVALID_OP;
+
+ // true means 2 operations have no conflict.
+ public static Boolean[][] CONFLICT_MATRIX;
Review comment:
If true means 2 operations have no conflict. which should be `compatibilityMatrix`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #3201:
[Bug][Refactor] Fix the conflict of temp partition and dynamic partition
operations
Posted by GitBox <gi...@apache.org>.
morningman merged pull request #3201: [Bug][Refactor] Fix the conflict of temp partition and dynamic partition operations
URL: https://github.com/apache/incubator-doris/pull/3201
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request
#3201: [Bug][Refactor] Fix the conflict of temp partition and dynamic
partition operations
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #3201: [Bug][Refactor] Fix the conflict of temp partition and dynamic partition operations
URL: https://github.com/apache/incubator-doris/pull/3201#discussion_r398628232
##########
File path: fe/src/main/java/org/apache/doris/alter/AlterOpType.java
##########
@@ -0,0 +1,62 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.alter;
+
+public enum AlterOpType {
+ // rollup
+ ADD_ROLLUP,
+ DROP_ROLLUP,
+ // schema change
+ SCHEMA_CHANGE,
+ // partition
+ ADD_PARTITION,
+ DROP_PARTITION,
+ REPLACE_PARTITION,
+ MODIFY_PARTITION,
+ // rename
+ RENAME,
+ // table property
+ MODIFY_TABLE_PROPERTY,
+ MODIFY_TABLE_PROPERTY_SYNC, // Some operations are performed synchronously, so we distinguish them by suffix _SYNC
+ // others operation, such as add/drop backend. currently we do not care about them
+ ALTER_OTHER,
+
+ INVALID_OP;
+
+ // true means 2 operations have no conflict.
+ public static Boolean[][] CONFLICT_MATRIX;
Review comment:
OK
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request
#3201: [Bug][Refactor] Fix the conflict of temp partition and dynamic
partition operations
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on a change in pull request #3201: [Bug][Refactor] Fix the conflict of temp partition and dynamic partition operations
URL: https://github.com/apache/incubator-doris/pull/3201#discussion_r398572556
##########
File path: fe/src/main/java/org/apache/doris/analysis/AddPartitionClause.java
##########
@@ -46,12 +47,11 @@ public AddPartitionClause(SingleRangePartitionDesc partitionDesc,
DistributionDesc distributionDesc,
Map<String, String> properties,
boolean isTempPartition) {
+ super(AlterOpType.ADD_PARTITION);
this.partitionDesc = partitionDesc;
this.distributionDesc = distributionDesc;
this.properties = properties;
this.isTempPartition = isTempPartition;
-
- this.needTableStable = false;
Review comment:
Why remove this code？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request
#3201: [Bug][Refactor] Fix the conflict of temp partition and dynamic
partition operations
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on a change in pull request #3201: [Bug][Refactor] Fix the conflict of temp partition and dynamic partition operations
URL: https://github.com/apache/incubator-doris/pull/3201#discussion_r398568426
##########
File path: fe/src/main/java/org/apache/doris/alter/AlterOperations.java
##########
@@ -0,0 +1,104 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.alter;
+
+import org.apache.doris.analysis.AlterClause;
+import org.apache.doris.common.DdlException;
+
+import com.google.common.base.Joiner;
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Set;
+
+/*
+ * AlterOperations contains a set alter operations generated from a AlterStmt's alter clause.
+ * This class is mainly used to integrate these operation types and check whether they have conflicts.
+ */
+public class AlterOperations {
+ private Set<AlterOpType> currentOps = Sets.newHashSet();
+
+ public AlterOperations() {
+ }
+
+ public Set<AlterOpType> getCurrentOps() {
+ return currentOps;
+ }
+
+ // check the conflicts of the given list of alter clauses
+ public void checkConflict(List<AlterClause> alterClauses) throws DdlException {
+ for (AlterClause alterClause : alterClauses) {
+ checkOp(alterClause.getOpType());
+ }
+ }
+
+ // some operations take up disk space. so we need to check the disk capacity before processing.
+ // return true if we see these kind of opertions.
+ public boolean needCheckCapacity() {
+ for (AlterOpType currentOp : currentOps) {
+ if (currentOp.needCheckCapacity()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ public boolean hasPartitionOp() {
+ return currentOps.contains(AlterOpType.ADD_PARTITION) || currentOps.contains(AlterOpType.DROP_PARTITION)
+ || currentOps.contains(AlterOpType.REPLACE_PARTITION) || currentOps.contains(AlterOpType.MODIFY_PARTITION);
+ }
+
+ // MODIFY_TABLE_PROPERTY is also processed by SchemaChangeHandler
+ public boolean hasSchemaChangeOp() {
+ return currentOps.contains(AlterOpType.SCHEMA_CHANGE) || currentOps.contains(AlterOpType.MODIFY_TABLE_PROPERTY);
+ }
+
+ public boolean hasRollupOp() {
+ return currentOps.contains(AlterOpType.ADD_ROLLUP) || currentOps.contains(AlterOpType.DROP_ROLLUP);
+ }
+
+ public boolean hasRenameOp() {
+ return currentOps.contains(AlterOpType.RENAME);
+ }
+
+ // return true if it has specified operation
Review comment:
This comment could remove.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request
#3201: [Bug][Refactor] Fix the conflict of temp partition and dynamic
partition operations
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #3201: [Bug][Refactor] Fix the conflict of temp partition and dynamic partition operations
URL: https://github.com/apache/incubator-doris/pull/3201#discussion_r398630183
##########
File path: fe/src/main/java/org/apache/doris/analysis/AddPartitionClause.java
##########
@@ -46,12 +47,11 @@ public AddPartitionClause(SingleRangePartitionDesc partitionDesc,
DistributionDesc distributionDesc,
Map<String, String> properties,
boolean isTempPartition) {
+ super(AlterOpType.ADD_PARTITION);
this.partitionDesc = partitionDesc;
this.distributionDesc = distributionDesc;
this.properties = properties;
this.isTempPartition = isTempPartition;
-
- this.needTableStable = false;
Review comment:
Add it back
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request
#3201: [Bug][Refactor] Fix the conflict of temp partition and dynamic
partition operations
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #3201: [Bug][Refactor] Fix the conflict of temp partition and dynamic partition operations
URL: https://github.com/apache/incubator-doris/pull/3201#discussion_r398629773
##########
File path: fe/src/main/java/org/apache/doris/alter/AlterOperations.java
##########
@@ -0,0 +1,104 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.alter;
+
+import org.apache.doris.analysis.AlterClause;
+import org.apache.doris.common.DdlException;
+
+import com.google.common.base.Joiner;
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Set;
+
+/*
+ * AlterOperations contains a set alter operations generated from a AlterStmt's alter clause.
+ * This class is mainly used to integrate these operation types and check whether they have conflicts.
+ */
+public class AlterOperations {
+ private Set<AlterOpType> currentOps = Sets.newHashSet();
+
+ public AlterOperations() {
+ }
+
+ public Set<AlterOpType> getCurrentOps() {
+ return currentOps;
+ }
+
+ // check the conflicts of the given list of alter clauses
+ public void checkConflict(List<AlterClause> alterClauses) throws DdlException {
+ for (AlterClause alterClause : alterClauses) {
+ checkOp(alterClause.getOpType());
+ }
+ }
+
+ // some operations take up disk space. so we need to check the disk capacity before processing.
+ // return true if we see these kind of opertions.
+ public boolean needCheckCapacity() {
+ for (AlterOpType currentOp : currentOps) {
+ if (currentOp.needCheckCapacity()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ public boolean hasPartitionOp() {
+ return currentOps.contains(AlterOpType.ADD_PARTITION) || currentOps.contains(AlterOpType.DROP_PARTITION)
+ || currentOps.contains(AlterOpType.REPLACE_PARTITION) || currentOps.contains(AlterOpType.MODIFY_PARTITION);
+ }
+
+ // MODIFY_TABLE_PROPERTY is also processed by SchemaChangeHandler
+ public boolean hasSchemaChangeOp() {
+ return currentOps.contains(AlterOpType.SCHEMA_CHANGE) || currentOps.contains(AlterOpType.MODIFY_TABLE_PROPERTY);
+ }
+
+ public boolean hasRollupOp() {
+ return currentOps.contains(AlterOpType.ADD_ROLLUP) || currentOps.contains(AlterOpType.DROP_ROLLUP);
+ }
+
+ public boolean hasRenameOp() {
+ return currentOps.contains(AlterOpType.RENAME);
+ }
+
+ // return true if it has specified operation
Review comment:
OK
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org