You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/03/11 09:00:58 UTC
[GitHub] [incubator-doris] WingsGo opened a new pull request #3081: [Dynamic
Partition] Support for automatically drop partitions
WingsGo opened a new pull request #3081: [Dynamic Partition] Support for automatically drop partitions
URL: https://github.com/apache/incubator-doris/pull/3081
For #3080
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request
#3081: [Dynamic Partition] Support for automatically drop partitions
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on a change in pull request #3081: [Dynamic Partition] Support for automatically drop partitions
URL: https://github.com/apache/incubator-doris/pull/3081#discussion_r392108873
##########
File path: fe/src/main/java/org/apache/doris/common/Config.java
##########
@@ -981,7 +981,7 @@
/*
* Decide how often to check dynamic partition
*/
- @ConfField(mutable = true, masterOnly = true)
+ @ConfField(mutable = false, masterOnly = true)
Review comment:
```suggestion
@ConfField
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] WingsGo commented on a change in pull request
#3081: [Dynamic Partition] Support for automatically drop partitions
Posted by GitBox <gi...@apache.org>.
WingsGo commented on a change in pull request #3081: [Dynamic Partition] Support for automatically drop partitions
URL: https://github.com/apache/incubator-doris/pull/3081#discussion_r396207273
##########
File path: fe/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
##########
@@ -168,98 +281,62 @@ private void dynamicAddPartition() {
try {
partitionFormat = DynamicPartitionUtil.getPartitionFormat(partitionColumn);
} catch (DdlException e) {
- recordFailedMsg(olapTable.getName(), e.getMessage());
+ recordCreatePartitionFailedMsg(olapTable.getName(), e.getMessage());
continue;
}
- Calendar calendar = Calendar.getInstance();
- TableProperty tableProperty = olapTable.getTableProperty();
- DynamicPartitionProperty dynamicPartitionProperty = tableProperty.getDynamicPartitionProperty();
-
- for (int i = 0; i <= dynamicPartitionProperty.getEnd(); i++) {
- String dynamicPartitionPrefix = dynamicPartitionProperty.getPrefix();
- String prevBorder = DynamicPartitionUtil.getPartitionRange(dynamicPartitionProperty.getTimeUnit(),
- i, (Calendar) calendar.clone(), partitionFormat);
- String partitionName = dynamicPartitionPrefix + DynamicPartitionUtil.getFormattedPartitionName(prevBorder);
-
- // continue if partition already exists
- String nextBorder = DynamicPartitionUtil.getPartitionRange(dynamicPartitionProperty.getTimeUnit(),
- i + 1, (Calendar) calendar.clone(), partitionFormat);
- PartitionValue lowerValue = new PartitionValue(prevBorder);
- PartitionValue upperValue = new PartitionValue(nextBorder);
- PartitionInfo partitionInfo = olapTable.getPartitionInfo();
- RangePartitionInfo info = (RangePartitionInfo) (partitionInfo);
- boolean isPartitionExists = false;
- Range<PartitionKey> addPartitionKeyRange = null;
- try {
- PartitionKey lowerBound = PartitionKey.createPartitionKey(Collections.singletonList(lowerValue), Collections.singletonList(partitionColumn));
- PartitionKey upperBound = PartitionKey.createPartitionKey(Collections.singletonList(upperValue), Collections.singletonList(partitionColumn));
- addPartitionKeyRange = Range.closedOpen(lowerBound, upperBound);
- } catch (AnalysisException e) {
- // keys.size is always equal to column.size, cannot reach this exception
- LOG.error("Keys size is not equl to column size.");
- continue;
- }
- for (Range<PartitionKey> partitionKeyRange : info.getIdToRange().values()) {
- // only support single column partition now
- try {
- RangeUtils.checkRangeIntersect(partitionKeyRange, addPartitionKeyRange);
- } catch (DdlException e) {
- isPartitionExists = true;
- if (addPartitionKeyRange.equals(partitionKeyRange)) {
- clearFailedMsg(olapTable.getName());
- } else {
- recordFailedMsg(olapTable.getName(), e.getMessage());
- }
- break;
- }
- }
- if (isPartitionExists) {
- continue;
- }
-
- // construct partition desc
- PartitionKeyDesc partitionKeyDesc = new PartitionKeyDesc(Collections.singletonList(lowerValue), Collections.singletonList(upperValue));
- HashMap<String, String> partitionProperties = new HashMap<>(1);
- partitionProperties.put("replication_num", String.valueOf(DynamicPartitionUtil.estimateReplicateNum(olapTable)));
- SingleRangePartitionDesc rangePartitionDesc = new SingleRangePartitionDesc(true, partitionName,
- partitionKeyDesc, partitionProperties);
-
- // construct distribution desc
- HashDistributionInfo hashDistributionInfo = (HashDistributionInfo) olapTable.getDefaultDistributionInfo();
- List<String> distColumnNames = new ArrayList<>();
- for (Column distributionColumn : hashDistributionInfo.getDistributionColumns()) {
- distColumnNames.add(distributionColumn.getName());
- }
- DistributionDesc distributionDesc = new HashDistributionDesc(dynamicPartitionProperty.getBuckets(), distColumnNames);
-
- // add partition according to partition desc and distribution desc
- addPartitionClauses.add(new AddPartitionClause(rangePartitionDesc, distributionDesc, null, false));
- }
+ addPartitionClauses = getAddPartitionClause(olapTable, partitionColumn, partitionFormat);
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request
#3081: [Dynamic Partition] Support for automatically drop partitions
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on a change in pull request #3081: [Dynamic Partition] Support for automatically drop partitions
URL: https://github.com/apache/incubator-doris/pull/3081#discussion_r392107716
##########
File path: fe/src/test/java/org/apache/doris/catalog/DynamicPartitionTableTest.java
##########
@@ -293,6 +294,50 @@ public void testMissTimeUnit(@Injectable SystemInfoService systemInfoService,
catalog.createTable(stmt);
}
+ @Test
+ public void testMissSTART(@Injectable SystemInfoService systemInfoService,
Review comment:
You could refer to `QueryPlanTest`, and create real table.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #3081: [Dynamic
Partition] Support for automatically drop partitions
Posted by GitBox <gi...@apache.org>.
morningman merged pull request #3081: [Dynamic Partition] Support for automatically drop partitions
URL: https://github.com/apache/incubator-doris/pull/3081
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request
#3081: [Dynamic Partition] Support for automatically drop partitions
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #3081: [Dynamic Partition] Support for automatically drop partitions
URL: https://github.com/apache/incubator-doris/pull/3081#discussion_r396185829
##########
File path: fe/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
##########
@@ -121,11 +123,118 @@ public void createOrUpdateRuntimeInfo(String tableName, String key, String value
defaultRuntimeInfo.put(LAST_UPDATE_TIME, DEFAULT_RUNTIME_VALUE);
defaultRuntimeInfo.put(LAST_SCHEDULER_TIME, DEFAULT_RUNTIME_VALUE);
defaultRuntimeInfo.put(DYNAMIC_PARTITION_STATE, State.NORMAL.toString());
- defaultRuntimeInfo.put(MSG, DEFAULT_RUNTIME_VALUE);
+ defaultRuntimeInfo.put(CREATE_PARTITION_MSG, DEFAULT_RUNTIME_VALUE);
+ defaultRuntimeInfo.put(DROP_PARTITION_MSG, DEFAULT_RUNTIME_VALUE);
return defaultRuntimeInfo;
}
- private void dynamicAddPartition() {
+ private ArrayList<AddPartitionClause> getAddPartitionClause(OlapTable olapTable, Column partitionColumn, String partitionFormat) {
+ ArrayList<AddPartitionClause> addPartitionClauses = new ArrayList<>();
+ Calendar calendar = Calendar.getInstance();
+ DynamicPartitionProperty dynamicPartitionProperty = olapTable.getTableProperty().getDynamicPartitionProperty();
+ for (int i = 0; i <= dynamicPartitionProperty.getEnd(); i++) {
+ String prevBorder = DynamicPartitionUtil.getPartitionRange(dynamicPartitionProperty.getTimeUnit(),
+ i, (Calendar) calendar.clone(), partitionFormat);
+ // continue if partition already exists
+ String nextBorder = DynamicPartitionUtil.getPartitionRange(dynamicPartitionProperty.getTimeUnit(),
+ i + 1, (Calendar) calendar.clone(), partitionFormat);
+ PartitionValue lowerValue = new PartitionValue(prevBorder);
+ PartitionValue upperValue = new PartitionValue(nextBorder);
+ PartitionInfo partitionInfo = olapTable.getPartitionInfo();
+ RangePartitionInfo info = (RangePartitionInfo) (partitionInfo);
+ boolean isPartitionExists = false;
+ Range<PartitionKey> addPartitionKeyRange;
+ try {
+ PartitionKey lowerBound = PartitionKey.createPartitionKey(Collections.singletonList(lowerValue), Collections.singletonList(partitionColumn));
+ PartitionKey upperBound = PartitionKey.createPartitionKey(Collections.singletonList(upperValue), Collections.singletonList(partitionColumn));
+ addPartitionKeyRange = Range.closedOpen(lowerBound, upperBound);
+ } catch (AnalysisException e) {
+ // keys.size is always equal to column.size, cannot reach this exception
+ LOG.error("Keys size is not equal to column size.");
Review comment:
Do not use `error` level, error means fatal.
And it better to add details in log, such as e.getMessage().
Same suggest to the `getDropPartitionClause()`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] WingsGo commented on a change in pull request
#3081: [Dynamic Partition] Support for automatically drop partitions
Posted by GitBox <gi...@apache.org>.
WingsGo commented on a change in pull request #3081: [Dynamic Partition] Support for automatically drop partitions
URL: https://github.com/apache/incubator-doris/pull/3081#discussion_r392303997
##########
File path: fe/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
##########
@@ -136,12 +245,15 @@ private void dynamicAddPartition() {
iterator.remove();
continue;
}
- String tableName;
- ArrayList<AddPartitionClause> addPartitionClauses = new ArrayList<>();
+
db.readLock();
+ ArrayList<AddPartitionClause> addPartitionClauses;
+ ArrayList<DropPartitionClause> dropPartitionClauses;
+ OlapTable olapTable = (OlapTable) db.getTable(tableId);
+ String tableName;
+ boolean skipAddPartition = false;
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request
#3081: [Dynamic Partition] Support for automatically drop partitions
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #3081: [Dynamic Partition] Support for automatically drop partitions
URL: https://github.com/apache/incubator-doris/pull/3081#discussion_r392275596
##########
File path: fe/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
##########
@@ -136,12 +245,15 @@ private void dynamicAddPartition() {
iterator.remove();
continue;
}
- String tableName;
- ArrayList<AddPartitionClause> addPartitionClauses = new ArrayList<>();
+
db.readLock();
+ ArrayList<AddPartitionClause> addPartitionClauses;
+ ArrayList<DropPartitionClause> dropPartitionClauses;
+ OlapTable olapTable = (OlapTable) db.getTable(tableId);
+ String tableName;
+ boolean skipAddPartition = false;
Review comment:
the pattern should be:
```
db.readLock();
try {
OlapTable olapTable = (OlapTable) db.getTable(tableId);
.....
} finally {
db.readUnlock();
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] WingsGo commented on a change in pull request
#3081: [Dynamic Partition] Support for automatically drop partitions
Posted by GitBox <gi...@apache.org>.
WingsGo commented on a change in pull request #3081: [Dynamic Partition] Support for automatically drop partitions
URL: https://github.com/apache/incubator-doris/pull/3081#discussion_r396207348
##########
File path: fe/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
##########
@@ -121,11 +123,118 @@ public void createOrUpdateRuntimeInfo(String tableName, String key, String value
defaultRuntimeInfo.put(LAST_UPDATE_TIME, DEFAULT_RUNTIME_VALUE);
defaultRuntimeInfo.put(LAST_SCHEDULER_TIME, DEFAULT_RUNTIME_VALUE);
defaultRuntimeInfo.put(DYNAMIC_PARTITION_STATE, State.NORMAL.toString());
- defaultRuntimeInfo.put(MSG, DEFAULT_RUNTIME_VALUE);
+ defaultRuntimeInfo.put(CREATE_PARTITION_MSG, DEFAULT_RUNTIME_VALUE);
+ defaultRuntimeInfo.put(DROP_PARTITION_MSG, DEFAULT_RUNTIME_VALUE);
return defaultRuntimeInfo;
}
- private void dynamicAddPartition() {
+ private ArrayList<AddPartitionClause> getAddPartitionClause(OlapTable olapTable, Column partitionColumn, String partitionFormat) {
+ ArrayList<AddPartitionClause> addPartitionClauses = new ArrayList<>();
+ Calendar calendar = Calendar.getInstance();
+ DynamicPartitionProperty dynamicPartitionProperty = olapTable.getTableProperty().getDynamicPartitionProperty();
+ for (int i = 0; i <= dynamicPartitionProperty.getEnd(); i++) {
+ String prevBorder = DynamicPartitionUtil.getPartitionRange(dynamicPartitionProperty.getTimeUnit(),
+ i, (Calendar) calendar.clone(), partitionFormat);
+ // continue if partition already exists
+ String nextBorder = DynamicPartitionUtil.getPartitionRange(dynamicPartitionProperty.getTimeUnit(),
+ i + 1, (Calendar) calendar.clone(), partitionFormat);
+ PartitionValue lowerValue = new PartitionValue(prevBorder);
+ PartitionValue upperValue = new PartitionValue(nextBorder);
+ PartitionInfo partitionInfo = olapTable.getPartitionInfo();
+ RangePartitionInfo info = (RangePartitionInfo) (partitionInfo);
+ boolean isPartitionExists = false;
+ Range<PartitionKey> addPartitionKeyRange;
+ try {
+ PartitionKey lowerBound = PartitionKey.createPartitionKey(Collections.singletonList(lowerValue), Collections.singletonList(partitionColumn));
+ PartitionKey upperBound = PartitionKey.createPartitionKey(Collections.singletonList(upperValue), Collections.singletonList(partitionColumn));
+ addPartitionKeyRange = Range.closedOpen(lowerBound, upperBound);
+ } catch (AnalysisException e) {
+ // keys.size is always equal to column.size, cannot reach this exception
+ LOG.error("Keys size is not equal to column size.");
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request
#3081: [Dynamic Partition] Support for automatically drop partitions
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #3081: [Dynamic Partition] Support for automatically drop partitions
URL: https://github.com/apache/incubator-doris/pull/3081#discussion_r396480004
##########
File path: fe/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
##########
@@ -150,7 +150,7 @@ public void createOrUpdateRuntimeInfo(String tableName, String key, String value
addPartitionKeyRange = Range.closedOpen(lowerBound, upperBound);
} catch (AnalysisException e) {
// keys.size is always equal to column.size, cannot reach this exception
- LOG.error("Keys size is not equal to column size.");
+ LOG.warn("Keys size is not equal to column size. Error=" + e.getMessage());
Review comment:
```suggestion
LOG.warn("Keys size is not equal to column size. Error={}", e.getMessage());
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request
#3081: [Dynamic Partition] Support for automatically drop partitions
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on a change in pull request #3081: [Dynamic Partition] Support for automatically drop partitions
URL: https://github.com/apache/incubator-doris/pull/3081#discussion_r392107853
##########
File path: fe/src/main/java/org/apache/doris/service/Untitled-2.md
##########
@@ -0,0 +1 @@
+# 1 Master + 2 Follower
Review comment:
Why add this file?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request
#3081: [Dynamic Partition] Support for automatically drop partitions
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on a change in pull request #3081: [Dynamic Partition] Support for automatically drop partitions
URL: https://github.com/apache/incubator-doris/pull/3081#discussion_r392106592
##########
File path: fe/src/test/java/org/apache/doris/persist/ModifyDynamicPartitionInfoTest.java
##########
@@ -49,6 +49,7 @@ public void testNormal() throws IOException {
HashMap<String, String> properties = new HashMap<>();
properties.put(DynamicPartitionProperty.ENABLE, "true");
properties.put(DynamicPartitionProperty.TIME_UNIT, "day");
+ properties.put(DynamicPartitionProperty.START, "3");
Review comment:
-3?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] WingsGo commented on a change in pull request
#3081: [Dynamic Partition] Support for automatically drop partitions
Posted by GitBox <gi...@apache.org>.
WingsGo commented on a change in pull request #3081: [Dynamic Partition] Support for automatically drop partitions
URL: https://github.com/apache/incubator-doris/pull/3081#discussion_r392293473
##########
File path: fe/src/main/java/org/apache/doris/service/Untitled-2.md
##########
@@ -0,0 +1 @@
+# 1 Master + 2 Follower
Review comment:
my fault…I will remove it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] WingsGo commented on a change in pull request
#3081: [Dynamic Partition] Support for automatically drop partitions
Posted by GitBox <gi...@apache.org>.
WingsGo commented on a change in pull request #3081: [Dynamic Partition] Support for automatically drop partitions
URL: https://github.com/apache/incubator-doris/pull/3081#discussion_r392293805
##########
File path: fe/src/test/java/org/apache/doris/catalog/DynamicPartitionTableTest.java
##########
@@ -293,6 +294,50 @@ public void testMissTimeUnit(@Injectable SystemInfoService systemInfoService,
catalog.createTable(stmt);
}
+ @Test
+ public void testMissSTART(@Injectable SystemInfoService systemInfoService,
Review comment:
ok, i will change it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] WingsGo commented on a change in pull request
#3081: [Dynamic Partition] Support for automatically drop partitions
Posted by GitBox <gi...@apache.org>.
WingsGo commented on a change in pull request #3081: [Dynamic Partition] Support for automatically drop partitions
URL: https://github.com/apache/incubator-doris/pull/3081#discussion_r394770914
##########
File path: fe/src/test/java/org/apache/doris/catalog/DynamicPartitionTableTest.java
##########
@@ -293,6 +294,50 @@ public void testMissTimeUnit(@Injectable SystemInfoService systemInfoService,
catalog.createTable(stmt);
}
+ @Test
+ public void testMissSTART(@Injectable SystemInfoService systemInfoService,
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request
#3081: [Dynamic Partition] Support for automatically drop partitions
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #3081: [Dynamic Partition] Support for automatically drop partitions
URL: https://github.com/apache/incubator-doris/pull/3081#discussion_r396480911
##########
File path: fe/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
##########
@@ -211,7 +211,7 @@ public void createOrUpdateRuntimeInfo(String tableName, String key, String value
reservePartitionKeyRange = Range.closedOpen(lowerBound, upperBound);
} catch (AnalysisException e) {
// keys.size is always equal to column.size, cannot reach this exception
- LOG.error("Keys size is not equal to column size.");
+ LOG.warn("Keys size is not equal to column size. Error=" + e.getMessage());
Review comment:
```suggestion
LOG.warn("Keys size is not equal to column size. Error={}", e.getMessage());
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request
#3081: [Dynamic Partition] Support for automatically drop partitions
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #3081: [Dynamic Partition] Support for automatically drop partitions
URL: https://github.com/apache/incubator-doris/pull/3081#discussion_r396185388
##########
File path: fe/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
##########
@@ -168,98 +281,62 @@ private void dynamicAddPartition() {
try {
partitionFormat = DynamicPartitionUtil.getPartitionFormat(partitionColumn);
} catch (DdlException e) {
- recordFailedMsg(olapTable.getName(), e.getMessage());
+ recordCreatePartitionFailedMsg(olapTable.getName(), e.getMessage());
continue;
}
- Calendar calendar = Calendar.getInstance();
- TableProperty tableProperty = olapTable.getTableProperty();
- DynamicPartitionProperty dynamicPartitionProperty = tableProperty.getDynamicPartitionProperty();
-
- for (int i = 0; i <= dynamicPartitionProperty.getEnd(); i++) {
- String dynamicPartitionPrefix = dynamicPartitionProperty.getPrefix();
- String prevBorder = DynamicPartitionUtil.getPartitionRange(dynamicPartitionProperty.getTimeUnit(),
- i, (Calendar) calendar.clone(), partitionFormat);
- String partitionName = dynamicPartitionPrefix + DynamicPartitionUtil.getFormattedPartitionName(prevBorder);
-
- // continue if partition already exists
- String nextBorder = DynamicPartitionUtil.getPartitionRange(dynamicPartitionProperty.getTimeUnit(),
- i + 1, (Calendar) calendar.clone(), partitionFormat);
- PartitionValue lowerValue = new PartitionValue(prevBorder);
- PartitionValue upperValue = new PartitionValue(nextBorder);
- PartitionInfo partitionInfo = olapTable.getPartitionInfo();
- RangePartitionInfo info = (RangePartitionInfo) (partitionInfo);
- boolean isPartitionExists = false;
- Range<PartitionKey> addPartitionKeyRange = null;
- try {
- PartitionKey lowerBound = PartitionKey.createPartitionKey(Collections.singletonList(lowerValue), Collections.singletonList(partitionColumn));
- PartitionKey upperBound = PartitionKey.createPartitionKey(Collections.singletonList(upperValue), Collections.singletonList(partitionColumn));
- addPartitionKeyRange = Range.closedOpen(lowerBound, upperBound);
- } catch (AnalysisException e) {
- // keys.size is always equal to column.size, cannot reach this exception
- LOG.error("Keys size is not equl to column size.");
- continue;
- }
- for (Range<PartitionKey> partitionKeyRange : info.getIdToRange().values()) {
- // only support single column partition now
- try {
- RangeUtils.checkRangeIntersect(partitionKeyRange, addPartitionKeyRange);
- } catch (DdlException e) {
- isPartitionExists = true;
- if (addPartitionKeyRange.equals(partitionKeyRange)) {
- clearFailedMsg(olapTable.getName());
- } else {
- recordFailedMsg(olapTable.getName(), e.getMessage());
- }
- break;
- }
- }
- if (isPartitionExists) {
- continue;
- }
-
- // construct partition desc
- PartitionKeyDesc partitionKeyDesc = new PartitionKeyDesc(Collections.singletonList(lowerValue), Collections.singletonList(upperValue));
- HashMap<String, String> partitionProperties = new HashMap<>(1);
- partitionProperties.put("replication_num", String.valueOf(DynamicPartitionUtil.estimateReplicateNum(olapTable)));
- SingleRangePartitionDesc rangePartitionDesc = new SingleRangePartitionDesc(true, partitionName,
- partitionKeyDesc, partitionProperties);
-
- // construct distribution desc
- HashDistributionInfo hashDistributionInfo = (HashDistributionInfo) olapTable.getDefaultDistributionInfo();
- List<String> distColumnNames = new ArrayList<>();
- for (Column distributionColumn : hashDistributionInfo.getDistributionColumns()) {
- distColumnNames.add(distributionColumn.getName());
- }
- DistributionDesc distributionDesc = new HashDistributionDesc(dynamicPartitionProperty.getBuckets(), distColumnNames);
-
- // add partition according to partition desc and distribution desc
- addPartitionClauses.add(new AddPartitionClause(rangePartitionDesc, distributionDesc, null, false));
- }
+ addPartitionClauses = getAddPartitionClause(olapTable, partitionColumn, partitionFormat);
Review comment:
No need to get add partition clause is `skipAddPartition` is true.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request
#3081: [Dynamic Partition] Support for automatically drop partitions
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on a change in pull request #3081: [Dynamic Partition] Support for automatically drop partitions
URL: https://github.com/apache/incubator-doris/pull/3081#discussion_r392106710
##########
File path: fe/src/test/java/org/apache/doris/catalog/TablePropertyTest.java
##########
@@ -49,6 +49,7 @@ public void testNormal() throws IOException {
HashMap<String, String> properties = new HashMap<>();
properties.put(DynamicPartitionProperty.ENABLE, "true");
properties.put(DynamicPartitionProperty.TIME_UNIT, "day");
+ properties.put(DynamicPartitionProperty.START, "3");
Review comment:
-3
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] WingsGo commented on a change in pull request
#3081: [Dynamic Partition] Support for automatically drop partitions
Posted by GitBox <gi...@apache.org>.
WingsGo commented on a change in pull request #3081: [Dynamic Partition] Support for automatically drop partitions
URL: https://github.com/apache/incubator-doris/pull/3081#discussion_r394770795
##########
File path: fe/src/main/java/org/apache/doris/common/Config.java
##########
@@ -981,7 +981,7 @@
/*
* Decide how often to check dynamic partition
*/
- @ConfField(mutable = true, masterOnly = true)
+ @ConfField(mutable = false, masterOnly = true)
Review comment:
I think maybe we should allow dynamic change the check interval seconds?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org