You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "morrySnow (via GitHub)" <gi...@apache.org> on 2024/06/04 04:27:26 UTC
Re: [PR] [feature](statistics)Remove not exist partition stats while analyzing. [doris]
morrySnow commented on code in PR #35814:
URL: https://github.com/apache/doris/pull/35814#discussion_r1625325245
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/BaseAnalysisTask.java:
##########
@@ -346,22 +348,29 @@ public void setJob(AnalysisJob job) {
}
/**
- * 1. Get stats of each partition
- * 2. insert partition in batch
- * 3. calculate column stats based on partition stats
+ * 1. Remove not exist partition stats
+ * 2. Get stats of each partition
+ * 3. insert partition in batch
+ * 4. calculate column stats based on partition stats
*/
protected void doPartitionTable() throws Exception {
+ deleteNotExistPartitionStats();
Map<String, String> params = buildSqlParams();
params.put("dataSizeFunction", getDataSizeFunction(col, false));
Set<String> partitionNames = tbl.getPartitionNames();
List<String> sqls = Lists.newArrayList();
int count = 0;
TableStatsMeta tableStatsStatus = Env.getServingEnv().getAnalysisManager().findTableStatsStatus(tbl.getId());
for (String part : partitionNames) {
+ params.put("partId", "-1");
Partition partition = tbl.getPartition(part);
+ if (partition != null) {
+ params.put("partId", String.valueOf(partition.getId()));
+ }
Review Comment:
```java
params.put("partId", partition == null ? "-1" : String.valueOf(partition.getId()));
```
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/BaseAnalysisTask.java:
##########
@@ -346,22 +348,29 @@ public void setJob(AnalysisJob job) {
}
/**
- * 1. Get stats of each partition
- * 2. insert partition in batch
- * 3. calculate column stats based on partition stats
+ * 1. Remove not exist partition stats
+ * 2. Get stats of each partition
+ * 3. insert partition in batch
+ * 4. calculate column stats based on partition stats
*/
protected void doPartitionTable() throws Exception {
+ deleteNotExistPartitionStats();
Map<String, String> params = buildSqlParams();
params.put("dataSizeFunction", getDataSizeFunction(col, false));
Set<String> partitionNames = tbl.getPartitionNames();
List<String> sqls = Lists.newArrayList();
int count = 0;
TableStatsMeta tableStatsStatus = Env.getServingEnv().getAnalysisManager().findTableStatsStatus(tbl.getId());
for (String part : partitionNames) {
+ params.put("partId", "-1");
Partition partition = tbl.getPartition(part);
+ if (partition != null) {
+ params.put("partId", String.valueOf(partition.getId()));
+ }
// Skip partitions that not changed after last analyze.
// External table getPartition always return null. So external table doesn't skip any partitions.
if (partition != null && tableStatsStatus != null && tableStatsStatus.partitionUpdateRows != null) {
+ params.put("partId", String.valueOf(partition.getId()));
Review Comment:
why add again?
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/BaseAnalysisTask.java:
##########
@@ -400,6 +409,10 @@ protected void doPartitionTable() throws Exception {
runQuery(stringSubstitutor.replace(MERGE_PARTITION_TEMPLATE));
}
+ protected void deleteNotExistPartitionStats() throws DdlException {
Review Comment:
abstract is better?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org