You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/07/20 04:26:24 UTC
[GitHub] [doris] weizhengte opened a new pull request, #11030: [feature-wip](statistics) step4.1: manually inject statistics for a table or column
weizhengte opened a new pull request, #11030:
URL: https://github.com/apache/doris/pull/11030
# Proposed changes
This pr mainly to supplement the syntax of the previous pr(#8861), it supports users to manually inject statistics, including table, partition, and column statistics.
Modify table or partition statistics:
`
ALTER TABLE table_name
SET STATS ('k1' = 'v1', ...) [ PARTITIONS(p_name1, p_name2...) ]
`
Modify column statistics:
`
ALTER TABLE table_name MODIFY COLUMN columnName
SET STATS ('k1' = 'v1', ...) [ PARTITIONS(p_name1, p_name2...) ]
`
Some notes:
- Only support statistics injected into olap type tables.
- Statistics injected into temporary partitions are not supported.
- When injecting statistics, if it is a partitioned table, users need to specify a partition name.
- If multiple partitions are specified, the same stats will be injected on multiple partitions.
- The current code also has mock statistics @zhengshij
Issue Number: close #xxx
## Problem Summary:
Describe the overview of changes.
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know)
2. Has unit tests been added: (Yes/No/No Need)
3. Has document been added or modified: (Yes/No/No Need)
4. Does it need to update dependencies: (Yes/No)
5. Are there any changes that cannot be rolled back: (Yes/No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] weizhengte commented on a diff in pull request #11030: [feature-wip](statistics) step4.1: manually inject statistics for a table or column
Posted by GitBox <gi...@apache.org>.
weizhengte commented on code in PR #11030:
URL: https://github.com/apache/doris/pull/11030#discussion_r938109803
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/AlterColumnStatsStmt.java:
##########
@@ -47,59 +60,124 @@ public class AlterColumnStatsStmt extends DdlStmt {
.add(ColumnStats.MAX_VALUE)
.build();
- private TableName tableName;
- private String columnName;
- private Map<String, String> properties;
- public final Map<StatsType, String> statsTypeToValue = Maps.newHashMap();
+ private final TableName tableName;
+ private final PartitionNames optPartitionNames;
+ private final String columnName;
+ private final Map<String, String> properties;
- public AlterColumnStatsStmt(TableName tableName, String columnName, Map<String, String> properties) {
+ private final List<String> partitionNames = Lists.newArrayList();
+ private final Map<StatsType, String> statsTypeToValue = Maps.newHashMap();
+
+ public AlterColumnStatsStmt(TableName tableName, String columnName,
+ Map<String, String> properties, PartitionNames optPartitionNames) {
this.tableName = tableName;
this.columnName = columnName;
- this.properties = properties;
+ this.properties = properties == null ? Maps.newHashMap() : properties;
+ this.optPartitionNames = optPartitionNames;
+ }
+
+ public TableName getTableName() {
+ return tableName;
+ }
+
+ public String getColumnName() {
+ return columnName;
+ }
+
+ public List<String> getPartitionNames() {
+ return partitionNames;
+ }
+
+ public Map<StatsType, String> getStatsTypeToValue() {
+ return statsTypeToValue;
}
@Override
public void analyze(Analyzer analyzer) throws UserException {
super.analyze(analyzer);
+
// check table name
tableName.analyze(analyzer);
+
// disallow external catalog
Util.prohibitExternalCatalog(tableName.getCtl(), this.getClass().getSimpleName());
+
+ // check partition & column
+ checkPartitionAndColumnNames();
+
// check properties
Optional<StatsType> optional = properties.keySet().stream().map(StatsType::fromString)
- .filter(statsType -> !CONFIGURABLE_PROPERTIES_SET.contains(statsType)).findFirst();
+ .filter(statsType -> !CONFIGURABLE_PROPERTIES_SET.contains(statsType))
+ .findFirst();
if (optional.isPresent()) {
- throw new AnalysisException(optional.get() + " is invalid statistic");
+ throw new AnalysisException(optional.get() + " is invalid statistics");
}
+
// check auth
- if (!Env.getCurrentEnv().getAuth().checkTblPriv(
- ConnectContext.get(), tableName.getDb(), tableName.getTbl(), PrivPredicate.ALTER)) {
+ if (!Env.getCurrentEnv().getAuth()
+ .checkTblPriv(ConnectContext.get(), tableName.getDb(), tableName.getTbl(), PrivPredicate.ALTER)) {
ErrorReport.reportAnalysisException(ErrorCode.ERR_TABLEACCESS_DENIED_ERROR, "ALTER COLUMN STATS",
- ConnectContext.get().getQualifiedUser(),
- ConnectContext.get().getRemoteIP(),
+ ConnectContext.get().getQualifiedUser(), ConnectContext.get().getRemoteIP(),
tableName.getDb() + ": " + tableName.getTbl());
}
+
// get statsTypeToValue
properties.forEach((key, value) -> {
StatsType statsType = StatsType.fromString(key);
statsTypeToValue.put(statsType, value);
});
}
- public TableName getTableName() {
- return tableName;
- }
+ private void checkPartitionAndColumnNames() throws AnalysisException {
+ Database db = analyzer.getEnv().getInternalDataSource()
+ .getDbOrAnalysisException(tableName.getDb());
+ Table table = db.getTableOrAnalysisException(tableName.getTbl());
- public String getColumnName() {
- return columnName;
- }
+ if (table.getType() != Table.TableType.OLAP) {
+ throw new AnalysisException("Only OLAP table statistics are supported");
+ }
- public Map<StatsType, String> getStatsTypeToValue() {
- return statsTypeToValue;
+ OlapTable olapTable = (OlapTable) table;
+
+ if (olapTable.getColumn(columnName) == null) {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_WRONG_COLUMN_NAME, columnName);
+ }
+
+ if (optPartitionNames != null) {
+ optPartitionNames.analyze(analyzer);
+ if (!olapTable.isPartitioned()) {
+ throw new AnalysisException("Not a partitioned table: " + olapTable.getName());
+ }
+ Set<String> olapPartitionNames = olapTable.getPartitionNames();
+ Optional<String> optional = optPartitionNames.getPartitionNames().stream()
+ .filter(name -> !olapPartitionNames.contains(name)).findFirst();
Review Comment:
indeed anyMatch looks better, but I use findFirst here mainly to tell the user the name of the partition that doesn't exist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] weizhengte commented on a diff in pull request #11030: [feature-wip](statistics) step4.1: manually inject statistics for a table or column
Posted by GitBox <gi...@apache.org>.
weizhengte commented on code in PR #11030:
URL: https://github.com/apache/doris/pull/11030#discussion_r938407214
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/AlterTableStatsStmt.java:
##########
@@ -35,61 +39,125 @@
import java.util.List;
import java.util.Map;
import java.util.Optional;
+import java.util.Set;
+/**
+ * Manually inject statistics for tables or partitions.
+ * Only OLAP table statistics are supported.
+ * e.g.
+ * ALTER TABLE table_name
+ * SET STATS ('k1' = 'v1', ...) [ PARTITIONS(p_name1, p_name2...) ]
+ */
public class AlterTableStatsStmt extends DdlStmt {
private static final ImmutableSet<StatsType> CONFIGURABLE_PROPERTIES_SET = new ImmutableSet.Builder<StatsType>()
.add(TableStats.DATA_SIZE)
.add(TableStats.ROW_COUNT)
.build();
- private TableName tableName;
- private Map<String, String> properties;
- public final Map<StatsType, String> statsTypeToValue = Maps.newHashMap();
+ private final TableName tableName;
+ private final PartitionNames optPartitionNames;
+ private final Map<String, String> properties;
- public AlterTableStatsStmt(TableName tableName, Map<String, String> properties) {
+ private final List<String> partitionNames = Lists.newArrayList();
+ private final Map<StatsType, String> statsTypeToValue = Maps.newHashMap();
+
+ public AlterTableStatsStmt(TableName tableName, Map<String, String> properties,
+ PartitionNames optPartitionNames) {
this.tableName = tableName;
- this.properties = properties;
+ this.properties = properties == null ? Maps.newHashMap() : properties;
+ this.optPartitionNames = optPartitionNames;
+ }
+
+ public TableName getTableName() {
+ return tableName;
+ }
+
+ public List<String> getPartitionNames() {
+ return partitionNames;
+ }
+
+ public Map<StatsType, String> getStatsTypeToValue() {
+ return statsTypeToValue;
}
@Override
public void analyze(Analyzer analyzer) throws UserException {
super.analyze(analyzer);
+
// check table name
tableName.analyze(analyzer);
+
// disallow external catalog
Util.prohibitExternalCatalog(tableName.getCtl(), this.getClass().getSimpleName());
+
+ // check partition
+ checkPartitionNames();
+
// check properties
Optional<StatsType> optional = properties.keySet().stream().map(StatsType::fromString)
- .filter(statsType -> !CONFIGURABLE_PROPERTIES_SET.contains(statsType)).findFirst();
+ .filter(statsType -> !CONFIGURABLE_PROPERTIES_SET.contains(statsType))
+ .findFirst();
if (optional.isPresent()) {
- throw new AnalysisException(optional.get() + " is invalid statistic");
+ throw new AnalysisException(optional.get() + " is invalid statistics");
}
+
// check auth
- if (!Env.getCurrentEnv().getAuth().checkTblPriv(
- ConnectContext.get(), tableName.getDb(), tableName.getTbl(), PrivPredicate.ALTER)) {
+ if (!Env.getCurrentEnv().getAuth()
+ .checkTblPriv(ConnectContext.get(), tableName.getDb(), tableName.getTbl(), PrivPredicate.ALTER)) {
ErrorReport.reportAnalysisException(ErrorCode.ERR_TABLEACCESS_DENIED_ERROR, "ALTER TABLE STATS",
- ConnectContext.get().getQualifiedUser(),
- ConnectContext.get().getRemoteIP(),
+ ConnectContext.get().getQualifiedUser(), ConnectContext.get().getRemoteIP(),
tableName.getDb() + ": " + tableName.getTbl());
}
+
// get statsTypeToValue
properties.forEach((key, value) -> {
StatsType statsType = StatsType.fromString(key);
statsTypeToValue.put(statsType, value);
});
}
- public TableName getTableName() {
- return tableName;
- }
+ private void checkPartitionNames() throws AnalysisException {
+ if (optPartitionNames != null) {
+ optPartitionNames.analyze(analyzer);
+ Database db = analyzer.getEnv().getInternalDataSource()
+ .getDbOrAnalysisException(tableName.getDb());
+ Table table = db.getTableOrAnalysisException(tableName.getTbl());
- public Map<StatsType, String> getStatsTypeToValue() {
- return statsTypeToValue;
+ if (table.getType() != Table.TableType.OLAP) {
+ throw new AnalysisException("Only OLAP table statistics are supported");
+ }
+
+ OlapTable olapTable = (OlapTable) table;
+
+ if (olapTable.isPartitioned()) {
+ List<String> names = optPartitionNames.getPartitionNames();
+ Set<String> olapPartitionNames = olapTable.getPartitionNames();
+ Optional<String> optional = names.stream()
+ .filter(name -> !olapPartitionNames.contains(name)).findFirst();
+ if (optional.isPresent()) {
+ throw new AnalysisException("Partition does not exist: " + optional.get());
+ }
+ partitionNames.addAll(optPartitionNames.getPartitionNames());
+ } else {
+ throw new AnalysisException("Not a partitioned table: " + olapTable.getName());
+ }
+ }
Review Comment:
There are table level and partition level statistics. For a partitioned table, if a partition is specified by the user, the statistics of the corresponding partition will be updated, and if no partition is specified, the statistics of the table level will be updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] weizhengte commented on a diff in pull request #11030: [feature-wip](statistics) step4.1: manually inject statistics for a table or column
Posted by GitBox <gi...@apache.org>.
weizhengte commented on code in PR #11030:
URL: https://github.com/apache/doris/pull/11030#discussion_r938109310
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/AlterTableStatsStmt.java:
##########
@@ -35,61 +39,125 @@
import java.util.List;
import java.util.Map;
import java.util.Optional;
+import java.util.Set;
+/**
+ * Manually inject statistics for tables or partitions.
+ * Only OLAP table statistics are supported.
+ * e.g.
+ * ALTER TABLE table_name
+ * SET STATS ('k1' = 'v1', ...) [ PARTITIONS(p_name1, p_name2...) ]
+ */
public class AlterTableStatsStmt extends DdlStmt {
private static final ImmutableSet<StatsType> CONFIGURABLE_PROPERTIES_SET = new ImmutableSet.Builder<StatsType>()
.add(TableStats.DATA_SIZE)
.add(TableStats.ROW_COUNT)
.build();
- private TableName tableName;
- private Map<String, String> properties;
- public final Map<StatsType, String> statsTypeToValue = Maps.newHashMap();
+ private final TableName tableName;
+ private final PartitionNames optPartitionNames;
+ private final Map<String, String> properties;
- public AlterTableStatsStmt(TableName tableName, Map<String, String> properties) {
+ private final List<String> partitionNames = Lists.newArrayList();
+ private final Map<StatsType, String> statsTypeToValue = Maps.newHashMap();
+
+ public AlterTableStatsStmt(TableName tableName, Map<String, String> properties,
+ PartitionNames optPartitionNames) {
this.tableName = tableName;
- this.properties = properties;
+ this.properties = properties == null ? Maps.newHashMap() : properties;
+ this.optPartitionNames = optPartitionNames;
+ }
+
+ public TableName getTableName() {
+ return tableName;
+ }
+
+ public List<String> getPartitionNames() {
+ return partitionNames;
+ }
+
+ public Map<StatsType, String> getStatsTypeToValue() {
+ return statsTypeToValue;
}
@Override
public void analyze(Analyzer analyzer) throws UserException {
super.analyze(analyzer);
+
// check table name
tableName.analyze(analyzer);
+
// disallow external catalog
Util.prohibitExternalCatalog(tableName.getCtl(), this.getClass().getSimpleName());
+
+ // check partition
+ checkPartitionNames();
+
// check properties
Optional<StatsType> optional = properties.keySet().stream().map(StatsType::fromString)
- .filter(statsType -> !CONFIGURABLE_PROPERTIES_SET.contains(statsType)).findFirst();
+ .filter(statsType -> !CONFIGURABLE_PROPERTIES_SET.contains(statsType))
+ .findFirst();
if (optional.isPresent()) {
- throw new AnalysisException(optional.get() + " is invalid statistic");
+ throw new AnalysisException(optional.get() + " is invalid statistics");
}
+
// check auth
- if (!Env.getCurrentEnv().getAuth().checkTblPriv(
- ConnectContext.get(), tableName.getDb(), tableName.getTbl(), PrivPredicate.ALTER)) {
+ if (!Env.getCurrentEnv().getAuth()
+ .checkTblPriv(ConnectContext.get(), tableName.getDb(), tableName.getTbl(), PrivPredicate.ALTER)) {
ErrorReport.reportAnalysisException(ErrorCode.ERR_TABLEACCESS_DENIED_ERROR, "ALTER TABLE STATS",
- ConnectContext.get().getQualifiedUser(),
- ConnectContext.get().getRemoteIP(),
+ ConnectContext.get().getQualifiedUser(), ConnectContext.get().getRemoteIP(),
tableName.getDb() + ": " + tableName.getTbl());
}
+
// get statsTypeToValue
properties.forEach((key, value) -> {
StatsType statsType = StatsType.fromString(key);
statsTypeToValue.put(statsType, value);
});
}
- public TableName getTableName() {
- return tableName;
- }
+ private void checkPartitionNames() throws AnalysisException {
+ if (optPartitionNames != null) {
+ optPartitionNames.analyze(analyzer);
+ Database db = analyzer.getEnv().getInternalDataSource()
+ .getDbOrAnalysisException(tableName.getDb());
+ Table table = db.getTableOrAnalysisException(tableName.getTbl());
- public Map<StatsType, String> getStatsTypeToValue() {
- return statsTypeToValue;
+ if (table.getType() != Table.TableType.OLAP) {
+ throw new AnalysisException("Only OLAP table statistics are supported");
+ }
+
+ OlapTable olapTable = (OlapTable) table;
+
+ if (olapTable.isPartitioned()) {
+ List<String> names = optPartitionNames.getPartitionNames();
+ Set<String> olapPartitionNames = olapTable.getPartitionNames();
+ Optional<String> optional = names.stream()
+ .filter(name -> !olapPartitionNames.contains(name)).findFirst();
+ if (optional.isPresent()) {
+ throw new AnalysisException("Partition does not exist: " + optional.get());
+ }
+ partitionNames.addAll(optPartitionNames.getPartitionNames());
+ } else {
+ throw new AnalysisException("Not a partitioned table: " + olapTable.getName());
+ }
+ }
Review Comment:
sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman merged pull request #11030: [feature-wip](statistics) step4.1: manually inject statistics for a table or column
Posted by GitBox <gi...@apache.org>.
morningman merged PR #11030:
URL: https://github.com/apache/doris/pull/11030
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] weizhengte commented on a diff in pull request #11030: [feature-wip](statistics) step4.1: manually inject statistics for a table or column
Posted by GitBox <gi...@apache.org>.
weizhengte commented on code in PR #11030:
URL: https://github.com/apache/doris/pull/11030#discussion_r938109310
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/AlterTableStatsStmt.java:
##########
@@ -35,61 +39,125 @@
import java.util.List;
import java.util.Map;
import java.util.Optional;
+import java.util.Set;
+/**
+ * Manually inject statistics for tables or partitions.
+ * Only OLAP table statistics are supported.
+ * e.g.
+ * ALTER TABLE table_name
+ * SET STATS ('k1' = 'v1', ...) [ PARTITIONS(p_name1, p_name2...) ]
+ */
public class AlterTableStatsStmt extends DdlStmt {
private static final ImmutableSet<StatsType> CONFIGURABLE_PROPERTIES_SET = new ImmutableSet.Builder<StatsType>()
.add(TableStats.DATA_SIZE)
.add(TableStats.ROW_COUNT)
.build();
- private TableName tableName;
- private Map<String, String> properties;
- public final Map<StatsType, String> statsTypeToValue = Maps.newHashMap();
+ private final TableName tableName;
+ private final PartitionNames optPartitionNames;
+ private final Map<String, String> properties;
- public AlterTableStatsStmt(TableName tableName, Map<String, String> properties) {
+ private final List<String> partitionNames = Lists.newArrayList();
+ private final Map<StatsType, String> statsTypeToValue = Maps.newHashMap();
+
+ public AlterTableStatsStmt(TableName tableName, Map<String, String> properties,
+ PartitionNames optPartitionNames) {
this.tableName = tableName;
- this.properties = properties;
+ this.properties = properties == null ? Maps.newHashMap() : properties;
+ this.optPartitionNames = optPartitionNames;
+ }
+
+ public TableName getTableName() {
+ return tableName;
+ }
+
+ public List<String> getPartitionNames() {
+ return partitionNames;
+ }
+
+ public Map<StatsType, String> getStatsTypeToValue() {
+ return statsTypeToValue;
}
@Override
public void analyze(Analyzer analyzer) throws UserException {
super.analyze(analyzer);
+
// check table name
tableName.analyze(analyzer);
+
// disallow external catalog
Util.prohibitExternalCatalog(tableName.getCtl(), this.getClass().getSimpleName());
+
+ // check partition
+ checkPartitionNames();
+
// check properties
Optional<StatsType> optional = properties.keySet().stream().map(StatsType::fromString)
- .filter(statsType -> !CONFIGURABLE_PROPERTIES_SET.contains(statsType)).findFirst();
+ .filter(statsType -> !CONFIGURABLE_PROPERTIES_SET.contains(statsType))
+ .findFirst();
if (optional.isPresent()) {
- throw new AnalysisException(optional.get() + " is invalid statistic");
+ throw new AnalysisException(optional.get() + " is invalid statistics");
}
+
// check auth
- if (!Env.getCurrentEnv().getAuth().checkTblPriv(
- ConnectContext.get(), tableName.getDb(), tableName.getTbl(), PrivPredicate.ALTER)) {
+ if (!Env.getCurrentEnv().getAuth()
+ .checkTblPriv(ConnectContext.get(), tableName.getDb(), tableName.getTbl(), PrivPredicate.ALTER)) {
ErrorReport.reportAnalysisException(ErrorCode.ERR_TABLEACCESS_DENIED_ERROR, "ALTER TABLE STATS",
- ConnectContext.get().getQualifiedUser(),
- ConnectContext.get().getRemoteIP(),
+ ConnectContext.get().getQualifiedUser(), ConnectContext.get().getRemoteIP(),
tableName.getDb() + ": " + tableName.getTbl());
}
+
// get statsTypeToValue
properties.forEach((key, value) -> {
StatsType statsType = StatsType.fromString(key);
statsTypeToValue.put(statsType, value);
});
}
- public TableName getTableName() {
- return tableName;
- }
+ private void checkPartitionNames() throws AnalysisException {
+ if (optPartitionNames != null) {
+ optPartitionNames.analyze(analyzer);
+ Database db = analyzer.getEnv().getInternalDataSource()
+ .getDbOrAnalysisException(tableName.getDb());
+ Table table = db.getTableOrAnalysisException(tableName.getTbl());
- public Map<StatsType, String> getStatsTypeToValue() {
- return statsTypeToValue;
+ if (table.getType() != Table.TableType.OLAP) {
+ throw new AnalysisException("Only OLAP table statistics are supported");
+ }
+
+ OlapTable olapTable = (OlapTable) table;
+
+ if (olapTable.isPartitioned()) {
+ List<String> names = optPartitionNames.getPartitionNames();
+ Set<String> olapPartitionNames = olapTable.getPartitionNames();
+ Optional<String> optional = names.stream()
+ .filter(name -> !olapPartitionNames.contains(name)).findFirst();
+ if (optional.isPresent()) {
+ throw new AnalysisException("Partition does not exist: " + optional.get());
+ }
+ partitionNames.addAll(optPartitionNames.getPartitionNames());
+ } else {
+ throw new AnalysisException("Not a partitioned table: " + olapTable.getName());
+ }
+ }
Review Comment:
sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #11030: [feature-wip](statistics) step4.1: manually inject statistics for a table or column
Posted by GitBox <gi...@apache.org>.
morrySnow commented on code in PR #11030:
URL: https://github.com/apache/doris/pull/11030#discussion_r937306232
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/AlterColumnStatsStmt.java:
##########
@@ -47,59 +60,124 @@ public class AlterColumnStatsStmt extends DdlStmt {
.add(ColumnStats.MAX_VALUE)
.build();
- private TableName tableName;
- private String columnName;
- private Map<String, String> properties;
- public final Map<StatsType, String> statsTypeToValue = Maps.newHashMap();
+ private final TableName tableName;
+ private final PartitionNames optPartitionNames;
+ private final String columnName;
+ private final Map<String, String> properties;
- public AlterColumnStatsStmt(TableName tableName, String columnName, Map<String, String> properties) {
+ private final List<String> partitionNames = Lists.newArrayList();
+ private final Map<StatsType, String> statsTypeToValue = Maps.newHashMap();
+
+ public AlterColumnStatsStmt(TableName tableName, String columnName,
+ Map<String, String> properties, PartitionNames optPartitionNames) {
this.tableName = tableName;
this.columnName = columnName;
- this.properties = properties;
+ this.properties = properties == null ? Maps.newHashMap() : properties;
+ this.optPartitionNames = optPartitionNames;
+ }
+
+ public TableName getTableName() {
+ return tableName;
+ }
+
+ public String getColumnName() {
+ return columnName;
+ }
+
+ public List<String> getPartitionNames() {
+ return partitionNames;
+ }
+
+ public Map<StatsType, String> getStatsTypeToValue() {
+ return statsTypeToValue;
}
@Override
public void analyze(Analyzer analyzer) throws UserException {
super.analyze(analyzer);
+
// check table name
tableName.analyze(analyzer);
+
// disallow external catalog
Util.prohibitExternalCatalog(tableName.getCtl(), this.getClass().getSimpleName());
+
+ // check partition & column
+ checkPartitionAndColumnNames();
+
// check properties
Optional<StatsType> optional = properties.keySet().stream().map(StatsType::fromString)
- .filter(statsType -> !CONFIGURABLE_PROPERTIES_SET.contains(statsType)).findFirst();
+ .filter(statsType -> !CONFIGURABLE_PROPERTIES_SET.contains(statsType))
+ .findFirst();
if (optional.isPresent()) {
- throw new AnalysisException(optional.get() + " is invalid statistic");
+ throw new AnalysisException(optional.get() + " is invalid statistics");
}
+
// check auth
- if (!Env.getCurrentEnv().getAuth().checkTblPriv(
- ConnectContext.get(), tableName.getDb(), tableName.getTbl(), PrivPredicate.ALTER)) {
+ if (!Env.getCurrentEnv().getAuth()
+ .checkTblPriv(ConnectContext.get(), tableName.getDb(), tableName.getTbl(), PrivPredicate.ALTER)) {
ErrorReport.reportAnalysisException(ErrorCode.ERR_TABLEACCESS_DENIED_ERROR, "ALTER COLUMN STATS",
- ConnectContext.get().getQualifiedUser(),
- ConnectContext.get().getRemoteIP(),
+ ConnectContext.get().getQualifiedUser(), ConnectContext.get().getRemoteIP(),
tableName.getDb() + ": " + tableName.getTbl());
}
+
// get statsTypeToValue
properties.forEach((key, value) -> {
StatsType statsType = StatsType.fromString(key);
statsTypeToValue.put(statsType, value);
});
}
- public TableName getTableName() {
- return tableName;
- }
+ private void checkPartitionAndColumnNames() throws AnalysisException {
Review Comment:
add a todo to support external catalog and external table
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/AlterTableStatsStmt.java:
##########
@@ -35,61 +39,125 @@
import java.util.List;
import java.util.Map;
import java.util.Optional;
+import java.util.Set;
+/**
+ * Manually inject statistics for tables or partitions.
+ * Only OLAP table statistics are supported.
+ * e.g.
+ * ALTER TABLE table_name
+ * SET STATS ('k1' = 'v1', ...) [ PARTITIONS(p_name1, p_name2...) ]
+ */
public class AlterTableStatsStmt extends DdlStmt {
private static final ImmutableSet<StatsType> CONFIGURABLE_PROPERTIES_SET = new ImmutableSet.Builder<StatsType>()
.add(TableStats.DATA_SIZE)
.add(TableStats.ROW_COUNT)
.build();
- private TableName tableName;
- private Map<String, String> properties;
- public final Map<StatsType, String> statsTypeToValue = Maps.newHashMap();
+ private final TableName tableName;
+ private final PartitionNames optPartitionNames;
+ private final Map<String, String> properties;
- public AlterTableStatsStmt(TableName tableName, Map<String, String> properties) {
+ private final List<String> partitionNames = Lists.newArrayList();
+ private final Map<StatsType, String> statsTypeToValue = Maps.newHashMap();
+
+ public AlterTableStatsStmt(TableName tableName, Map<String, String> properties,
+ PartitionNames optPartitionNames) {
this.tableName = tableName;
- this.properties = properties;
+ this.properties = properties == null ? Maps.newHashMap() : properties;
+ this.optPartitionNames = optPartitionNames;
+ }
+
+ public TableName getTableName() {
+ return tableName;
+ }
+
+ public List<String> getPartitionNames() {
+ return partitionNames;
+ }
+
+ public Map<StatsType, String> getStatsTypeToValue() {
+ return statsTypeToValue;
}
@Override
public void analyze(Analyzer analyzer) throws UserException {
super.analyze(analyzer);
+
// check table name
tableName.analyze(analyzer);
+
// disallow external catalog
Util.prohibitExternalCatalog(tableName.getCtl(), this.getClass().getSimpleName());
+
+ // check partition
+ checkPartitionNames();
+
// check properties
Optional<StatsType> optional = properties.keySet().stream().map(StatsType::fromString)
- .filter(statsType -> !CONFIGURABLE_PROPERTIES_SET.contains(statsType)).findFirst();
+ .filter(statsType -> !CONFIGURABLE_PROPERTIES_SET.contains(statsType))
+ .findFirst();
if (optional.isPresent()) {
- throw new AnalysisException(optional.get() + " is invalid statistic");
+ throw new AnalysisException(optional.get() + " is invalid statistics");
}
+
// check auth
- if (!Env.getCurrentEnv().getAuth().checkTblPriv(
- ConnectContext.get(), tableName.getDb(), tableName.getTbl(), PrivPredicate.ALTER)) {
+ if (!Env.getCurrentEnv().getAuth()
+ .checkTblPriv(ConnectContext.get(), tableName.getDb(), tableName.getTbl(), PrivPredicate.ALTER)) {
ErrorReport.reportAnalysisException(ErrorCode.ERR_TABLEACCESS_DENIED_ERROR, "ALTER TABLE STATS",
- ConnectContext.get().getQualifiedUser(),
- ConnectContext.get().getRemoteIP(),
+ ConnectContext.get().getQualifiedUser(), ConnectContext.get().getRemoteIP(),
tableName.getDb() + ": " + tableName.getTbl());
}
+
// get statsTypeToValue
properties.forEach((key, value) -> {
StatsType statsType = StatsType.fromString(key);
statsTypeToValue.put(statsType, value);
});
}
- public TableName getTableName() {
- return tableName;
- }
+ private void checkPartitionNames() throws AnalysisException {
+ if (optPartitionNames != null) {
+ optPartitionNames.analyze(analyzer);
+ Database db = analyzer.getEnv().getInternalDataSource()
+ .getDbOrAnalysisException(tableName.getDb());
+ Table table = db.getTableOrAnalysisException(tableName.getTbl());
- public Map<StatsType, String> getStatsTypeToValue() {
- return statsTypeToValue;
+ if (table.getType() != Table.TableType.OLAP) {
+ throw new AnalysisException("Only OLAP table statistics are supported");
+ }
+
+ OlapTable olapTable = (OlapTable) table;
+
+ if (olapTable.isPartitioned()) {
+ List<String> names = optPartitionNames.getPartitionNames();
+ Set<String> olapPartitionNames = olapTable.getPartitionNames();
+ Optional<String> optional = names.stream()
+ .filter(name -> !olapPartitionNames.contains(name)).findFirst();
+ if (optional.isPresent()) {
+ throw new AnalysisException("Partition does not exist: " + optional.get());
+ }
+ partitionNames.addAll(optPartitionNames.getPartitionNames());
+ } else {
+ throw new AnalysisException("Not a partitioned table: " + olapTable.getName());
+ }
+ }
Review Comment:
should we need to handle table is unpartitioned and optPartitionNames is not empty just like AlterColumnStats?
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/AlterColumnStatsStmt.java:
##########
@@ -47,59 +60,124 @@ public class AlterColumnStatsStmt extends DdlStmt {
.add(ColumnStats.MAX_VALUE)
.build();
- private TableName tableName;
- private String columnName;
- private Map<String, String> properties;
- public final Map<StatsType, String> statsTypeToValue = Maps.newHashMap();
+ private final TableName tableName;
+ private final PartitionNames optPartitionNames;
+ private final String columnName;
+ private final Map<String, String> properties;
- public AlterColumnStatsStmt(TableName tableName, String columnName, Map<String, String> properties) {
+ private final List<String> partitionNames = Lists.newArrayList();
+ private final Map<StatsType, String> statsTypeToValue = Maps.newHashMap();
+
+ public AlterColumnStatsStmt(TableName tableName, String columnName,
+ Map<String, String> properties, PartitionNames optPartitionNames) {
this.tableName = tableName;
this.columnName = columnName;
- this.properties = properties;
+ this.properties = properties == null ? Maps.newHashMap() : properties;
Review Comment:
```suggestion
this.properties = properties == null ? Collections.emptyMap() : properties;
```
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/AlterColumnStatsStmt.java:
##########
@@ -47,59 +60,124 @@ public class AlterColumnStatsStmt extends DdlStmt {
.add(ColumnStats.MAX_VALUE)
.build();
- private TableName tableName;
- private String columnName;
- private Map<String, String> properties;
- public final Map<StatsType, String> statsTypeToValue = Maps.newHashMap();
+ private final TableName tableName;
+ private final PartitionNames optPartitionNames;
+ private final String columnName;
+ private final Map<String, String> properties;
- public AlterColumnStatsStmt(TableName tableName, String columnName, Map<String, String> properties) {
+ private final List<String> partitionNames = Lists.newArrayList();
+ private final Map<StatsType, String> statsTypeToValue = Maps.newHashMap();
+
+ public AlterColumnStatsStmt(TableName tableName, String columnName,
+ Map<String, String> properties, PartitionNames optPartitionNames) {
this.tableName = tableName;
this.columnName = columnName;
- this.properties = properties;
+ this.properties = properties == null ? Maps.newHashMap() : properties;
+ this.optPartitionNames = optPartitionNames;
+ }
+
+ public TableName getTableName() {
+ return tableName;
+ }
+
+ public String getColumnName() {
+ return columnName;
+ }
+
+ public List<String> getPartitionNames() {
+ return partitionNames;
+ }
+
+ public Map<StatsType, String> getStatsTypeToValue() {
+ return statsTypeToValue;
}
@Override
public void analyze(Analyzer analyzer) throws UserException {
super.analyze(analyzer);
+
// check table name
tableName.analyze(analyzer);
+
// disallow external catalog
Util.prohibitExternalCatalog(tableName.getCtl(), this.getClass().getSimpleName());
+
+ // check partition & column
+ checkPartitionAndColumnNames();
+
// check properties
Optional<StatsType> optional = properties.keySet().stream().map(StatsType::fromString)
- .filter(statsType -> !CONFIGURABLE_PROPERTIES_SET.contains(statsType)).findFirst();
+ .filter(statsType -> !CONFIGURABLE_PROPERTIES_SET.contains(statsType))
+ .findFirst();
if (optional.isPresent()) {
- throw new AnalysisException(optional.get() + " is invalid statistic");
+ throw new AnalysisException(optional.get() + " is invalid statistics");
}
+
// check auth
- if (!Env.getCurrentEnv().getAuth().checkTblPriv(
- ConnectContext.get(), tableName.getDb(), tableName.getTbl(), PrivPredicate.ALTER)) {
+ if (!Env.getCurrentEnv().getAuth()
+ .checkTblPriv(ConnectContext.get(), tableName.getDb(), tableName.getTbl(), PrivPredicate.ALTER)) {
ErrorReport.reportAnalysisException(ErrorCode.ERR_TABLEACCESS_DENIED_ERROR, "ALTER COLUMN STATS",
- ConnectContext.get().getQualifiedUser(),
- ConnectContext.get().getRemoteIP(),
+ ConnectContext.get().getQualifiedUser(), ConnectContext.get().getRemoteIP(),
tableName.getDb() + ": " + tableName.getTbl());
}
+
// get statsTypeToValue
properties.forEach((key, value) -> {
StatsType statsType = StatsType.fromString(key);
statsTypeToValue.put(statsType, value);
});
}
- public TableName getTableName() {
- return tableName;
- }
+ private void checkPartitionAndColumnNames() throws AnalysisException {
+ Database db = analyzer.getEnv().getInternalDataSource()
+ .getDbOrAnalysisException(tableName.getDb());
+ Table table = db.getTableOrAnalysisException(tableName.getTbl());
- public String getColumnName() {
- return columnName;
- }
+ if (table.getType() != Table.TableType.OLAP) {
+ throw new AnalysisException("Only OLAP table statistics are supported");
+ }
- public Map<StatsType, String> getStatsTypeToValue() {
- return statsTypeToValue;
+ OlapTable olapTable = (OlapTable) table;
+
+ if (olapTable.getColumn(columnName) == null) {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_WRONG_COLUMN_NAME, columnName);
+ }
+
+ if (optPartitionNames != null) {
+ optPartitionNames.analyze(analyzer);
+ if (!olapTable.isPartitioned()) {
+ throw new AnalysisException("Not a partitioned table: " + olapTable.getName());
+ }
+ Set<String> olapPartitionNames = olapTable.getPartitionNames();
+ Optional<String> optional = optPartitionNames.getPartitionNames().stream()
+ .filter(name -> !olapPartitionNames.contains(name)).findFirst();
Review Comment:
it is better to use anyMatch instead of findFirst and isPresent.
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/AlterColumnStatsStmt.java:
##########
@@ -47,59 +60,124 @@ public class AlterColumnStatsStmt extends DdlStmt {
.add(ColumnStats.MAX_VALUE)
.build();
- private TableName tableName;
- private String columnName;
- private Map<String, String> properties;
- public final Map<StatsType, String> statsTypeToValue = Maps.newHashMap();
+ private final TableName tableName;
+ private final PartitionNames optPartitionNames;
+ private final String columnName;
+ private final Map<String, String> properties;
- public AlterColumnStatsStmt(TableName tableName, String columnName, Map<String, String> properties) {
+ private final List<String> partitionNames = Lists.newArrayList();
+ private final Map<StatsType, String> statsTypeToValue = Maps.newHashMap();
+
+ public AlterColumnStatsStmt(TableName tableName, String columnName,
+ Map<String, String> properties, PartitionNames optPartitionNames) {
this.tableName = tableName;
this.columnName = columnName;
- this.properties = properties;
+ this.properties = properties == null ? Maps.newHashMap() : properties;
+ this.optPartitionNames = optPartitionNames;
+ }
+
+ public TableName getTableName() {
+ return tableName;
+ }
+
+ public String getColumnName() {
+ return columnName;
+ }
+
+ public List<String> getPartitionNames() {
+ return partitionNames;
+ }
+
+ public Map<StatsType, String> getStatsTypeToValue() {
+ return statsTypeToValue;
}
@Override
public void analyze(Analyzer analyzer) throws UserException {
super.analyze(analyzer);
+
// check table name
tableName.analyze(analyzer);
+
// disallow external catalog
Util.prohibitExternalCatalog(tableName.getCtl(), this.getClass().getSimpleName());
+
+ // check partition & column
+ checkPartitionAndColumnNames();
+
// check properties
Optional<StatsType> optional = properties.keySet().stream().map(StatsType::fromString)
- .filter(statsType -> !CONFIGURABLE_PROPERTIES_SET.contains(statsType)).findFirst();
+ .filter(statsType -> !CONFIGURABLE_PROPERTIES_SET.contains(statsType))
+ .findFirst();
if (optional.isPresent()) {
- throw new AnalysisException(optional.get() + " is invalid statistic");
+ throw new AnalysisException(optional.get() + " is invalid statistics");
}
+
// check auth
- if (!Env.getCurrentEnv().getAuth().checkTblPriv(
- ConnectContext.get(), tableName.getDb(), tableName.getTbl(), PrivPredicate.ALTER)) {
+ if (!Env.getCurrentEnv().getAuth()
+ .checkTblPriv(ConnectContext.get(), tableName.getDb(), tableName.getTbl(), PrivPredicate.ALTER)) {
ErrorReport.reportAnalysisException(ErrorCode.ERR_TABLEACCESS_DENIED_ERROR, "ALTER COLUMN STATS",
- ConnectContext.get().getQualifiedUser(),
- ConnectContext.get().getRemoteIP(),
+ ConnectContext.get().getQualifiedUser(), ConnectContext.get().getRemoteIP(),
tableName.getDb() + ": " + tableName.getTbl());
}
+
// get statsTypeToValue
properties.forEach((key, value) -> {
StatsType statsType = StatsType.fromString(key);
statsTypeToValue.put(statsType, value);
});
}
- public TableName getTableName() {
- return tableName;
- }
+ private void checkPartitionAndColumnNames() throws AnalysisException {
+ Database db = analyzer.getEnv().getInternalDataSource()
+ .getDbOrAnalysisException(tableName.getDb());
+ Table table = db.getTableOrAnalysisException(tableName.getTbl());
- public String getColumnName() {
- return columnName;
- }
+ if (table.getType() != Table.TableType.OLAP) {
+ throw new AnalysisException("Only OLAP table statistics are supported");
+ }
- public Map<StatsType, String> getStatsTypeToValue() {
- return statsTypeToValue;
+ OlapTable olapTable = (OlapTable) table;
+
+ if (olapTable.getColumn(columnName) == null) {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_WRONG_COLUMN_NAME, columnName);
+ }
+
+ if (optPartitionNames != null) {
+ optPartitionNames.analyze(analyzer);
Review Comment:
mv L147 below to L148's if statement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11030: [feature-wip](statistics) step4.1: manually inject statistics for a table or column
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #11030:
URL: https://github.com/apache/doris/pull/11030#issuecomment-1205996407
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org