You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2019/12/25 11:48:58 UTC
[GitHub] [incubator-doris] yangzhg opened a new pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
yangzhg opened a new pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573
## support for syntax
### create table with index
```
CREATE TABLE table1
(
siteid INT DEFAULT '10',
citycode SMALLINT,
username VARCHAR(32) DEFAULT '',
pv BIGINT SUM DEFAULT '0',
INDEX index_name [USING BITMAP] (siteid, citycode) COMMENT 'balabala'
)
AGGREGATE KEY(siteid, citycode, username)
DISTRIBUTED BY HASH(siteid) BUCKETS 10
PROPERTIES("replication_num" = "1");
```
### create index
```
CREATE INDEX index_name ON table1 (siteid, citycod) [USING BITMAP] COMMENT 'balabala';
or
ALTER TABLE table1 ADD INDEX index_name [USING BITMAP] (siteid, citycod) COMMENT 'balabala';
```
### drop index
```
DROP INDEX index_name ON table1;
or
ALTER TABLE table1 DROP INDEX index_name
```
### show index
```
SHOW INDEX[ES] FROM table1
```
output
```
+---------+-------------+-----------------+------------+---------+
| Table | Index_name | Column_name | Index_type | Comment |
+---------+-------------+-----------------+------------+---------+
| table1 | index_name | siteid,citycode | BITMAMP | balabala|
+---------+-------------+-----------------+------------+---------+
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang commented on a change in pull
request #2573: implements create drop show index syntax for bitmap index
[#2487]
Posted by GitBox <gi...@apache.org>.
kangpinghuang commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361603273
##########
File path: fe/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java
##########
@@ -779,6 +783,14 @@ private void createJob(long dbId, OlapTable olapTable, Map<Long, LinkedList<Colu
} // end for property keys
}
+ // for bitmapIndex
+ boolean hasIndexChange = false;
+ Set<Index> newSet = new HashSet<>(indexes);
+ Set<Index> oriSet = new HashSet<>(olapTable.getIndexes());
+ if (!newSet.equals(oriSet)) {
+ hasIndexChange = true;
+ }
Review comment:
what if the user just modify the index name on the same column by creating the same index type?
In this case, it seems that we just modify the column's index name and return ok.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request
#2573: implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361367406
##########
File path: fe/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java
##########
@@ -838,6 +854,20 @@ public void write(DataOutput out) throws IOException {
}
out.writeLong(watershedTxnId);
+
+ // index
+ if (FeMetaVersion.VERSION_CURRENT >= FeMetaVersion.VERSION_70) {
Review comment:
write don't need version check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r362231564
##########
File path: fe/src/main/cup/sql_parser.cup
##########
@@ -1711,6 +1759,13 @@ column_definition ::=
:}
;
+index_definition ::=
+ KW_INDEX ident:indexName opt_index_type:indexType LPAREN ident_list:cols RPAREN opt_comment:comment
Review comment:
I think its better to move index_type after ident_list
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request
#2573: implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
yangzhg commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361885483
##########
File path: fe/src/main/java/org/apache/doris/analysis/AlterTableStmt.java
##########
@@ -61,7 +66,41 @@ public void analyze(Analyzer analyzer) throws AnalysisException, UserException {
ErrorReport.reportAnalysisException(ErrorCode.ERR_NO_ALTER_OPERATION);
}
for (AlterClause op : ops) {
- op.analyze(analyzer);
+ if (op instanceof CreateIndexClause) {
+ Table table = Catalog.getInstance().getDb(tbl.getDb()).getTable(tbl.getTbl());
+ if (table instanceof OlapTable) {
+ op.analyze(analyzer);
+ List<Index> indexes = ((OlapTable) table).getIndexes();
+ IndexDef indexDef = ((CreateIndexClause) op).getIndexDef();
+ for (Index idx : indexes) {
+ if (idx.getIndexName().equalsIgnoreCase(indexDef.getIndexName())) {
+ throw new AnalysisException("index `" + indexDef.getIndexName() + "` already exist.");
+ }
+ }
+ }
+ } else if (op instanceof DropIndexClause) {
Review comment:
no
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r362321613
##########
File path: fe/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java
##########
@@ -1438,4 +1457,19 @@ public void cancel(CancelStmt stmt) throws DdlException {
jobDone(schemaChangeJob);
}
}
+
+ private void processAddIndex(CreateIndexClause alterClause, List<Index> indexes) {
+ if (alterClause.getIndex() != null) {
+ indexes.add(alterClause.getIndex());
+ }
+ }
+
+ private void processDropIndex(DropIndexClause alterClause, List<Index> indexes) {
+ for (Index idx : indexes) {
Review comment:
This is not a good idea to iterate list in this way when element will be deleted in the loop block
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request
#2573: implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
yangzhg commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361563299
##########
File path: be/src/olap/tablet_schema.h
##########
@@ -80,6 +81,8 @@ class TabletColumn {
bool _has_referenced_column;
int32_t _referenced_column_id;
std::string _referenced_column;
+
+ bool _is_bitmap_column;
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request
#2573: implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
yangzhg commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361561284
##########
File path: be/src/olap/tablet_schema.h
##########
@@ -40,6 +40,7 @@ class TabletColumn {
inline bool is_key() const { return _is_key; }
inline bool is_nullable() const { return _is_nullable; }
inline bool is_bf_column() const { return _is_bf_column; }
+ inline bool is_bitmap_column() const {return _is_bitmap_column; }
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang commented on a change in pull
request #2573: implements create drop show index syntax for bitmap index
[#2487]
Posted by GitBox <gi...@apache.org>.
kangpinghuang commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361580213
##########
File path: be/src/olap/olap_snapshot_converter.cpp
##########
@@ -379,6 +382,9 @@ OLAPStatus OlapSnapshotConverter::to_column_msg(const ColumnPB& column_pb, Colum
if (column_pb.has_is_bf_column()) {
column_msg->set_is_bf_column(column_pb.is_bf_column());
}
+ if (column_pb.has_is_bitmap_column()) {
Review comment:
ditto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361905542
##########
File path: be/src/olap/tablet_schema.h
##########
@@ -80,6 +81,8 @@ class TabletColumn {
bool _has_referenced_column;
int32_t _referenced_column_id;
std::string _referenced_column;
+
+ bool _is_bitmap_column = false;
Review comment:
_is_bitmap_index
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang commented on a change in pull
request #2573: implements create drop show index syntax for bitmap index
[#2487]
Posted by GitBox <gi...@apache.org>.
kangpinghuang commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361600238
##########
File path: docs/documentation/cn/sql-reference/sql-statements/Data Definition/CREATE TABLE.md
##########
@@ -156,260 +183,321 @@ under the License.
3) 分区为左闭右开区间，首个分区的左边界为做最小值
4) NULL 值只会存放在包含最小值的分区中。当包含最小值的分区被删除后，NULL 值将无法导入。
5) 可以指定一列或多列作为分区列。如果分区值缺省，则会默认填充最小值。
-
+
注意：
1) 分区一般用于时间维度的数据管理
2) 有数据回溯需求的，可以考虑首个分区为空分区，以便后续增加分区
-
- 2）Fixed Range
+
+ 2）Fixed Range
语法：
+ ```
PARTITION BY RANGE (k1, k2, k3, ...)
(
PARTITION partition_name1 VALUES [("k1-lower1", "k2-lower1", "k3-lower1",...), ("k1-upper1", "k2-upper1", "k3-upper1", ...)),
PARTITION partition_name2 VALUES [("k1-lower1-2", "k2-lower1-2", ...), ("k1-upper1-2", MAXVALUE, ))
"k3-upper1-2", ...
)
+ ```
说明：
1）Fixed Range比LESS THAN相对灵活些，左右区间完全由用户自己确定
2）其他与LESS THAN保持同步
- 5. distribution_desc
+3. distribution_desc
1) Hash 分桶
语法：
- DISTRIBUTED BY HASH (k1[,k2 ...]) [BUCKETS num]
+ `DISTRIBUTED BY HASH (k1[,k2 ...]) [BUCKETS num]`
说明：
使用指定的 key 列进行哈希分桶。默认分区数为10
- 建议:建议使用Hash分桶方式
+ 建议:建议使用Hash分桶方式
- 6. PROPERTIES
+4. PROPERTIES
1) 如果 ENGINE 类型为 olap，则可以在 properties 中指定列存（目前我们仅支持列存）
-
- PROPERTIES (
+ ```
+ PROPERTIES (
"storage_type" = "[column]",
)
-
- 2) 如果 ENGINE 类型为 olap
+ ```
+ 1) 如果 ENGINE 类型为 olap
Review comment:
incorrect sequence id? 2)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request
#2573: implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361366779
##########
File path: be/src/olap/tablet_manager.cpp
##########
@@ -1215,7 +1215,15 @@ OLAPStatus TabletManager::_create_inital_rowset(
context.tablet_id = tablet->tablet_id();
context.partition_id = tablet->partition_id();
context.tablet_schema_hash = tablet->schema_hash();
- context.rowset_type = StorageEngine::instance()->default_rowset_type();
+ if (!request.__isset.storage_format || request.storage_format == TStorageFormat::DEFAULT) {
+ context.rowset_type = StorageEngine::instance()->default_rowset_type();
+ } else if (request.storage_format == TStorageFormat::V1){
+ context.rowset_type = RowsetTypePB::ALPHA_ROWSET;
+ } else if (request.storage_format == TStorageFormat::V2) {
+ context.rowset_type = RowsetTypePB::BETA_ROWSET;
+ } else {
+ context.rowset_type = StorageEngine::instance()->default_rowset_type();
Review comment:
Do we need this branch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361560812
##########
File path: fe/src/main/java/org/apache/doris/analysis/IndexDef.java
##########
@@ -0,0 +1,165 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.AggregateType;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.PrimitiveType;
+import org.apache.doris.common.AnalysisException;
+
+import org.apache.commons.lang.StringUtils;
+
+import java.util.List;
+import java.util.TreeSet;
+
+public class IndexDef {
+ private String indexName;
+ private List<String> columns;
+ private IndexType indexType;
+ private String comment;
+
+ public IndexDef(String indexName, List<String> columns, IndexType indexType, String comment) {
+ this.indexName = indexName;
+ this.columns = columns;
+ if (indexType == null) {
+ this.indexType = IndexType.BITMAP;
+ } else {
+ this.indexType = indexType;
+ }
+ if (columns == null) {
+ this.comment = "";
+ } else {
+ this.comment = comment;
+ }
+ }
+
+ public void analyze(boolean isOlap, List<Column> tableColumns) throws AnalysisException {
+ if (indexType == IndexDef.IndexType.BITMAP) {
+ if (!isOlap) {
+ throw new AnalysisException("bitmap index can only be used with olap engine");
+ }
+ if (columns == null || columns.size() != 1) {
+ throw new AnalysisException("bitmap index definition expect at least one column.");
+ }
+ if (StringUtils.isBlank(indexName)) {
+ throw new AnalysisException("index name cannot be blank.");
+ }
+ if (indexName.length() > 64) {
+ throw new AnalysisException("index name too long, the index name length at most is 64.");
+ }
+ TreeSet<String> distinct = new TreeSet<>(String.CASE_INSENSITIVE_ORDER);
+ distinct.addAll(columns);
+ if (columns.size() != distinct.size()) {
+ throw new AnalysisException("columns of index has duplicated.");
+ }
+ for (String indexColName : columns) {
Review comment:
This should do by out of this class
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361906248
##########
File path: fe/src/main/java/org/apache/doris/qe/ShowExecutor.java
##########
@@ -639,6 +643,35 @@ private void handleShowColumn() throws AnalysisException {
resultSet = new ShowResultSet(showStmt.getMetaData(), rows);
}
+ // Show index statement.
+ private void handleShowIndex() throws AnalysisException {
+ ShowIndexStmt showStmt = (ShowIndexStmt) stmt;
+ List<List<String>> rows = Lists.newArrayList();
+ Database db = ctx.getCatalog().getDb(showStmt.getDbName());
+ if (db != null) {
+ db.readLock();
+ try {
+ Table table = db.getTable(showStmt.getTableName().getTbl());
+ if (table != null && table instanceof OlapTable) {
+ List<Index> indexes = ((OlapTable) table).getIndexes();
+ for (Index index : indexes) {
+ rows.add(Lists.newArrayList(showStmt.getTableName().toString(), index.getIndexName(),
+ index.getColumns().stream().collect(Collectors.joining(",")),
+ index.getType().name(), index.getComment()));
+ }
+ } else {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_BAD_TABLE_ERROR,
+ db.getFullName() + "." + showStmt.getTableName().toString());
+ }
+ } finally {
+ db.readUnlock();
+ }
+ } else {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_BAD_TABLE_ERROR, showStmt.getTableName().toString());
Review comment:
better throw it early to make less indentation
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang commented on a change in pull
request #2573: implements create drop show index syntax for bitmap index
[#2487]
Posted by GitBox <gi...@apache.org>.
kangpinghuang commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361605201
##########
File path: fe/src/main/java/org/apache/doris/analysis/AlterTableStmt.java
##########
@@ -61,7 +66,41 @@ public void analyze(Analyzer analyzer) throws AnalysisException, UserException {
ErrorReport.reportAnalysisException(ErrorCode.ERR_NO_ALTER_OPERATION);
}
for (AlterClause op : ops) {
- op.analyze(analyzer);
+ if (op instanceof CreateIndexClause) {
+ Table table = Catalog.getInstance().getDb(tbl.getDb()).getTable(tbl.getTbl());
+ if (table instanceof OlapTable) {
+ op.analyze(analyzer);
+ List<Index> indexes = ((OlapTable) table).getIndexes();
+ IndexDef indexDef = ((CreateIndexClause) op).getIndexDef();
+ for (Index idx : indexes) {
+ if (idx.getIndexName().equalsIgnoreCase(indexDef.getIndexName())) {
+ throw new AnalysisException("index `" + indexDef.getIndexName() + "` already exist.");
+ }
+ }
+ }
+ } else if (op instanceof DropIndexClause) {
Review comment:
do we support just modifying index name?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r362231987
##########
File path: fe/src/main/java/org/apache/doris/alter/Alter.java
##########
@@ -169,9 +176,50 @@ public void processAlterTable(AlterTableStmt stmt) throws UserException {
|| alterClause instanceof AddColumnsClause
|| alterClause instanceof DropColumnClause
|| alterClause instanceof ModifyColumnClause
- || alterClause instanceof ReorderColumnsClause)
+ || alterClause instanceof ReorderColumnsClause
+ || alterClause instanceof CreateIndexClause
+ || alterClause instanceof DropIndexClause)
&& !hasAddMaterializedView && !hasDropRollup && !hasPartition && !hasRename) {
hasSchemaChange = true;
+ if (alterClause instanceof CreateIndexClause) {
+ Table table = db.getTable(dbTableName.getTbl());
+ if (table instanceof OlapTable) {
Review comment:
if this is not olaptable, should throw exception?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361906179
##########
File path: fe/src/main/java/org/apache/doris/task/CreateReplicaTask.java
##########
@@ -153,6 +161,16 @@ public TCreateTabletReq toThrift() {
}
tSchema.setColumns(tColumns);
+ if (CollectionUtils.isNotEmpty(indexes)) {
+ List<TOlapTableIndex> tIndexes = new ArrayList<>();
+ for (Index index : indexes) {
+ tIndexes.add(index.toThrift());
+ }
+ tSchema.setIndexes(tIndexes);
+ LOG.info("using bitmap index, force set StorageFormat to V2");
Review comment:
remove this log?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang commented on a change in pull
request #2573: implements create drop show index syntax for bitmap index
[#2487]
Posted by GitBox <gi...@apache.org>.
kangpinghuang commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361407245
##########
File path: be/src/olap/tablet_schema.cpp
##########
@@ -295,6 +295,11 @@ OLAPStatus TabletColumn::init_from_pb(const ColumnPB& column) {
} else {
_is_bf_column = false;
}
+ if (column.has_is_bitmap_column()) {
+ _is_bitmap_column = column.is_bitmap_column();
+ } else {
+ _is_bitmap_column = false;
Review comment:
this can be done in constructor?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361474488
##########
File path: gensrc/thrift/Descriptors.thrift
##########
@@ -166,6 +170,13 @@ struct TOlapTableSchemaParam {
6: required list<TOlapTableIndexSchema> indexes
}
+struct TOlapTableIndex {
Review comment:
better to use optional for all fields
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang commented on a change in pull
request #2573: implements create drop show index syntax for bitmap index
[#2487]
Posted by GitBox <gi...@apache.org>.
kangpinghuang commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361603461
##########
File path: fe/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java
##########
@@ -1298,7 +1313,7 @@ public void process(List<AlterClause> alterClauses, String clusterName, Database
for (Map.Entry<Long, List<Column>> entry : olapTable.getIndexIdToSchema().entrySet()) {
indexSchemaMap.put(entry.getKey(), new LinkedList<Column>(entry.getValue()));
}
-
+ List<Index> newIndex = olapTable.getCopiedIndexes();
Review comment:
```suggestion
List<Index> newIndexes = olapTable.getCopiedIndexes();
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361475012
##########
File path: be/src/olap/tablet_schema.h
##########
@@ -80,6 +81,8 @@ class TabletColumn {
bool _has_referenced_column;
int32_t _referenced_column_id;
std::string _referenced_column;
+
+ bool _is_bitmap_column;
Review comment:
give a default value
```suggestion
bool _is_bitmap_column = false;
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361560459
##########
File path: fe/src/main/java/org/apache/doris/analysis/IndexDef.java
##########
@@ -0,0 +1,165 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.AggregateType;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.PrimitiveType;
+import org.apache.doris.common.AnalysisException;
+
+import org.apache.commons.lang.StringUtils;
+
+import java.util.List;
+import java.util.TreeSet;
+
+public class IndexDef {
+ private String indexName;
+ private List<String> columns;
+ private IndexType indexType;
+ private String comment;
+
+ public IndexDef(String indexName, List<String> columns, IndexType indexType, String comment) {
+ this.indexName = indexName;
+ this.columns = columns;
+ if (indexType == null) {
+ this.indexType = IndexType.BITMAP;
+ } else {
+ this.indexType = indexType;
+ }
+ if (columns == null) {
+ this.comment = "";
+ } else {
+ this.comment = comment;
+ }
+ }
+
+ public void analyze(boolean isOlap, List<Column> tableColumns) throws AnalysisException {
+ if (indexType == IndexDef.IndexType.BITMAP) {
+ if (!isOlap) {
Review comment:
This class is a common index definition, better not to check isOlap here.
Maybe Doris support other types of table which is not olap, but it supports bitmap index too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r362504816
##########
File path: fe/src/main/java/org/apache/doris/alter/Alter.java
##########
@@ -169,9 +175,49 @@ public void processAlterTable(AlterTableStmt stmt) throws UserException {
|| alterClause instanceof AddColumnsClause
|| alterClause instanceof DropColumnClause
|| alterClause instanceof ModifyColumnClause
- || alterClause instanceof ReorderColumnsClause)
+ || alterClause instanceof ReorderColumnsClause
+ || alterClause instanceof CreateIndexClause
+ || alterClause instanceof DropIndexClause)
&& !hasAddMaterializedView && !hasDropRollup && !hasPartition && !hasRename) {
hasSchemaChange = true;
+ if (alterClause instanceof CreateIndexClause) {
+ Table table = db.getTable(dbTableName.getTbl());
+ if (!(table instanceof OlapTable)) {
+ throw new AnalysisException("create index only support in olap table at current version.");
+ }
+ List<Index> indexes = ((OlapTable) table).getIndexes();
+ IndexDef indexDef = ((CreateIndexClause) alterClause).getIndexDef();
+ Set<String> newColset = new TreeSet<>(String.CASE_INSENSITIVE_ORDER);
+ newColset.addAll(indexDef.getColumns());
+ for (Index idx : indexes) {
+ if (idx.getIndexName().equalsIgnoreCase(indexDef.getIndexName())) {
+ throw new AnalysisException("index `" + indexDef.getIndexName() + "` already exist.");
+ }
+ Set<String> idxSet = new TreeSet<>(String.CASE_INSENSITIVE_ORDER);
+ idxSet.addAll(idx.getColumns());
+ if (newColset.equals(idxSet)) {
+ throw new AnalysisException("index for columns (" + String
+ .join(",", indexDef.getColumns()) + " ) already exist.");
+ }
+ }
+
+ } else if (alterClause instanceof DropIndexClause) {
+ Table table = db.getTable(dbTableName.getTbl());
+ if (table instanceof OlapTable) {
Review comment:
If table is not OlapTable, throw exception?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361556160
##########
File path: be/src/olap/tablet_manager.cpp
##########
@@ -1215,7 +1215,15 @@ OLAPStatus TabletManager::_create_inital_rowset(
context.tablet_id = tablet->tablet_id();
context.partition_id = tablet->partition_id();
context.tablet_schema_hash = tablet->schema_hash();
- context.rowset_type = StorageEngine::instance()->default_rowset_type();
+ if (!request.__isset.storage_format || request.storage_format == TStorageFormat::DEFAULT) {
+ context.rowset_type = StorageEngine::instance()->default_rowset_type();
+ } else if (request.storage_format == TStorageFormat::V1){
+ context.rowset_type = RowsetTypePB::ALPHA_ROWSET;
+ } else if (request.storage_format == TStorageFormat::V2) {
+ context.rowset_type = RowsetTypePB::BETA_ROWSET;
+ } else {
+ context.rowset_type = StorageEngine::instance()->default_rowset_type();
Review comment:
maybe leave a DCHECK here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #2573: implements
create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
morningman merged pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request
#2573: implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361889351
##########
File path: fe/src/main/java/org/apache/doris/analysis/AlterTableStmt.java
##########
@@ -61,7 +68,49 @@ public void analyze(Analyzer analyzer) throws AnalysisException, UserException {
ErrorReport.reportAnalysisException(ErrorCode.ERR_NO_ALTER_OPERATION);
}
for (AlterClause op : ops) {
- op.analyze(analyzer);
+ if (op instanceof CreateIndexClause) {
+ Table table = Catalog.getInstance().getDb(tbl.getDb()).getTable(tbl.getTbl());
Review comment:
Database `tbl.getDb()` and Table `tbl.getTbl()` may not exist. This will throw NPE.
And the analyze phase should only do the semantic check. The check you wrote here, such as whether indexes exist, should be done when creating alter job, in `Alter.java`'s `processAlterTable()` method.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request
#2573: implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361367733
##########
File path: fe/src/main/java/org/apache/doris/analysis/IndexDef.java
##########
@@ -0,0 +1,168 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.AggregateType;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.PrimitiveType;
+import org.apache.doris.common.AnalysisException;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang.StringUtils;
+
+import java.util.List;
+import java.util.TreeSet;
+
+public class IndexDef {
+ private String indexName;
+ private List<String> columns;
+ private IndexType indexType;
+ private String comment;
+
+ public IndexDef(String indexName, List<String> columns, IndexType indexType, String comment) {
+ this.indexName = indexName;
+ this.columns = columns;
+ if (indexType == null) {
+ this.indexType = IndexType.BITMAP;
+ } else {
+ this.indexType = indexType;
+ }
+ if (columns == null) {
+ this.comment = "";
+ } else {
+ this.comment = comment;
+ }
+ }
+
+ public void analyze(boolean isOlap, List<Column> tableColumns) throws AnalysisException {
+ if (indexType == IndexDef.IndexType.BITMAP) {
+ if (!isOlap) {
+ throw new AnalysisException("bitmap index can only be used with olap engine");
+ }
+ if (CollectionUtils.isEmpty(columns)) {
+ throw new AnalysisException("index definition expect at least one column.");
+ }
+ if (columns.size() != 1) {
+ throw new AnalysisException("bitmap index must apply to one column.");
+ }
+ if (StringUtils.isBlank(indexName)) {
+ throw new AnalysisException("index name cannot be blank.");
+ }
+ if (indexName.length() > 64) {
+ throw new AnalysisException("index name too long.");
Review comment:
throw new AnalysisException("index name too long, the index name length at most is 64");
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request
#2573: implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
yangzhg commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361885108
##########
File path: fe/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java
##########
@@ -779,6 +783,14 @@ private void createJob(long dbId, OlapTable olapTable, Map<Long, LinkedList<Colu
} // end for property keys
}
+ // for bitmapIndex
+ boolean hasIndexChange = false;
+ Set<Index> newSet = new HashSet<>(indexes);
+ Set<Index> oriSet = new HashSet<>(olapTable.getIndexes());
+ if (!newSet.equals(oriSet)) {
+ hasIndexChange = true;
+ }
Review comment:
don`t support rename index for current version
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r362321667
##########
File path: fe/src/main/java/org/apache/doris/analysis/CreateIndexClause.java
##########
@@ -0,0 +1,75 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.Index;
+import org.apache.doris.common.AnalysisException;
+
+import java.util.Map;
+
+public class CreateIndexClause extends AlterClause {
+ private TableName tableName;
+ private IndexDef indexDef;
+ private boolean alter;
Review comment:
give this field some comments to help other to understand
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361559786
##########
File path: fe/src/main/java/org/apache/doris/catalog/Index.java
##########
@@ -0,0 +1,157 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.catalog;
+
+import org.apache.doris.analysis.IndexDef;
+import org.apache.doris.common.io.Text;
+import org.apache.doris.common.io.Writable;
+import org.apache.doris.thrift.TIndexType;
+import org.apache.doris.thrift.TOlapTableIndex;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+public class Index implements Writable {
Review comment:
Add some comments for it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang commented on a change in pull
request #2573: implements create drop show index syntax for bitmap index
[#2487]
Posted by GitBox <gi...@apache.org>.
kangpinghuang commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361606624
##########
File path: fe/src/main/java/org/apache/doris/qe/ShowExecutor.java
##########
@@ -639,6 +643,35 @@ private void handleShowColumn() throws AnalysisException {
resultSet = new ShowResultSet(showStmt.getMetaData(), rows);
}
+ // Show index statement.
+ private void handleShowIndex() throws AnalysisException {
+ ShowIndexStmt showStmt = (ShowIndexStmt) stmt;
+ List<List<String>> rows = Lists.newArrayList();
+ Database db = ctx.getCatalog().getDb(showStmt.getDbName());
+ if (db != null) {
+ db.readLock();
+ try {
+ Table table = db.getTable(showStmt.getTableName().getTbl());
+ if (table != null && table instanceof OlapTable) {
+ List<Index> indexes = ((OlapTable) table).getIndexes();
+ for (Index index : indexes) {
+ rows.add(Lists.newArrayList(showStmt.getTableName().toString(), index.getIndexName(),
+ index.getColumns().stream().collect(Collectors.joining(",")),
+ index.getType().name(), index.getComment()));
+ }
+ } else {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_BAD_TABLE_ERROR,
+ db.getFullName() + "." + showStmt.getTableName().toString());
+ }
+ } finally {
+ db.readUnlock();
+ }
+ } else {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_BAD_TABLE_ERROR, showStmt.getTableName().toString());
Review comment:
BAD_DB?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r362321676
##########
File path: fe/src/main/java/org/apache/doris/analysis/CreateIndexClause.java
##########
@@ -0,0 +1,75 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.Index;
+import org.apache.doris.common.AnalysisException;
+
+import java.util.Map;
+
+public class CreateIndexClause extends AlterClause {
+ private TableName tableName;
+ private IndexDef indexDef;
+ private boolean alter;
+ private Index index;
+ public CreateIndexClause(TableName tableName, IndexDef indexDef, boolean alter) {
+ this.tableName = tableName;
+ this.indexDef = indexDef;
+ this.alter = alter;
+ }
+
+ @Override
+ public Map<String, String> getProperties() {
+ return null;
Review comment:
Maybe empty map is preferred
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang commented on a change in pull
request #2573: implements create drop show index syntax for bitmap index
[#2487]
Posted by GitBox <gi...@apache.org>.
kangpinghuang commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361603736
##########
File path: fe/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java
##########
@@ -1298,7 +1313,7 @@ public void process(List<AlterClause> alterClauses, String clusterName, Database
for (Map.Entry<Long, List<Column>> entry : olapTable.getIndexIdToSchema().entrySet()) {
indexSchemaMap.put(entry.getKey(), new LinkedList<Column>(entry.getValue()));
}
-
+ List<Index> newIndex = olapTable.getCopiedIndexes();
Review comment:
here is originIndexes or newIndexes?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang commented on a change in pull
request #2573: implements create drop show index syntax for bitmap index
[#2487]
Posted by GitBox <gi...@apache.org>.
kangpinghuang commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361605951
##########
File path: fe/src/main/java/org/apache/doris/catalog/Index.java
##########
@@ -0,0 +1,161 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.catalog;
+
+import org.apache.doris.analysis.IndexDef;
+import org.apache.doris.common.io.Text;
+import org.apache.doris.common.io.Writable;
+import org.apache.doris.thrift.TIndexType;
+import org.apache.doris.thrift.TOlapTableIndex;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Internal representation of index, including index type, name, columns and comments.
+ * this class will used in olaptable
Review comment:
```suggestion
* This class will used in olaptable
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r362513561
##########
File path: fe/src/main/java/org/apache/doris/catalog/Index.java
##########
@@ -0,0 +1,180 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.catalog;
+
+import org.apache.doris.analysis.IndexDef;
+import org.apache.doris.common.io.Text;
+import org.apache.doris.common.io.Writable;
+import org.apache.doris.thrift.TIndexType;
+import org.apache.doris.thrift.TOlapTableIndex;
+
+import com.google.common.base.Strings;
+
+import org.json.JSONArray;
+import org.json.JSONObject;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Internal representation of index, including index type, name, columns and comments.
+ * This class will used in olaptable
+ */
+public class Index implements Writable {
+ private String indexName;
+ private List<String> columns;
+ private IndexDef.IndexType type;
+ private String comment;
+
+ public Index(String indexName, List<String> columns, IndexDef.IndexType type, String comment) {
+ this.indexName = indexName;
+ this.columns = columns;
+ this.type = type;
+ this.comment = comment;
+ }
+
+ public Index() {
+ this.indexName = null;
+ this.columns = null;
+ this.type = null;
+ this.comment = null;
+ }
+
+ public String getIndexName() {
+ return indexName;
+ }
+
+ public void setIndexName(String indexName) {
+ this.indexName = indexName;
+ }
+
+ public List<String> getColumns() {
+ return columns;
+ }
+
+ public void setColumns(List<String> columns) {
+ this.columns = columns;
+ }
+
+ public IndexDef.IndexType getType() {
+ return type;
+ }
+
+ public void setType(IndexDef.IndexType type) {
+ this.type = type;
+ }
+
+ public String getComment() {
+ return comment;
+ }
+
+ public void setComment(String comment) {
+ this.comment = comment;
+ }
+
+ public JSONObject toJson() {
Review comment:
We can use annotation to finish it in a simple way, you can refer to Tag related class.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request
#2573: implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
yangzhg commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361410182
##########
File path: fe/src/main/java/org/apache/doris/analysis/IndexDef.java
##########
@@ -0,0 +1,168 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.AggregateType;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.PrimitiveType;
+import org.apache.doris.common.AnalysisException;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang.StringUtils;
+
+import java.util.List;
+import java.util.TreeSet;
+
+public class IndexDef {
+ private String indexName;
+ private List<String> columns;
+ private IndexType indexType;
+ private String comment;
+
+ public IndexDef(String indexName, List<String> columns, IndexType indexType, String comment) {
+ this.indexName = indexName;
+ this.columns = columns;
+ if (indexType == null) {
+ this.indexType = IndexType.BITMAP;
+ } else {
+ this.indexType = indexType;
+ }
+ if (columns == null) {
+ this.comment = "";
+ } else {
+ this.comment = comment;
+ }
+ }
+
+ public void analyze(boolean isOlap, List<Column> tableColumns) throws AnalysisException {
+ if (indexType == IndexDef.IndexType.BITMAP) {
+ if (!isOlap) {
+ throw new AnalysisException("bitmap index can only be used with olap engine");
+ }
+ if (CollectionUtils.isEmpty(columns)) {
+ throw new AnalysisException("index definition expect at least one column.");
+ }
+ if (columns.size() != 1) {
+ throw new AnalysisException("bitmap index must apply to one column.");
+ }
+ if (StringUtils.isBlank(indexName)) {
+ throw new AnalysisException("index name cannot be blank.");
+ }
+ if (indexName.length() > 64) {
+ throw new AnalysisException("index name too long.");
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361905201
##########
File path: be/src/olap/tablet_manager.cpp
##########
@@ -1215,7 +1215,17 @@ OLAPStatus TabletManager::_create_inital_rowset(
context.tablet_id = tablet->tablet_id();
context.partition_id = tablet->partition_id();
context.tablet_schema_hash = tablet->schema_hash();
- context.rowset_type = StorageEngine::instance()->default_rowset_type();
+ if (!request.__isset.storage_format || request.storage_format == TStorageFormat::DEFAULT) {
+ context.rowset_type = StorageEngine::instance()->default_rowset_type();
+ } else if (request.storage_format == TStorageFormat::V1){
Review comment:
```suggestion
} else if (request.storage_format == TStorageFormat::V1) {
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361560892
##########
File path: fe/src/main/java/org/apache/doris/analysis/DropIndexClause.java
##########
@@ -0,0 +1,87 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.Index;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.UserException;
+
+import org.apache.commons.collections.CollectionUtils;
+
+import java.util.List;
+import java.util.Map;
+
+public class DropIndexClause extends AlterClause {
+ private final String indexName;
+ private final TableName tableName;
+ private boolean alter;
+
+ public DropIndexClause(String indexName, TableName tableName, boolean alter) {
+ this.indexName = indexName;
+ this.tableName = tableName;
+ this.alter = alter;
+ }
+
+ public String getIndexName() {
+ return indexName;
+ }
+
+ public TableName getTableName() {
+ return tableName;
+ }
+
+ public boolean isAlter() {
+ return alter;
+ }
+
+ @Override
+ public void analyze(Analyzer analyzer) throws UserException {
Review comment:
Why this function is empty?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request
#2573: implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361367578
##########
File path: fe/src/main/java/org/apache/doris/analysis/IndexDef.java
##########
@@ -0,0 +1,168 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.AggregateType;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.PrimitiveType;
+import org.apache.doris.common.AnalysisException;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang.StringUtils;
+
+import java.util.List;
+import java.util.TreeSet;
+
+public class IndexDef {
+ private String indexName;
+ private List<String> columns;
+ private IndexType indexType;
+ private String comment;
+
+ public IndexDef(String indexName, List<String> columns, IndexType indexType, String comment) {
+ this.indexName = indexName;
+ this.columns = columns;
+ if (indexType == null) {
+ this.indexType = IndexType.BITMAP;
+ } else {
+ this.indexType = indexType;
+ }
+ if (columns == null) {
+ this.comment = "";
+ } else {
+ this.comment = comment;
+ }
+ }
+
+ public void analyze(boolean isOlap, List<Column> tableColumns) throws AnalysisException {
+ if (indexType == IndexDef.IndexType.BITMAP) {
+ if (!isOlap) {
+ throw new AnalysisException("bitmap index can only be used with olap engine");
+ }
+ if (CollectionUtils.isEmpty(columns)) {
Review comment:
This check is redundant. `columns.size() != 1` could do this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361552826
##########
File path: be/src/olap/tablet_schema.h
##########
@@ -40,6 +40,7 @@ class TabletColumn {
inline bool is_key() const { return _is_key; }
inline bool is_nullable() const { return _is_nullable; }
inline bool is_bf_column() const { return _is_bf_column; }
+ inline bool is_bitmap_column() const {return _is_bitmap_column; }
Review comment:
Prefer has_bitmap_index(). Becuase Doris has a column type which is bitmap, `is_bitmap_column` is confusing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request
#2573: implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
yangzhg commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361411291
##########
File path: be/src/olap/tablet_manager.cpp
##########
@@ -1215,7 +1215,15 @@ OLAPStatus TabletManager::_create_inital_rowset(
context.tablet_id = tablet->tablet_id();
context.partition_id = tablet->partition_id();
context.tablet_schema_hash = tablet->schema_hash();
- context.rowset_type = StorageEngine::instance()->default_rowset_type();
+ if (!request.__isset.storage_format || request.storage_format == TStorageFormat::DEFAULT) {
+ context.rowset_type = StorageEngine::instance()->default_rowset_type();
+ } else if (request.storage_format == TStorageFormat::V1){
+ context.rowset_type = RowsetTypePB::ALPHA_ROWSET;
+ } else if (request.storage_format == TStorageFormat::V2) {
+ context.rowset_type = RowsetTypePB::BETA_ROWSET;
+ } else {
+ context.rowset_type = StorageEngine::instance()->default_rowset_type();
Review comment:
in case of there has v3 format
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request
#2573: implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361890569
##########
File path: fe/src/main/java/org/apache/doris/catalog/OlapTable.java
##########
@@ -111,6 +112,8 @@
private double bfFpp;
private String colocateGroup;
+
+ private List<Index> indexes;
Review comment:
Plz create a new Class, such as `TableIndexes`, not a bare `List<Index>`, This is not easy to maintain if we want to add some other properties for table's indexes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request
#2573: implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
yangzhg commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361563323
##########
File path: docs/documentation/cn/sql-reference/sql-statements/Data Definition/CREATE TABLE.md
##########
@@ -18,136 +18,163 @@ under the License.
-->
# CREATE TABLE
+
## description
- 该语句用于创建 table。
+
+该语句用于创建 table。
+语法：
+
+```
+ CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [database.]table_name
+ (column_definition1[, column_definition2, ...]
+ [, index_definition1[, ndex_definition12,]])
+ [ENGINE = [olap|mysql|broker]]
+ [key_desc]
+ [COMMENT "table comment"];
+ [partition_desc]
+ [distribution_desc]
+ [PROPERTIES ("key"="value", ...)]
+ [BROKER PROPERTIES ("key"="value", ...)]
+```
+
+1. column_definition
语法：
- CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [database.]table_name
- (column_definition1[, column_definition2, ...])
- [ENGINE = [olap|mysql|broker]]
- [key_desc]
- [COMMENT "table comment"];
- [partition_desc]
- [distribution_desc]
- [PROPERTIES ("key"="value", ...)]
- [BROKER PROPERTIES ("key"="value", ...)]
-
- 1. column_definition
- 语法：
- col_name col_type [agg_type] [NULL | NOT NULL] [DEFAULT "default_value"]
-
- 说明：
- col_name：列名称
- col_type：列类型
- TINYINT（1字节）
- 范围：-2^7 + 1 ~ 2^7 - 1
- SMALLINT（2字节）
- 范围：-2^15 + 1 ~ 2^15 - 1
- INT（4字节）
- 范围：-2^31 + 1 ~ 2^31 - 1
- BIGINT（8字节）
- 范围：-2^63 + 1 ~ 2^63 - 1
- LARGEINT（16字节）
- 范围：-2^127 + 1 ~ 2^127 - 1
- FLOAT（4字节）
- 支持科学计数法
- DOUBLE（12字节）
- 支持科学计数法
- DECIMAL[(precision, scale)] (16字节)
- 保证精度的小数类型。默认是 DECIMAL(10, 0)
- precision: 1 ~ 27
- scale: 0 ~ 9
- 其中整数部分为 1 ~ 18
- 不支持科学计数法
- DATE（3字节）
- 范围：1900-01-01 ~ 9999-12-31
- DATETIME（8字节）
- 范围：1900-01-01 00:00:00 ~ 9999-12-31 23:59:59
- CHAR[(length)]
- 定长字符串。长度范围：1 ~ 255。默认为1
- VARCHAR[(length)]
- 变长字符串。长度范围：1 ~ 65533
- HLL (1~16385个字节)
- hll列类型，不需要指定长度和默认值、长度根据数据的聚合
- 程度系统内控制，并且HLL列只能通过配套的hll_union_agg、Hll_cardinality、hll_hash进行查询或使用
- BITMAP
- bitmap 列类型，不需要指定长度和默认值
- BITMAP 列只能通过配套的 BITMAP_UNION、BITMAP_COUNT、TO_BITMAP 进行查询或使用
-
- agg_type：聚合类型，如果不指定，则该列为 key 列。否则，该列为 value 列
-
- * SUM、MAX、MIN、REPLACE
- * HLL_UNION(仅用于HLL列，为HLL独有的聚合方式)、
- * BITMAP_UNION(仅用于 BITMAP 列，为 BITMAP 独有的聚合方式)、
- * REPLACE_IF_NOT_NULL：这个聚合类型的含义是当且仅当新导入数据是非NULL值时才会发生替换行为，如果新导入的数据是NULL，那么Doris仍然会保留原值。注意：如果用户在建表时REPLACE_IF_NOT_NULL列指定了NOT NULL，那么Doris仍然会将其转化为NULL，不会向用户报错。用户可以借助这个类型完成部分列导入的功能。
- *该类型只对聚合模型(key_desc的type为AGGREGATE KEY)有用，其它模型不需要指定这个。
-
- 是否允许为NULL: 默认不允许为 NULL。NULL 值在导入数据中用 \N 来表示
-
- 注意：
- BITMAP_UNION聚合类型列在导入时的原始数据类型必须是TINYINT,SMALLINT,INT。
-
- 2. ENGINE 类型
- 默认为 olap。可选 mysql, broker
- 1) 如果是 mysql，则需要在 properties 提供以下信息：
-
- PROPERTIES (
- "host" = "mysql_server_host",
- "port" = "mysql_server_port",
- "user" = "your_user_name",
- "password" = "your_password",
- "database" = "database_name",
- "table" = "table_name"
- )
-
- 注意：
- "table" 条目中的 "table_name" 是 mysql 中的真实表名。
- 而 CREATE TABLE 语句中的 table_name 是该 mysql 表在 Palo 中的名字，可以不同。
-
- 在 Palo 创建 mysql 表的目的是可以通过 Palo 访问 mysql 数据库。
- 而 Palo 本身并不维护、存储任何 mysql 数据。
- 2) 如果是 broker，表示表的访问需要通过指定的broker, 需要在 properties 提供以下信息：
- PROPERTIES (
- "broker_name" = "broker_name",
- "path" = "file_path1[,file_path2]",
- "column_separator" = "value_separator"
- "line_delimiter" = "value_delimiter"
- )
- 另外还需要提供Broker需要的Property信息，通过BROKER PROPERTIES来传递，例如HDFS需要传入
- BROKER PROPERTIES(
- "username" = "name",
- "password" = "password"
- )
- 这个根据不同的Broker类型，需要传入的内容也不相同
- 注意：
- "path" 中如果有多个文件，用逗号[,]分割。如果文件名中包含逗号，那么使用 %2c 来替代。如果文件名中包含 %，使用 %25 代替
- 现在文件内容格式支持CSV，支持GZ，BZ2，LZ4，LZO(LZOP) 压缩格式。
+ `col_name col_type [agg_type] [NULL | NOT NULL] [DEFAULT "default_value"]`
+
+ 说明：
+ col_name：列名称
+ col_type：列类型
+
+ ```
+ TINYINT（1字节）
+ 范围：-2^7 + 1 ~ 2^7 - 1
+ SMALLINT（2字节）
+ 范围：-2^15 + 1 ~ 2^15 - 1
+ INT（4字节）
+ 范围：-2^31 + 1 ~ 2^31 - 1
+ BIGINT（8字节）
+ 范围：-2^63 + 1 ~ 2^63 - 1
+ LARGEINT（16字节）
+ 范围：-2^127 + 1 ~ 2^127 - 1
+ FLOAT（4字节）
+ 支持科学计数法
+ DOUBLE（12字节）
+ 支持科学计数法
+ DECIMAL[(precision, scale)] (16字节)
+ 保证精度的小数类型。默认是 DECIMAL(10, 0)
+ precision: 1 ~ 27
+ scale: 0 ~ 9
+ 其中整数部分为 1 ~ 18
+ 不支持科学计数法
+ DATE（3字节）
+ 范围：1900-01-01 ~ 9999-12-31
+ DATETIME（8字节）
+ 范围：1900-01-01 00:00:00 ~ 9999-12-31 23:59:59
+ CHAR[(length)]
+ 定长字符串。长度范围：1 ~ 255。默认为1
+ VARCHAR[(length)]
+ 变长字符串。长度范围：1 ~ 65533
+ HLL (1~16385个字节)
+ hll列类型，不需要指定长度和默认值、长度根据数据的聚合
+ 程度系统内控制，并且HLL列只能通过配套的hll_union_agg、Hll_cardinality、hll_hash进行查询或使用
+ BITMAP
+ bitmap 列类型，不需要指定长度和默认值
+ BITMAP 列只能通过配套的 BITMAP_UNION、BITMAP_COUNT、TO_BITMAP 进行查询或使用
+ ```
+
+ agg_type：聚合类型，如果不指定，则该列为 key 列。否则，该列为 value 列
+ * SUM、MAX、MIN、REPLACE
+ * HLL_UNION(仅用于HLL列，为HLL独有的聚合方式)、
+ * BITMAP_UNION(仅用于 BITMAP 列，为 BITMAP 独有的聚合方式)、
+ * REPLACE_IF_NOT_NULL：这个聚合类型的含义是当且仅当新导入数据是非NULL值时会发生替换行为，如果新导入的数据是NULL，那么Doris仍然会保留原值。注意：如果用在建表时REPLACE_IF_NOT_NULL列指定了NOT NULL，那么Doris仍然会将其转化NULL，不会向用户报错。用户可以借助这个类型完成部分列导入的功能。
+ * 该类型只对聚合模型(key_desc的type为AGGREGATE KEY)有用，其它模型不需要指这个。
+
+ 是否允许为NULL: 默认不允许为 NULL。NULL 值在导入数据中用 \N 来表示
+
+ 注意：
+ BITMAP_UNION聚合类型列在导入时的原始数据类型必须是TINYINT,SMALLINT,INT。
+
+2. index_definition
+ 语法：
+ `INDEX index_name [USING BITMAP] (col_name[, col_name, ...]) COMMENT 'xxxxxx'`
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361560723
##########
File path: fe/src/main/java/org/apache/doris/analysis/IndexDef.java
##########
@@ -0,0 +1,165 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.AggregateType;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.PrimitiveType;
+import org.apache.doris.common.AnalysisException;
+
+import org.apache.commons.lang.StringUtils;
+
+import java.util.List;
+import java.util.TreeSet;
+
+public class IndexDef {
+ private String indexName;
+ private List<String> columns;
+ private IndexType indexType;
+ private String comment;
+
+ public IndexDef(String indexName, List<String> columns, IndexType indexType, String comment) {
+ this.indexName = indexName;
+ this.columns = columns;
+ if (indexType == null) {
+ this.indexType = IndexType.BITMAP;
+ } else {
+ this.indexType = indexType;
+ }
+ if (columns == null) {
+ this.comment = "";
+ } else {
+ this.comment = comment;
+ }
+ }
+
+ public void analyze(boolean isOlap, List<Column> tableColumns) throws AnalysisException {
+ if (indexType == IndexDef.IndexType.BITMAP) {
+ if (!isOlap) {
+ throw new AnalysisException("bitmap index can only be used with olap engine");
+ }
+ if (columns == null || columns.size() != 1) {
+ throw new AnalysisException("bitmap index definition expect at least one column.");
+ }
+ if (StringUtils.isBlank(indexName)) {
+ throw new AnalysisException("index name cannot be blank.");
+ }
+ if (indexName.length() > 64) {
+ throw new AnalysisException("index name too long, the index name length at most is 64.");
+ }
+ TreeSet<String> distinct = new TreeSet<>(String.CASE_INSENSITIVE_ORDER);
Review comment:
columns size must be 1, so it seems unnecessary.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang commented on a change in pull
request #2573: implements create drop show index syntax for bitmap index
[#2487]
Posted by GitBox <gi...@apache.org>.
kangpinghuang commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361602484
##########
File path: fe/src/main/cup/sql_parser.cup
##########
@@ -1411,6 +1439,10 @@ drop_stmt ::=
{:
RESULT = new DropFileStmt(fileName, dbName, properties);
:}
+ | KW_DROP KW_INDEX ident:indexName KW_ON table_name:tableName
+ {:
+ RESULT = new AlterTableStmt(tableName, Lists.newArrayList(new DropIndexClause(indexName, tableName, false)));
Review comment:
```suggestion
RESULT = new AlterTableStmt(tableName, Lists.newArrayList(new DropIndexClause(indexName, tableName, false)));
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r362321518
##########
File path: fe/src/main/java/org/apache/doris/alter/Alter.java
##########
@@ -169,9 +176,50 @@ public void processAlterTable(AlterTableStmt stmt) throws UserException {
|| alterClause instanceof AddColumnsClause
|| alterClause instanceof DropColumnClause
|| alterClause instanceof ModifyColumnClause
- || alterClause instanceof ReorderColumnsClause)
+ || alterClause instanceof ReorderColumnsClause
+ || alterClause instanceof CreateIndexClause
+ || alterClause instanceof DropIndexClause)
&& !hasAddMaterializedView && !hasDropRollup && !hasPartition && !hasRename) {
hasSchemaChange = true;
+ if (alterClause instanceof CreateIndexClause) {
+ Table table = db.getTable(dbTableName.getTbl());
+ if (table instanceof OlapTable) {
+ List<Index> indexes = ((OlapTable) table).getIndexes();
+ IndexDef indexDef = ((CreateIndexClause) alterClause).getIndexDef();
+ Set<String> newColset = new TreeSet<>(String.CASE_INSENSITIVE_ORDER);
+ newColset.addAll(indexDef.getColumns());
+ for (Index idx : indexes) {
+ if (idx.getIndexName().equalsIgnoreCase(indexDef.getIndexName())) {
+ throw new AnalysisException("index `" + indexDef.getIndexName() + "` already exist.");
+ }
+ Set<String> idxSet = new TreeSet<>(String.CASE_INSENSITIVE_ORDER);
+ idxSet.addAll(idx.getColumns());
+ if (newColset.equals(idxSet)) {
+ throw new AnalysisException("index for columns (" + String
+ .join(",", indexDef.getColumns()) + " ) already exist.");
+ }
+ }
+ }
+ } else if (alterClause instanceof DropIndexClause) {
+ Table table = db.getTable(dbTableName.getTbl());
+ if (table instanceof OlapTable) {
+ String indexName = ((DropIndexClause) alterClause).getIndexName();
+ List<Index> indexes = ((OlapTable) table).getIndexes();
+ if (CollectionUtils.isEmpty(indexes)) {
Review comment:
seems this is useless, following logic can handle this situation
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request
#2573: implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
yangzhg commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361563292
##########
File path: be/src/olap/tablet_meta.cpp
##########
@@ -152,6 +154,17 @@ TabletMeta::TabletMeta(int64_t table_id, int64_t partition_id,
column->set_is_bf_column(tcolumn.is_bloom_filter_column);
has_bf_columns = true;
}
+ if (tablet_schema.__isset.indexes) {
+ for (TOlapTableIndex index : tablet_schema.indexes) {
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request
#2573: implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361367995
##########
File path: fe/src/main/java/org/apache/doris/catalog/Index.java
##########
@@ -0,0 +1,157 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.catalog;
+
+import org.apache.doris.analysis.IndexDef;
+import org.apache.doris.common.io.Text;
+import org.apache.doris.common.io.Writable;
+import org.apache.doris.thrift.TIndexType;
+import org.apache.doris.thrift.TOlapTableIndex;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+public class Index implements Writable {
+ private String indexName;
+ private List<String> columns;
+ private IndexDef.IndexType type;
+ private String comment;
+
+ public Index(String indexName, List<String> columns, IndexDef.IndexType type, String comment) {
+ this.indexName = indexName;
+ this.columns = columns;
+ this.type = type;
+ this.comment = comment;
+ }
+
+ public static Index read(DataInput in) throws IOException {
Review comment:
New metadata class please use JSON to Serde
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang commented on a change in pull
request #2573: implements create drop show index syntax for bitmap index
[#2487]
Posted by GitBox <gi...@apache.org>.
kangpinghuang commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361580179
##########
File path: be/src/olap/olap_snapshot_converter.cpp
##########
@@ -326,6 +326,9 @@ OLAPStatus OlapSnapshotConverter::to_column_pb(const ColumnMessage& column_msg,
if (column_msg.has_is_bf_column()) {
column_pb->set_is_bf_column(column_msg.is_bf_column());
}
+ if (column_msg.has_is_bitmap_column()) {
Review comment:
it is not necessary to modify here because OlapSnapshotConverter is for upgrading from 0.10 to 0.11. and it will be removed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361557016
##########
File path: docs/documentation/cn/sql-reference/sql-statements/Data Definition/CREATE TABLE.md
##########
@@ -18,136 +18,163 @@ under the License.
-->
# CREATE TABLE
+
## description
- 该语句用于创建 table。
+
+该语句用于创建 table。
+语法：
+
+```
+ CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [database.]table_name
+ (column_definition1[, column_definition2, ...]
+ [, index_definition1[, ndex_definition12,]])
+ [ENGINE = [olap|mysql|broker]]
+ [key_desc]
+ [COMMENT "table comment"];
+ [partition_desc]
+ [distribution_desc]
+ [PROPERTIES ("key"="value", ...)]
+ [BROKER PROPERTIES ("key"="value", ...)]
+```
+
+1. column_definition
语法：
- CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [database.]table_name
- (column_definition1[, column_definition2, ...])
- [ENGINE = [olap|mysql|broker]]
- [key_desc]
- [COMMENT "table comment"];
- [partition_desc]
- [distribution_desc]
- [PROPERTIES ("key"="value", ...)]
- [BROKER PROPERTIES ("key"="value", ...)]
-
- 1. column_definition
- 语法：
- col_name col_type [agg_type] [NULL | NOT NULL] [DEFAULT "default_value"]
-
- 说明：
- col_name：列名称
- col_type：列类型
- TINYINT（1字节）
- 范围：-2^7 + 1 ~ 2^7 - 1
- SMALLINT（2字节）
- 范围：-2^15 + 1 ~ 2^15 - 1
- INT（4字节）
- 范围：-2^31 + 1 ~ 2^31 - 1
- BIGINT（8字节）
- 范围：-2^63 + 1 ~ 2^63 - 1
- LARGEINT（16字节）
- 范围：-2^127 + 1 ~ 2^127 - 1
- FLOAT（4字节）
- 支持科学计数法
- DOUBLE（12字节）
- 支持科学计数法
- DECIMAL[(precision, scale)] (16字节)
- 保证精度的小数类型。默认是 DECIMAL(10, 0)
- precision: 1 ~ 27
- scale: 0 ~ 9
- 其中整数部分为 1 ~ 18
- 不支持科学计数法
- DATE（3字节）
- 范围：1900-01-01 ~ 9999-12-31
- DATETIME（8字节）
- 范围：1900-01-01 00:00:00 ~ 9999-12-31 23:59:59
- CHAR[(length)]
- 定长字符串。长度范围：1 ~ 255。默认为1
- VARCHAR[(length)]
- 变长字符串。长度范围：1 ~ 65533
- HLL (1~16385个字节)
- hll列类型，不需要指定长度和默认值、长度根据数据的聚合
- 程度系统内控制，并且HLL列只能通过配套的hll_union_agg、Hll_cardinality、hll_hash进行查询或使用
- BITMAP
- bitmap 列类型，不需要指定长度和默认值
- BITMAP 列只能通过配套的 BITMAP_UNION、BITMAP_COUNT、TO_BITMAP 进行查询或使用
-
- agg_type：聚合类型，如果不指定，则该列为 key 列。否则，该列为 value 列
-
- * SUM、MAX、MIN、REPLACE
- * HLL_UNION(仅用于HLL列，为HLL独有的聚合方式)、
- * BITMAP_UNION(仅用于 BITMAP 列，为 BITMAP 独有的聚合方式)、
- * REPLACE_IF_NOT_NULL：这个聚合类型的含义是当且仅当新导入数据是非NULL值时才会发生替换行为，如果新导入的数据是NULL，那么Doris仍然会保留原值。注意：如果用户在建表时REPLACE_IF_NOT_NULL列指定了NOT NULL，那么Doris仍然会将其转化为NULL，不会向用户报错。用户可以借助这个类型完成部分列导入的功能。
- *该类型只对聚合模型(key_desc的type为AGGREGATE KEY)有用，其它模型不需要指定这个。
-
- 是否允许为NULL: 默认不允许为 NULL。NULL 值在导入数据中用 \N 来表示
-
- 注意：
- BITMAP_UNION聚合类型列在导入时的原始数据类型必须是TINYINT,SMALLINT,INT。
-
- 2. ENGINE 类型
- 默认为 olap。可选 mysql, broker
- 1) 如果是 mysql，则需要在 properties 提供以下信息：
-
- PROPERTIES (
- "host" = "mysql_server_host",
- "port" = "mysql_server_port",
- "user" = "your_user_name",
- "password" = "your_password",
- "database" = "database_name",
- "table" = "table_name"
- )
-
- 注意：
- "table" 条目中的 "table_name" 是 mysql 中的真实表名。
- 而 CREATE TABLE 语句中的 table_name 是该 mysql 表在 Palo 中的名字，可以不同。
-
- 在 Palo 创建 mysql 表的目的是可以通过 Palo 访问 mysql 数据库。
- 而 Palo 本身并不维护、存储任何 mysql 数据。
- 2) 如果是 broker，表示表的访问需要通过指定的broker, 需要在 properties 提供以下信息：
- PROPERTIES (
- "broker_name" = "broker_name",
- "path" = "file_path1[,file_path2]",
- "column_separator" = "value_separator"
- "line_delimiter" = "value_delimiter"
- )
- 另外还需要提供Broker需要的Property信息，通过BROKER PROPERTIES来传递，例如HDFS需要传入
- BROKER PROPERTIES(
- "username" = "name",
- "password" = "password"
- )
- 这个根据不同的Broker类型，需要传入的内容也不相同
- 注意：
- "path" 中如果有多个文件，用逗号[,]分割。如果文件名中包含逗号，那么使用 %2c 来替代。如果文件名中包含 %，使用 %25 代替
- 现在文件内容格式支持CSV，支持GZ，BZ2，LZ4，LZO(LZOP) 压缩格式。
+ `col_name col_type [agg_type] [NULL | NOT NULL] [DEFAULT "default_value"]`
+
+ 说明：
+ col_name：列名称
+ col_type：列类型
+
+ ```
+ TINYINT（1字节）
+ 范围：-2^7 + 1 ~ 2^7 - 1
+ SMALLINT（2字节）
+ 范围：-2^15 + 1 ~ 2^15 - 1
+ INT（4字节）
+ 范围：-2^31 + 1 ~ 2^31 - 1
+ BIGINT（8字节）
+ 范围：-2^63 + 1 ~ 2^63 - 1
+ LARGEINT（16字节）
+ 范围：-2^127 + 1 ~ 2^127 - 1
+ FLOAT（4字节）
+ 支持科学计数法
+ DOUBLE（12字节）
+ 支持科学计数法
+ DECIMAL[(precision, scale)] (16字节)
+ 保证精度的小数类型。默认是 DECIMAL(10, 0)
+ precision: 1 ~ 27
+ scale: 0 ~ 9
+ 其中整数部分为 1 ~ 18
+ 不支持科学计数法
+ DATE（3字节）
+ 范围：1900-01-01 ~ 9999-12-31
+ DATETIME（8字节）
+ 范围：1900-01-01 00:00:00 ~ 9999-12-31 23:59:59
+ CHAR[(length)]
+ 定长字符串。长度范围：1 ~ 255。默认为1
+ VARCHAR[(length)]
+ 变长字符串。长度范围：1 ~ 65533
+ HLL (1~16385个字节)
+ hll列类型，不需要指定长度和默认值、长度根据数据的聚合
+ 程度系统内控制，并且HLL列只能通过配套的hll_union_agg、Hll_cardinality、hll_hash进行查询或使用
+ BITMAP
+ bitmap 列类型，不需要指定长度和默认值
+ BITMAP 列只能通过配套的 BITMAP_UNION、BITMAP_COUNT、TO_BITMAP 进行查询或使用
+ ```
+
+ agg_type：聚合类型，如果不指定，则该列为 key 列。否则，该列为 value 列
+ * SUM、MAX、MIN、REPLACE
+ * HLL_UNION(仅用于HLL列，为HLL独有的聚合方式)、
+ * BITMAP_UNION(仅用于 BITMAP 列，为 BITMAP 独有的聚合方式)、
+ * REPLACE_IF_NOT_NULL：这个聚合类型的含义是当且仅当新导入数据是非NULL值时会发生替换行为，如果新导入的数据是NULL，那么Doris仍然会保留原值。注意：如果用在建表时REPLACE_IF_NOT_NULL列指定了NOT NULL，那么Doris仍然会将其转化NULL，不会向用户报错。用户可以借助这个类型完成部分列导入的功能。
+ * 该类型只对聚合模型(key_desc的type为AGGREGATE KEY)有用，其它模型不需要指这个。
+
+ 是否允许为NULL: 默认不允许为 NULL。NULL 值在导入数据中用 \N 来表示
+
+ 注意：
+ BITMAP_UNION聚合类型列在导入时的原始数据类型必须是TINYINT,SMALLINT,INT。
+
+2. index_definition
+ 语法：
+ `INDEX index_name [USING BITMAP] (col_name[, col_name, ...]) COMMENT 'xxxxxx'`
Review comment:
I think should keep it compatible with create index syntax
```
INDEX index_name (col_name[, col_name, ...]) [USING BITMAP] COMMENT 'xxxxxx'
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang commented on a change in pull
request #2573: implements create drop show index syntax for bitmap index
[#2487]
Posted by GitBox <gi...@apache.org>.
kangpinghuang commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361604852
##########
File path: fe/src/main/java/org/apache/doris/analysis/AlterTableStmt.java
##########
@@ -61,7 +66,41 @@ public void analyze(Analyzer analyzer) throws AnalysisException, UserException {
ErrorReport.reportAnalysisException(ErrorCode.ERR_NO_ALTER_OPERATION);
}
for (AlterClause op : ops) {
- op.analyze(analyzer);
+ if (op instanceof CreateIndexClause) {
+ Table table = Catalog.getInstance().getDb(tbl.getDb()).getTable(tbl.getTbl());
+ if (table instanceof OlapTable) {
+ op.analyze(analyzer);
+ List<Index> indexes = ((OlapTable) table).getIndexes();
+ IndexDef indexDef = ((CreateIndexClause) op).getIndexDef();
+ for (Index idx : indexes) {
+ if (idx.getIndexName().equalsIgnoreCase(indexDef.getIndexName())) {
Review comment:
what if user add two bitmap indexes with different names on the same column?
I think we should also check the column name and index type.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request
#2573: implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361367198
##########
File path: fe/src/main/java/org/apache/doris/alter/AlterJobV2.java
##########
@@ -107,8 +107,15 @@ public String getTableName() {
return tableName;
}
+ // compatible with V1
private boolean isTimeout() {
- return System.currentTimeMillis() - createTimeMs > timeoutMs;
Review comment:
Why need this compatibility?
I don't think no timeout is reasonable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request
#2573: implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
yangzhg commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361409976
##########
File path: fe/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java
##########
@@ -838,6 +854,20 @@ public void write(DataOutput out) throws IOException {
}
out.writeLong(watershedTxnId);
+
+ // index
+ if (FeMetaVersion.VERSION_CURRENT >= FeMetaVersion.VERSION_70) {
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request
#2573: implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
yangzhg commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361563282
##########
File path: gensrc/thrift/Descriptors.thrift
##########
@@ -166,6 +170,13 @@ struct TOlapTableSchemaParam {
6: required list<TOlapTableIndexSchema> indexes
}
+struct TOlapTableIndex {
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361904848
##########
File path: be/src/olap/olap_snapshot_converter.cpp
##########
@@ -326,6 +326,9 @@ OLAPStatus OlapSnapshotConverter::to_column_pb(const ColumnMessage& column_msg,
if (column_msg.has_is_bf_column()) {
column_pb->set_is_bf_column(column_msg.is_bf_column());
}
+ if (column_msg.has_is_bitmap_column()) {
Review comment:
better rename it to is_bitmap_index()
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request
#2573: implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361889726
##########
File path: fe/src/main/java/org/apache/doris/analysis/AlterTableStmt.java
##########
@@ -61,7 +68,49 @@ public void analyze(Analyzer analyzer) throws AnalysisException, UserException {
ErrorReport.reportAnalysisException(ErrorCode.ERR_NO_ALTER_OPERATION);
}
for (AlterClause op : ops) {
- op.analyze(analyzer);
+ if (op instanceof CreateIndexClause) {
+ Table table = Catalog.getInstance().getDb(tbl.getDb()).getTable(tbl.getTbl());
+ if (table instanceof OlapTable) {
+ op.analyze(analyzer);
+ List<Index> indexes = ((OlapTable) table).getIndexes();
+ IndexDef indexDef = ((CreateIndexClause) op).getIndexDef();
+ Set<String> newColset = new TreeSet<>(String.CASE_INSENSITIVE_ORDER);
+ newColset.addAll(indexDef.getColumns());
+ for (Index idx : indexes) {
+ if (idx.getIndexName().equalsIgnoreCase(indexDef.getIndexName())) {
+ throw new AnalysisException("index `" + indexDef.getIndexName() + "` already exist.");
+ }
+ Set<String> idxSet = new TreeSet<>(String.CASE_INSENSITIVE_ORDER);
+ idxSet.addAll(idx.getColumns());
+ if (newColset.equals(idxSet)) {
+ throw new AnalysisException("index for columns (" + String
+ .join(",", indexDef.getColumns()) + " ) already exist.");
+ }
+ }
+ }
+ } else if (op instanceof DropIndexClause) {
+ Table table = Catalog.getInstance().getDb(tbl.getDb()).getTable(tbl.getTbl());
+ if (table instanceof OlapTable) {
+ op.analyze(analyzer);
+ String indexName = ((DropIndexClause) op).getIndexName();
+ List<Index> indexes = ((OlapTable) table).getIndexes();
+ if (CollectionUtils.isEmpty(indexes)) {
+ throw new AnalysisException("index " + indexName + " does not exist");
+ }
+ Index found = null;
+ for (Index idx: indexes) {
+ if (idx.getIndexName().equalsIgnoreCase(indexName)) {
+ found = idx;
+ break;
+ }
+ }
+ if (found == null) {
+ throw new AnalysisException("index " + indexName + " does not exist");
+ }
+ }
+ } else {
+ op.analyze(analyzer);
+ }
}
if (!Catalog.getCurrentCatalog().getAuth().checkTblPriv(ConnectContext.get(), tbl.getDb(), tbl.getTbl(),
Review comment:
Move this privilege check at the frontend of `analyze()`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r362321239
##########
File path: run-ut.sh
##########
@@ -285,7 +285,7 @@ ${DORIS_TEST_BINARY_DIR}/olap/rowset/segment_v2/bloom_filter_index_reader_writer
${DORIS_TEST_BINARY_DIR}/olap/txn_manager_test
${DORIS_TEST_BINARY_DIR}/olap/storage_types_test
${DORIS_TEST_BINARY_DIR}/olap/generic_iterators_test
-${DORIS_TEST_BINARY_DIR}/olap/aggregate_func_test
+# ${DORIS_TEST_BINARY_DIR}/olap/aggregate_func_test
Review comment:
why disable this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request
#2573: implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361367850
##########
File path: fe/src/main/java/org/apache/doris/analysis/IndexDef.java
##########
@@ -0,0 +1,168 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.AggregateType;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.PrimitiveType;
+import org.apache.doris.common.AnalysisException;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang.StringUtils;
+
+import java.util.List;
+import java.util.TreeSet;
+
+public class IndexDef {
+ private String indexName;
+ private List<String> columns;
+ private IndexType indexType;
+ private String comment;
+
+ public IndexDef(String indexName, List<String> columns, IndexType indexType, String comment) {
+ this.indexName = indexName;
+ this.columns = columns;
+ if (indexType == null) {
+ this.indexType = IndexType.BITMAP;
+ } else {
+ this.indexType = indexType;
+ }
+ if (columns == null) {
+ this.comment = "";
+ } else {
+ this.comment = comment;
+ }
+ }
+
+ public void analyze(boolean isOlap, List<Column> tableColumns) throws AnalysisException {
+ if (indexType == IndexDef.IndexType.BITMAP) {
+ if (!isOlap) {
+ throw new AnalysisException("bitmap index can only be used with olap engine");
+ }
+ if (CollectionUtils.isEmpty(columns)) {
+ throw new AnalysisException("index definition expect at least one column.");
+ }
+ if (columns.size() != 1) {
+ throw new AnalysisException("bitmap index must apply to one column.");
+ }
+ if (StringUtils.isBlank(indexName)) {
+ throw new AnalysisException("index name cannot be blank.");
+ }
+ if (indexName.length() > 64) {
+ throw new AnalysisException("index name too long.");
+ }
+ TreeSet<String> distinct = new TreeSet<>(String.CASE_INSENSITIVE_ORDER);
+ distinct.addAll(columns);
+ if (columns.size() != distinct.size()) {
+ throw new AnalysisException("columns of index has duplicated.");
+ }
+ for (String idColumn : columns) {
+ idColumn = idColumn.trim();
+ boolean found = false;
+ for (Column column : tableColumns) {
+ if (column.getName().equalsIgnoreCase(idColumn)) {
+ idColumn = column.getName();
Review comment:
`idColumn ` please rename.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361474914
##########
File path: be/src/olap/tablet_meta.cpp
##########
@@ -152,6 +154,17 @@ TabletMeta::TabletMeta(int64_t table_id, int64_t partition_id,
column->set_is_bf_column(tcolumn.is_bloom_filter_column);
has_bf_columns = true;
}
+ if (tablet_schema.__isset.indexes) {
+ for (TOlapTableIndex index : tablet_schema.indexes) {
Review comment:
```suggestion
for (auto& index : tablet_schema.indexes) {
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361905955
##########
File path: fe/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
##########
@@ -362,6 +392,68 @@ public void analyze(Analyzer analyzer) throws AnalysisException, UserException {
}
columns.add(col);
}
+
+ if (CollectionUtils.isNotEmpty(indexDefs)) {
+ Set<String> distinct = new TreeSet<>(String.CASE_INSENSITIVE_ORDER);
+ Set<List<String>> distinctCol = new HashSet<>();
+
+ for (IndexDef indexDef : indexDefs) {
+ indexDef.analyze();
+ if (engineName.equalsIgnoreCase("olap")) {
Review comment:
```suggestion
if (!engineName.equalsIgnoreCase("olap")) {
throw new Exception;
}
```
which can make code less indentation
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request
#2573: implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361889102
##########
File path: fe/src/main/java/org/apache/doris/analysis/IndexDef.java
##########
@@ -0,0 +1,121 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.common.AnalysisException;
+
+import org.apache.commons.lang.StringUtils;
+
+import java.util.List;
+import java.util.TreeSet;
+
+public class IndexDef {
+ private String indexName;
+ private List<String> columns;
+ private IndexType indexType;
+ private String comment;
+
+ public IndexDef(String indexName, List<String> columns, IndexType indexType, String comment) {
+ this.indexName = indexName;
+ this.columns = columns;
+ if (indexType == null) {
+ this.indexType = IndexType.BITMAP;
+ } else {
+ this.indexType = indexType;
+ }
+ if (columns == null) {
+ this.comment = "";
+ } else {
+ this.comment = comment;
+ }
+ }
+
+ public void analyze() throws AnalysisException {
+ if (indexType == IndexDef.IndexType.BITMAP) {
+ if (columns == null || columns.size() != 1) {
+ throw new AnalysisException("bitmap index definition expect at least one column.");
+ }
+ if (StringUtils.isBlank(indexName)) {
Review comment:
We use Strings.isNullOrEmpty() in common.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361906126
##########
File path: fe/src/main/java/org/apache/doris/catalog/Index.java
##########
@@ -0,0 +1,161 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.catalog;
+
+import org.apache.doris.analysis.IndexDef;
+import org.apache.doris.common.io.Text;
+import org.apache.doris.common.io.Writable;
+import org.apache.doris.thrift.TIndexType;
+import org.apache.doris.thrift.TOlapTableIndex;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Internal representation of index, including index type, name, columns and comments.
+ * This class will used in olaptable
+ */
+public class Index implements Writable {
+ private String indexName;
+ private List<String> columns;
+ private IndexDef.IndexType type;
+ private String comment;
+
+ public Index(String indexName, List<String> columns, IndexDef.IndexType type, String comment) {
+ this.indexName = indexName;
+ this.columns = columns;
+ this.type = type;
+ this.comment = comment;
+ }
+
+ public static Index read(DataInput in) throws IOException {
+ String name = Text.readString(in);
+ List<String> cols = new ArrayList<>();
+ int columnCount = in.readInt();
+ for (int i = 0; i < columnCount; i++) {
+ cols.add(Text.readString(in));
+ }
+ IndexDef.IndexType type = IndexDef.IndexType.valueOf(Text.readString(in));
+ boolean hasComment = in.readBoolean();
+ String comment = null;
+ if (hasComment) {
+ comment = Text.readString(in);
+ }
+ return new Index(name, cols, type, comment);
+ }
+
+ @Override
+ public int hashCode() {
+ return 31 * (indexName.hashCode() + columns.hashCode() + type.hashCode());
+ }
+
+ public Index clone() {
+ return new Index(indexName, new ArrayList<>(columns), type, comment);
+ }
+
+ public String getIndexName() {
+ return indexName;
+ }
+
+ public void setIndexName(String indexName) {
+ this.indexName = indexName;
+ }
+
+ public List<String> getColumns() {
+ return columns;
+ }
+
+ public void setColumns(List<String> columns) {
+ this.columns = columns;
+ }
+
+ public IndexDef.IndexType getType() {
+ return type;
+ }
+
+ public void setType(IndexDef.IndexType type) {
+ this.type = type;
+ }
+
+ public String getComment() {
+ return comment;
+ }
+
+ public void setComment(String comment) {
+ this.comment = comment;
+ }
+
+ @Override
+ public void write(DataOutput out) throws IOException {
Review comment:
serialize in json format
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2573:
implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r362231170
##########
File path: fe/src/main/cup/sql_parser.cup
##########
@@ -1666,7 +1698,10 @@ opt_ext_properties ::=
;
column_definition_list ::=
- column_definition:column
+ {:
+ RESULT = null;
+ :}
+ | column_definition:column
Review comment:
why this could be null?
If this would be null, better to introduce another opt_column_definition_list
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang commented on a change in pull
request #2573: implements create drop show index syntax for bitmap index
[#2487]
Posted by GitBox <gi...@apache.org>.
kangpinghuang commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r361606537
##########
File path: fe/src/main/java/org/apache/doris/catalog/OlapTable.java
##########
@@ -139,10 +142,17 @@ public OlapTable() {
this.bfFpp = 0;
this.colocateGroup = null;
+
+ this.indexes = null;
+ }
+
+ public OlapTable(long id, String tableName, List<Column> baseSchema, KeysType keysType,
+ PartitionInfo partitionInfo, DistributionInfo defaultDistributionInfo) {
+ this(id, tableName, baseSchema, keysType, partitionInfo, defaultDistributionInfo, new ArrayList<>());
}
- public OlapTable(long id, String tableName, List<Column> baseSchema,
- KeysType keysType, PartitionInfo partitionInfo, DistributionInfo defaultDistributionInfo) {
+ public OlapTable(long id, String tableName, List<Column> baseSchema, KeysType keysType,
+ PartitionInfo partitionInfo, DistributionInfo defaultDistributionInfo, List<Index> indexes) {
Review comment:
```suggestion
PartitionInfo partitionInfo, DistributionInfo defaultDistributionInfo, List<Index> indexes) {
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request
#2573: implements create drop show index syntax for bitmap index [#2487]
Posted by GitBox <gi...@apache.org>.
yangzhg commented on a change in pull request #2573: implements create drop show index syntax for bitmap index [#2487]
URL: https://github.com/apache/incubator-doris/pull/2573#discussion_r362358633
##########
File path: fe/src/main/cup/sql_parser.cup
##########
@@ -1711,6 +1759,13 @@ column_definition ::=
:}
;
+index_definition ::=
+ KW_INDEX ident:indexName opt_index_type:indexType LPAREN ident_list:cols RPAREN opt_comment:comment
Review comment:
this is compatible with mysql
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org