You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "zclllyybb (via GitHub)" <gi...@apache.org> on 2023/08/21 03:49:56 UTC
[GitHub] [doris] zclllyybb opened a new pull request, #23236: [Feature-WIP](partitions) Support auto partition
zclllyybb opened a new pull request, #23236:
URL: https://github.com/apache/doris/pull/23236
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zclllyybb commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1704339080
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #23236:
URL: https://github.com/apache/doris/pull/23236#discussion_r1299873249
##########
be/src/exec/tablet_info.cpp:
##########
@@ -213,6 +214,27 @@ VOlapTablePartitionParam::VOlapTablePartitionParam(std::shared_ptr<OlapTableSche
_partition_block.insert(
{slot->get_empty_mutable_column(), slot->get_data_type_ptr(), slot->col_name()});
}
+
+ if (t_param.__isset.enable_automatic_partition && t_param.enable_automatic_partition) {
+ _is_auto_partiton = true;
+ DCHECK(t_param.__isset.partition_function_exprs);
+ //TODO: Is there a way to avoid the dependency of slot index?
+ DCHECK(t_param.partition_function_exprs[0].nodes[0].node_type ==
+ TExprNodeType::FUNCTION_CALL)
+ << t_param.partition_function_exprs[0].nodes[0].node_type;
+ DCHECK(t_param.partition_function_exprs[0].nodes[1].node_type == TExprNodeType::SLOT_REF)
+ << t_param.partition_function_exprs[0].nodes[1].node_type;
+ DCHECK(t_param.partition_function_exprs[0].nodes[2].node_type ==
+ TExprNodeType::STRING_LITERAL)
+ << t_param.partition_function_exprs[0].nodes[2].node_type;
+
+ vectorized::VExpr::create_expr_tree(t_param.partition_function_exprs[0],
+ _part_func_ctx); //RETURN IF ERROR
+ _partition_function = _part_func_ctx->root();
+ _part_interval =
Review Comment:
if not the data_trun() function. maybe do not have child(1) please recheck the logic
##########
be/src/exec/tablet_info.cpp:
##########
@@ -213,6 +214,27 @@ VOlapTablePartitionParam::VOlapTablePartitionParam(std::shared_ptr<OlapTableSche
_partition_block.insert(
{slot->get_empty_mutable_column(), slot->get_data_type_ptr(), slot->col_name()});
}
+
+ if (t_param.__isset.enable_automatic_partition && t_param.enable_automatic_partition) {
+ _is_auto_partiton = true;
+ DCHECK(t_param.__isset.partition_function_exprs);
+ //TODO: Is there a way to avoid the dependency of slot index?
+ DCHECK(t_param.partition_function_exprs[0].nodes[0].node_type ==
+ TExprNodeType::FUNCTION_CALL)
+ << t_param.partition_function_exprs[0].nodes[0].node_type;
+ DCHECK(t_param.partition_function_exprs[0].nodes[1].node_type == TExprNodeType::SLOT_REF)
+ << t_param.partition_function_exprs[0].nodes[1].node_type;
+ DCHECK(t_param.partition_function_exprs[0].nodes[2].node_type ==
+ TExprNodeType::STRING_LITERAL)
+ << t_param.partition_function_exprs[0].nodes[2].node_type;
+
+ vectorized::VExpr::create_expr_tree(t_param.partition_function_exprs[0],
+ _part_func_ctx); //RETURN IF ERROR
+ _partition_function = _part_func_ctx->root();
+ _part_interval =
Review Comment:
if not the data_trun() function. maybe do not have child(1) please recheck the logic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #23236:
URL: https://github.com/apache/doris/pull/23236#discussion_r1303696546
##########
be/src/exec/tablet_info.cpp:
##########
@@ -213,6 +214,27 @@ VOlapTablePartitionParam::VOlapTablePartitionParam(std::shared_ptr<OlapTableSche
_partition_block.insert(
{slot->get_empty_mutable_column(), slot->get_data_type_ptr(), slot->col_name()});
}
+
+ if (t_param.__isset.enable_automatic_partition && t_param.enable_automatic_partition) {
+ _is_auto_partiton = true;
+ DCHECK(t_param.__isset.partition_function_exprs);
+ //TODO: Is there a way to avoid the dependency of slot index?
+ DCHECK(t_param.partition_function_exprs[0].nodes[0].node_type ==
+ TExprNodeType::FUNCTION_CALL)
+ << t_param.partition_function_exprs[0].nodes[0].node_type;
+ DCHECK(t_param.partition_function_exprs[0].nodes[1].node_type == TExprNodeType::SLOT_REF)
+ << t_param.partition_function_exprs[0].nodes[1].node_type;
+ DCHECK(t_param.partition_function_exprs[0].nodes[2].node_type ==
+ TExprNodeType::STRING_LITERAL)
+ << t_param.partition_function_exprs[0].nodes[2].node_type;
+
+ vectorized::VExpr::create_expr_tree(t_param.partition_function_exprs[0],
+ _part_func_ctx); //RETURN IF ERROR
+ _partition_function = _part_func_ctx->root();
+ _part_interval =
Review Comment:
I think the work should do this PR, the way introduce lots of useless code
##########
be/src/exec/tablet_info.cpp:
##########
@@ -213,6 +214,27 @@ VOlapTablePartitionParam::VOlapTablePartitionParam(std::shared_ptr<OlapTableSche
_partition_block.insert(
{slot->get_empty_mutable_column(), slot->get_data_type_ptr(), slot->col_name()});
}
+
+ if (t_param.__isset.enable_automatic_partition && t_param.enable_automatic_partition) {
+ _is_auto_partiton = true;
+ DCHECK(t_param.__isset.partition_function_exprs);
+ //TODO: Is there a way to avoid the dependency of slot index?
+ DCHECK(t_param.partition_function_exprs[0].nodes[0].node_type ==
+ TExprNodeType::FUNCTION_CALL)
+ << t_param.partition_function_exprs[0].nodes[0].node_type;
+ DCHECK(t_param.partition_function_exprs[0].nodes[1].node_type == TExprNodeType::SLOT_REF)
+ << t_param.partition_function_exprs[0].nodes[1].node_type;
+ DCHECK(t_param.partition_function_exprs[0].nodes[2].node_type ==
+ TExprNodeType::STRING_LITERAL)
+ << t_param.partition_function_exprs[0].nodes[2].node_type;
+
+ vectorized::VExpr::create_expr_tree(t_param.partition_function_exprs[0],
+ _part_func_ctx); //RETURN IF ERROR
+ _partition_function = _part_func_ctx->root();
+ _part_interval =
Review Comment:
I think the work should do this PR, the way introduce lots of useless code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zclllyybb commented on a diff in pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on code in PR #23236:
URL: https://github.com/apache/doris/pull/23236#discussion_r1303741365
##########
be/src/exec/tablet_info.cpp:
##########
@@ -213,6 +214,27 @@ VOlapTablePartitionParam::VOlapTablePartitionParam(std::shared_ptr<OlapTableSche
_partition_block.insert(
{slot->get_empty_mutable_column(), slot->get_data_type_ptr(), slot->col_name()});
}
+
+ if (t_param.__isset.enable_automatic_partition && t_param.enable_automatic_partition) {
+ _is_auto_partiton = true;
+ DCHECK(t_param.__isset.partition_function_exprs);
+ //TODO: Is there a way to avoid the dependency of slot index?
+ DCHECK(t_param.partition_function_exprs[0].nodes[0].node_type ==
+ TExprNodeType::FUNCTION_CALL)
+ << t_param.partition_function_exprs[0].nodes[0].node_type;
+ DCHECK(t_param.partition_function_exprs[0].nodes[1].node_type == TExprNodeType::SLOT_REF)
+ << t_param.partition_function_exprs[0].nodes[1].node_type;
+ DCHECK(t_param.partition_function_exprs[0].nodes[2].node_type ==
+ TExprNodeType::STRING_LITERAL)
+ << t_param.partition_function_exprs[0].nodes[2].node_type;
+
+ vectorized::VExpr::create_expr_tree(t_param.partition_function_exprs[0],
+ _part_func_ctx); //RETURN IF ERROR
+ _partition_function = _part_func_ctx->root();
+ _part_interval =
Review Comment:
Yes. fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #23236:
URL: https://github.com/apache/doris/pull/23236#discussion_r1305466325
##########
be/src/exec/tablet_info.cpp:
##########
@@ -271,60 +283,14 @@ Status VOlapTablePartitionParam::init() {
};
}
- DCHECK(!_t_param.partitions.empty()) << "must have at least 1 partition";
- _is_in_partition = _t_param.partitions[0].__isset.in_keys;
+ // empty for auto partition table. checked in caller.
+ _is_in_partition = _t_param.partitions.empty() ? false : _t_param.partitions[0].__isset.in_keys;
Review Comment:
if in_partition empty(), the check way is not right
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1702005470
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1689172050
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1689506222
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1689519058
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23236: [Feature](partitions) Support auto partition
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1707784284
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #23236:
URL: https://github.com/apache/doris/pull/23236#discussion_r1308142273
##########
be/src/vec/runtime/vdatetime_value.cpp:
##########
@@ -50,6 +50,29 @@ static const char* s_ab_month_name[] = {"", "Jan", "Feb", "Mar", "Apr", "May"
static const char* s_ab_day_name[] = {"Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun", nullptr};
+TimeUnit parse_upper_to_timeunit(const std::string& str) {
Review Comment:
remove the unless code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #23236:
URL: https://github.com/apache/doris/pull/23236#discussion_r1308145522
##########
be/src/exec/tablet_info.cpp:
##########
@@ -457,4 +475,54 @@ Status VOlapTablePartitionParam::_create_partition_key(const TExprNode& t_expr,
return Status::OK();
}
+Status VOlapTablePartitionParam::add_partitions(
+ const std::vector<TOlapTablePartition>& partitions) {
+ for (const auto& t_part : partitions) {
+ auto part = _obj_pool.add(new VOlapTablePartition(&_partition_block));
+ part->id = t_part.id;
+ part->is_mutable = t_part.is_mutable;
+
+ // for auto partitions, so range partition only
+ if (t_part.__isset.start_keys) {
+ RETURN_IF_ERROR(_create_partition_keys(t_part.start_keys, &part->start_key));
+ }
+ if (t_part.__isset.end_keys) {
+ RETURN_IF_ERROR(_create_partition_keys(t_part.end_keys, &part->end_key));
+ }
+
+ part->num_buckets = t_part.num_buckets;
+ auto num_indexes = _schema->indexes().size();
+ if (t_part.indexes.size() != num_indexes) {
+ return Status::InternalError(
+ "number of partition's index is not equal with schema's"
+ ", num_part_indexes={}, num_schema_indexes={}",
+ t_part.indexes.size(), num_indexes);
+ }
+ part->indexes = t_part.indexes;
+ std::sort(part->indexes.begin(), part->indexes.end(),
+ [](const OlapTableIndexTablets& lhs, const OlapTableIndexTablets& rhs) {
+ return lhs.index_id < rhs.index_id;
+ });
+ // check index
+ for (int j = 0; j < num_indexes; ++j) {
+ if (part->indexes[j].index_id != _schema->indexes()[j]->index_id) {
+ std::stringstream ss;
+ ss << "partition's index is not equal with schema's"
+ << ", part_index=" << part->indexes[j].index_id
+ << ", schema_index=" << _schema->indexes()[j]->index_id;
+ return Status::InternalError(
+ "partition's index is not equal with schema's"
+ ", part_index={}, schema_index={}",
+ part->indexes[j].index_id, _schema->indexes()[j]->index_id);
+ }
+ }
+ _partitions.emplace_back(part);
+ for (auto& in_key : part->in_keys) {
Review Comment:
recheck the logic only in partition have `in_keys`, here must dispose the in partition and range partiton
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #23236:
URL: https://github.com/apache/doris/pull/23236#discussion_r1305480485
##########
be/src/vec/sink/vtablet_sink.cpp:
##########
@@ -364,6 +394,7 @@ Status VNodeChannel::init(RuntimeState* state) {
_timeout_watch.start();
// Initialize _cur_add_block_request
+ //FIXME: 把这些存一个param下来给requests用。
Review Comment:
do not use chinese
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1704315451
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1704354953
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.87 seconds
stream load tsv: 536 seconds loaded 74807831229 Bytes, about 133 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162096371 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1704570909
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1702008572
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23236: [Feature](partitions) Support auto partition
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1706808692
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1704340266
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1702471378
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zclllyybb commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1705593339
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1702663081
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #23236:
URL: https://github.com/apache/doris/pull/23236#discussion_r1299871744
##########
be/src/exec/tablet_info.cpp:
##########
@@ -213,6 +214,27 @@ VOlapTablePartitionParam::VOlapTablePartitionParam(std::shared_ptr<OlapTableSche
_partition_block.insert(
{slot->get_empty_mutable_column(), slot->get_data_type_ptr(), slot->col_name()});
}
+
+ if (t_param.__isset.enable_automatic_partition && t_param.enable_automatic_partition) {
+ _is_auto_partiton = true;
+ DCHECK(t_param.__isset.partition_function_exprs);
+ //TODO: Is there a way to avoid the dependency of slot index?
+ DCHECK(t_param.partition_function_exprs[0].nodes[0].node_type ==
+ TExprNodeType::FUNCTION_CALL)
+ << t_param.partition_function_exprs[0].nodes[0].node_type;
+ DCHECK(t_param.partition_function_exprs[0].nodes[1].node_type == TExprNodeType::SLOT_REF)
+ << t_param.partition_function_exprs[0].nodes[1].node_type;
+ DCHECK(t_param.partition_function_exprs[0].nodes[2].node_type ==
+ TExprNodeType::STRING_LITERAL)
+ << t_param.partition_function_exprs[0].nodes[2].node_type;
+
+ vectorized::VExpr::create_expr_tree(t_param.partition_function_exprs[0],
Review Comment:
`partition_function_exprs` seems a list?
##########
be/src/exec/tablet_info.cpp:
##########
@@ -213,6 +214,27 @@ VOlapTablePartitionParam::VOlapTablePartitionParam(std::shared_ptr<OlapTableSche
_partition_block.insert(
{slot->get_empty_mutable_column(), slot->get_data_type_ptr(), slot->col_name()});
}
+
+ if (t_param.__isset.enable_automatic_partition && t_param.enable_automatic_partition) {
+ _is_auto_partiton = true;
+ DCHECK(t_param.__isset.partition_function_exprs);
+ //TODO: Is there a way to avoid the dependency of slot index?
+ DCHECK(t_param.partition_function_exprs[0].nodes[0].node_type ==
+ TExprNodeType::FUNCTION_CALL)
+ << t_param.partition_function_exprs[0].nodes[0].node_type;
+ DCHECK(t_param.partition_function_exprs[0].nodes[1].node_type == TExprNodeType::SLOT_REF)
+ << t_param.partition_function_exprs[0].nodes[1].node_type;
+ DCHECK(t_param.partition_function_exprs[0].nodes[2].node_type ==
+ TExprNodeType::STRING_LITERAL)
+ << t_param.partition_function_exprs[0].nodes[2].node_type;
+
+ vectorized::VExpr::create_expr_tree(t_param.partition_function_exprs[0],
Review Comment:
`partition_function_exprs` seems a list?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #23236:
URL: https://github.com/apache/doris/pull/23236#discussion_r1299870678
##########
be/src/exec/tablet_info.cpp:
##########
@@ -213,6 +214,27 @@ VOlapTablePartitionParam::VOlapTablePartitionParam(std::shared_ptr<OlapTableSche
_partition_block.insert(
{slot->get_empty_mutable_column(), slot->get_data_type_ptr(), slot->col_name()});
}
+
+ if (t_param.__isset.enable_automatic_partition && t_param.enable_automatic_partition) {
+ _is_auto_partiton = true;
+ DCHECK(t_param.__isset.partition_function_exprs);
+ //TODO: Is there a way to avoid the dependency of slot index?
+ DCHECK(t_param.partition_function_exprs[0].nodes[0].node_type ==
Review Comment:
delete the useless dcheck
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #23236:
URL: https://github.com/apache/doris/pull/23236#discussion_r1308346845
##########
be/src/vec/sink/vtablet_sink.cpp:
##########
@@ -1328,30 +1538,114 @@ Status VOlapTableSink::send(RuntimeState* state, vectorized::Block* input_block,
_row_distribution_watch.start();
auto num_rows = block->rows();
size_t partition_num = _vpartition->get_partitions().size();
- if (partition_num == 1 && _tablet_finder->is_find_tablet_every_sink()) {
+ if (!_vpartition->is_auto_partition() && partition_num == 1 &&
+ _tablet_finder->is_find_tablet_every_sink()) {
RETURN_IF_ERROR(_single_partition_generate(state, block.get(), channel_to_payload, num_rows,
has_filtered_rows));
} else {
- for (int i = 0; i < num_rows; ++i) {
- if (UNLIKELY(has_filtered_rows) && _block_convertor->filter_bitmap().Get(i)) {
- continue;
+ if (_vpartition->is_auto_partition()) {
+ std::vector<uint16_t> partition_keys = _vpartition->get_partition_keys();
+ //TODO: use loop to create missing_vals for multi column.
+ CHECK(partition_keys.size() == 1)
+ << "now support only 1 partition column for auto partitions.";
+ auto partition_col = block->get_by_position(partition_keys[0]);
+
+ auto missing_vals_col = partition_col.clone_empty();
+
+ // try to find tablet and save missing value
+ for (int i = 0; i < num_rows; ++i) {
+ if (UNLIKELY(has_filtered_rows) && _block_convertor->filter_bitmap().Get(i)) {
+ continue;
+ }
+ const VOlapTablePartition* partition = nullptr;
+ bool is_continue = false;
+ uint32_t tablet_index = 0;
+ bool missing_this = false;
+ RETURN_IF_ERROR(_tablet_finder->find_tablet(state, block.get(), i, &partition,
+ tablet_index, stop_processing,
+ is_continue, &missing_this));
+ if (missing_this) {
+ missing_vals_col.column->assume_mutable()->insert_from(*partition_col.column,
+ i);
+ } else {
+ _generate_row_distribution_payload(channel_to_payload, partition, tablet_index,
+ i, 1);
+ }
}
- const VOlapTablePartition* partition = nullptr;
- bool is_continue = false;
- uint32_t tablet_index = 0;
- RETURN_IF_ERROR(_tablet_finder->find_tablet(
- state, block.get(), i, &partition, tablet_index, stop_processing, is_continue));
- if (is_continue) {
- continue;
+
+ // for missing partition keys, calc the missing partition and save in _partitions_need_create
+ auto type = missing_vals_col.type;
+ if (missing_vals_col.column->size() > 0) {
+ vectorized::Block tmp_block = {missing_vals_col};
+ auto [part_ctx, part_func] = _get_partition_function();
+ int result_idx;
+ // calc the start value of missing partition ranges.
+ part_func->prepare(_state, *_output_row_desc, part_ctx.get());
+ part_func->execute(part_ctx.get(), &tmp_block, &result_idx);
+ VLOG_DEBUG << tmp_block.dump_data();
+
+ char convert_buffer[30];
+ // expose the data column
+ vectorized::ColumnPtr col = tmp_block.get_by_position(result_idx).column;
+ if (auto* nullable = check_and_get_column<vectorized::ColumnNullable>(*col)) {
+ col = nullable->get_nested_column_ptr();
+ }
+ // calc the end value and save them.
+ if (type->get_type_id() == vectorized::TypeIndex::DateV2) {
+ auto* arg_col = assert_cast<const vectorized::ColumnDateV2*>(col.get());
+ for (int row = 0; row < arg_col->size(); row++) {
+ _save_a_missing_val<vectorized::DateV2Value<vectorized::DateV2ValueType>>(
Review Comment:
why not call type to string?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1700441555
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zclllyybb commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1704567609
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1705587457
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #23236:
URL: https://github.com/apache/doris/pull/23236#discussion_r1308324404
##########
be/src/vec/sink/vtablet_sink.cpp:
##########
@@ -1328,30 +1538,114 @@ Status VOlapTableSink::send(RuntimeState* state, vectorized::Block* input_block,
_row_distribution_watch.start();
auto num_rows = block->rows();
size_t partition_num = _vpartition->get_partitions().size();
- if (partition_num == 1 && _tablet_finder->is_find_tablet_every_sink()) {
+ if (!_vpartition->is_auto_partition() && partition_num == 1 &&
+ _tablet_finder->is_find_tablet_every_sink()) {
RETURN_IF_ERROR(_single_partition_generate(state, block.get(), channel_to_payload, num_rows,
has_filtered_rows));
} else {
- for (int i = 0; i < num_rows; ++i) {
- if (UNLIKELY(has_filtered_rows) && _block_convertor->filter_bitmap().Get(i)) {
- continue;
+ if (_vpartition->is_auto_partition()) {
+ std::vector<uint16_t> partition_keys = _vpartition->get_partition_keys();
+ //TODO: use loop to create missing_vals for multi column.
+ CHECK(partition_keys.size() == 1)
+ << "now support only 1 partition column for auto partitions.";
+ auto partition_col = block->get_by_position(partition_keys[0]);
+
+ auto missing_vals_col = partition_col.clone_empty();
+
+ // try to find tablet and save missing value
+ for (int i = 0; i < num_rows; ++i) {
+ if (UNLIKELY(has_filtered_rows) && _block_convertor->filter_bitmap().Get(i)) {
+ continue;
+ }
+ const VOlapTablePartition* partition = nullptr;
+ bool is_continue = false;
+ uint32_t tablet_index = 0;
+ bool missing_this = false;
+ RETURN_IF_ERROR(_tablet_finder->find_tablet(state, block.get(), i, &partition,
+ tablet_index, stop_processing,
+ is_continue, &missing_this));
+ if (missing_this) {
+ missing_vals_col.column->assume_mutable()->insert_from(*partition_col.column,
+ i);
+ } else {
+ _generate_row_distribution_payload(channel_to_payload, partition, tablet_index,
+ i, 1);
+ }
}
- const VOlapTablePartition* partition = nullptr;
- bool is_continue = false;
- uint32_t tablet_index = 0;
- RETURN_IF_ERROR(_tablet_finder->find_tablet(
- state, block.get(), i, &partition, tablet_index, stop_processing, is_continue));
- if (is_continue) {
- continue;
+
+ // for missing partition keys, calc the missing partition and save in _partitions_need_create
+ auto type = missing_vals_col.type;
+ if (missing_vals_col.column->size() > 0) {
+ vectorized::Block tmp_block = {missing_vals_col};
+ auto [part_ctx, part_func] = _get_partition_function();
+ int result_idx;
+ // calc the start value of missing partition ranges.
+ part_func->prepare(_state, *_output_row_desc, part_ctx.get());
Review Comment:
the func only need to prepare one time, and must dispose the error status
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #23236:
URL: https://github.com/apache/doris/pull/23236#discussion_r1308304337
##########
be/src/vec/sink/vtablet_sink.cpp:
##########
@@ -381,17 +409,20 @@ Status VNodeChannel::init(RuntimeState* state) {
return Status::OK();
}
-void VNodeChannel::open() {
+void VNodeChannel::_open_internal(int64_t index_id, std::vector<PTabletWithPartition>& tablets,
Review Comment:
where use the param `index_id` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zclllyybb commented on pull request #23236: [Feature](partitions) Support auto partition
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1706785945
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zclllyybb commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1702365549
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zclllyybb commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1704313879
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zclllyybb commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1704319580
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1705498804
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1705505339
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zclllyybb commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1705157488
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] selectdb-robot commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "selectdb-robot (via GitHub)" <gi...@apache.org>.
selectdb-robot commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1705542768
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.8 seconds
stream load tsv: 537 seconds loaded 74807831229 Bytes, about 132 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162002283 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1703767908
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1685601485
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #23236:
URL: https://github.com/apache/doris/pull/23236#discussion_r1308322494
##########
be/src/vec/sink/vtablet_sink.cpp:
##########
@@ -1328,30 +1538,114 @@ Status VOlapTableSink::send(RuntimeState* state, vectorized::Block* input_block,
_row_distribution_watch.start();
auto num_rows = block->rows();
size_t partition_num = _vpartition->get_partitions().size();
- if (partition_num == 1 && _tablet_finder->is_find_tablet_every_sink()) {
+ if (!_vpartition->is_auto_partition() && partition_num == 1 &&
+ _tablet_finder->is_find_tablet_every_sink()) {
RETURN_IF_ERROR(_single_partition_generate(state, block.get(), channel_to_payload, num_rows,
has_filtered_rows));
} else {
- for (int i = 0; i < num_rows; ++i) {
- if (UNLIKELY(has_filtered_rows) && _block_convertor->filter_bitmap().Get(i)) {
- continue;
+ if (_vpartition->is_auto_partition()) {
+ std::vector<uint16_t> partition_keys = _vpartition->get_partition_keys();
+ //TODO: use loop to create missing_vals for multi column.
+ CHECK(partition_keys.size() == 1)
+ << "now support only 1 partition column for auto partitions.";
+ auto partition_col = block->get_by_position(partition_keys[0]);
+
+ auto missing_vals_col = partition_col.clone_empty();
+
+ // try to find tablet and save missing value
+ for (int i = 0; i < num_rows; ++i) {
+ if (UNLIKELY(has_filtered_rows) && _block_convertor->filter_bitmap().Get(i)) {
+ continue;
+ }
+ const VOlapTablePartition* partition = nullptr;
+ bool is_continue = false;
+ uint32_t tablet_index = 0;
+ bool missing_this = false;
+ RETURN_IF_ERROR(_tablet_finder->find_tablet(state, block.get(), i, &partition,
+ tablet_index, stop_processing,
+ is_continue, &missing_this));
+ if (missing_this) {
+ missing_vals_col.column->assume_mutable()->insert_from(*partition_col.column,
Review Comment:
TODO:reduce the useless virtual function call
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1700967689
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1702133257
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zclllyybb commented on a diff in pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on code in PR #23236:
URL: https://github.com/apache/doris/pull/23236#discussion_r1302381512
##########
be/src/exec/tablet_info.cpp:
##########
@@ -213,6 +214,27 @@ VOlapTablePartitionParam::VOlapTablePartitionParam(std::shared_ptr<OlapTableSche
_partition_block.insert(
{slot->get_empty_mutable_column(), slot->get_data_type_ptr(), slot->col_name()});
}
+
+ if (t_param.__isset.enable_automatic_partition && t_param.enable_automatic_partition) {
+ _is_auto_partiton = true;
+ DCHECK(t_param.__isset.partition_function_exprs);
+ //TODO: Is there a way to avoid the dependency of slot index?
+ DCHECK(t_param.partition_function_exprs[0].nodes[0].node_type ==
+ TExprNodeType::FUNCTION_CALL)
+ << t_param.partition_function_exprs[0].nodes[0].node_type;
+ DCHECK(t_param.partition_function_exprs[0].nodes[1].node_type == TExprNodeType::SLOT_REF)
+ << t_param.partition_function_exprs[0].nodes[1].node_type;
+ DCHECK(t_param.partition_function_exprs[0].nodes[2].node_type ==
+ TExprNodeType::STRING_LITERAL)
+ << t_param.partition_function_exprs[0].nodes[2].node_type;
+
+ vectorized::VExpr::create_expr_tree(t_param.partition_function_exprs[0],
+ _part_func_ctx); //RETURN IF ERROR
+ _partition_function = _part_func_ctx->root();
+ _part_interval =
Review Comment:
will do in next pr. FE send the interval_slot_ref to BE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #23236:
URL: https://github.com/apache/doris/pull/23236#discussion_r1305471324
##########
be/src/exec/tablet_info.cpp:
##########
@@ -456,4 +476,54 @@ Status VOlapTablePartitionParam::_create_partition_key(const TExprNode& t_expr,
return Status::OK();
}
+Status VOlapTablePartitionParam::add_partitions(
Review Comment:
seems no where call this function?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #23236:
URL: https://github.com/apache/doris/pull/23236#discussion_r1308142396
##########
be/src/vec/runtime/vdatetime_value.h:
##########
@@ -69,6 +71,8 @@ enum TimeUnit {
YEAR_MONTH
};
+TimeUnit parse_upper_to_timeunit(const std::string& str);
Review Comment:
remove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #23236:
URL: https://github.com/apache/doris/pull/23236#discussion_r1308310665
##########
be/src/vec/sink/vtablet_sink.cpp:
##########
@@ -637,6 +594,16 @@ Status VNodeChannel::add_block(vectorized::Block* block, const Payload* payload,
}
SCOPED_RAW_TIMER(&_stat.append_node_channel_ns);
+ // Initialize _cur_add_block_request
+ //FIXME: move to init
+ auto cur_add_block_request = PTabletWriterAddBlockRequest {};
+ cur_add_block_request.set_allocated_id(&_parent->_load_id);
+ for (const auto& [index_id, tablets] : _index_tablets_map) {
+ cur_add_block_request.set_index_id(index_id);
Review Comment:
recheck the logic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23236: [Feature](partitions) Support auto partition
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1707784347
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1704595189
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 49.31 seconds
stream load tsv: 541 seconds loaded 74807831229 Bytes, about 131 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17162166014 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zclllyybb commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1705505863
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1704724224
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zclllyybb commented on a diff in pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on code in PR #23236:
URL: https://github.com/apache/doris/pull/23236#discussion_r1299940879
##########
be/src/exec/tablet_info.cpp:
##########
@@ -213,6 +214,27 @@ VOlapTablePartitionParam::VOlapTablePartitionParam(std::shared_ptr<OlapTableSche
_partition_block.insert(
{slot->get_empty_mutable_column(), slot->get_data_type_ptr(), slot->col_name()});
}
+
+ if (t_param.__isset.enable_automatic_partition && t_param.enable_automatic_partition) {
+ _is_auto_partiton = true;
+ DCHECK(t_param.__isset.partition_function_exprs);
+ //TODO: Is there a way to avoid the dependency of slot index?
+ DCHECK(t_param.partition_function_exprs[0].nodes[0].node_type ==
+ TExprNodeType::FUNCTION_CALL)
+ << t_param.partition_function_exprs[0].nodes[0].node_type;
+ DCHECK(t_param.partition_function_exprs[0].nodes[1].node_type == TExprNodeType::SLOT_REF)
+ << t_param.partition_function_exprs[0].nodes[1].node_type;
+ DCHECK(t_param.partition_function_exprs[0].nodes[2].node_type ==
+ TExprNodeType::STRING_LITERAL)
+ << t_param.partition_function_exprs[0].nodes[2].node_type;
+
+ vectorized::VExpr::create_expr_tree(t_param.partition_function_exprs[0],
Review Comment:
yes. for multi partition keys&functions. now we only support one partition key column. so use [0]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #23236:
URL: https://github.com/apache/doris/pull/23236#discussion_r1305471324
##########
be/src/exec/tablet_info.cpp:
##########
@@ -456,4 +476,54 @@ Status VOlapTablePartitionParam::_create_partition_key(const TExprNode& t_expr,
return Status::OK();
}
+Status VOlapTablePartitionParam::add_partitions(
Review Comment:
seems no where call this function?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1702239058
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #23236:
URL: https://github.com/apache/doris/pull/23236#discussion_r1308349211
##########
be/src/vec/sink/vtablet_sink.cpp:
##########
@@ -1328,30 +1538,114 @@ Status VOlapTableSink::send(RuntimeState* state, vectorized::Block* input_block,
_row_distribution_watch.start();
auto num_rows = block->rows();
size_t partition_num = _vpartition->get_partitions().size();
- if (partition_num == 1 && _tablet_finder->is_find_tablet_every_sink()) {
+ if (!_vpartition->is_auto_partition() && partition_num == 1 &&
+ _tablet_finder->is_find_tablet_every_sink()) {
RETURN_IF_ERROR(_single_partition_generate(state, block.get(), channel_to_payload, num_rows,
has_filtered_rows));
} else {
- for (int i = 0; i < num_rows; ++i) {
- if (UNLIKELY(has_filtered_rows) && _block_convertor->filter_bitmap().Get(i)) {
- continue;
+ if (_vpartition->is_auto_partition()) {
+ std::vector<uint16_t> partition_keys = _vpartition->get_partition_keys();
+ //TODO: use loop to create missing_vals for multi column.
+ CHECK(partition_keys.size() == 1)
+ << "now support only 1 partition column for auto partitions.";
+ auto partition_col = block->get_by_position(partition_keys[0]);
+
+ auto missing_vals_col = partition_col.clone_empty();
+
+ // try to find tablet and save missing value
+ for (int i = 0; i < num_rows; ++i) {
+ if (UNLIKELY(has_filtered_rows) && _block_convertor->filter_bitmap().Get(i)) {
+ continue;
+ }
+ const VOlapTablePartition* partition = nullptr;
+ bool is_continue = false;
+ uint32_t tablet_index = 0;
+ bool missing_this = false;
+ RETURN_IF_ERROR(_tablet_finder->find_tablet(state, block.get(), i, &partition,
+ tablet_index, stop_processing,
+ is_continue, &missing_this));
+ if (missing_this) {
+ missing_vals_col.column->assume_mutable()->insert_from(*partition_col.column,
+ i);
+ } else {
+ _generate_row_distribution_payload(channel_to_payload, partition, tablet_index,
+ i, 1);
+ }
}
- const VOlapTablePartition* partition = nullptr;
- bool is_continue = false;
- uint32_t tablet_index = 0;
- RETURN_IF_ERROR(_tablet_finder->find_tablet(
- state, block.get(), i, &partition, tablet_index, stop_processing, is_continue));
- if (is_continue) {
- continue;
+
+ // for missing partition keys, calc the missing partition and save in _partitions_need_create
+ auto type = missing_vals_col.type;
+ if (missing_vals_col.column->size() > 0) {
+ vectorized::Block tmp_block = {missing_vals_col};
+ auto [part_ctx, part_func] = _get_partition_function();
+ int result_idx;
+ // calc the start value of missing partition ranges.
+ part_func->prepare(_state, *_output_row_desc, part_ctx.get());
+ part_func->execute(part_ctx.get(), &tmp_block, &result_idx);
+ VLOG_DEBUG << tmp_block.dump_data();
+
+ char convert_buffer[30];
+ // expose the data column
+ vectorized::ColumnPtr col = tmp_block.get_by_position(result_idx).column;
+ if (auto* nullable = check_and_get_column<vectorized::ColumnNullable>(*col)) {
+ col = nullable->get_nested_column_ptr();
+ }
+ // calc the end value and save them.
+ if (type->get_type_id() == vectorized::TypeIndex::DateV2) {
+ auto* arg_col = assert_cast<const vectorized::ColumnDateV2*>(col.get());
+ for (int row = 0; row < arg_col->size(); row++) {
+ _save_a_missing_val<vectorized::DateV2Value<vectorized::DateV2ValueType>>(
+ arg_col, row, convert_buffer, PrimitiveType::TYPE_DATEV2);
+ }
+ } else if (type->get_type_id() == vectorized::TypeIndex::DateTimeV2) {
+ auto* arg_col = assert_cast<const vectorized::ColumnDateTimeV2*>(col.get());
+ for (int row = 0; row < arg_col->size(); row++) {
+ _save_a_missing_val<
+ vectorized::DateV2Value<vectorized::DateTimeV2ValueType>>(
+ arg_col, row, convert_buffer, PrimitiveType::TYPE_DATETIMEV2);
+ }
+ } else if (type->get_type_id() == vectorized::TypeIndex::Date) {
+ auto* arg_col = assert_cast<const vectorized::ColumnDate*>(col.get());
+ for (int row = 0; row < arg_col->size(); row++) {
+ _save_a_missing_val<vectorized::VecDateTimeValue>(
+ arg_col, row, convert_buffer, PrimitiveType::TYPE_DATE);
+ }
+ } else if (type->get_type_id() == vectorized::TypeIndex::DateTime) {
+ auto* arg_col = assert_cast<const vectorized::ColumnDateTime*>(col.get());
+ for (int row = 0; row < arg_col->size(); row++) {
+ _save_a_missing_val<vectorized::VecDateTimeValue>(
+ arg_col, row, convert_buffer, PrimitiveType::TYPE_DATETIME);
+ }
+ }
+ // then call FE to create it. then FragmentExecutor will redo the load.
+ _automatic_create_partition();
Review Comment:
here need to check the ` _automatic_create_partition` status
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] selectdb-robot commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "selectdb-robot (via GitHub)" <gi...@apache.org>.
selectdb-robot commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1705619838
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 49 seconds
stream load tsv: 539 seconds loaded 74807831229 Bytes, about 132 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162062100 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1704675904
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] selectdb-robot commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
Posted by "selectdb-robot (via GitHub)" <gi...@apache.org>.
selectdb-robot commented on PR #23236:
URL: https://github.com/apache/doris/pull/23236#issuecomment-1705300228
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.19 seconds
stream load tsv: 543 seconds loaded 74807831229 Bytes, about 131 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162056093 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] HappenLee merged pull request #23236: [Feature](partitions) Support auto partition
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee merged PR #23236:
URL: https://github.com/apache/doris/pull/23236
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org