You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "zzzxl1993 (via GitHub)" <gi...@apache.org> on 2023/06/15 15:20:23 UTC
[GitHub] [doris] zzzxl1993 opened a new pull request, #20876: [Improvement](load) single partition load optimize
zzzxl1993 opened a new pull request, #20876:
URL: https://github.com/apache/doris/pull/20876
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] eldenmoon commented on a diff in pull request #20876: [Improvement](load) single partition load optimize
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on code in PR #20876:
URL: https://github.com/apache/doris/pull/20876#discussion_r1231723925
##########
be/src/vec/exec/format/json/new_json_reader.cpp:
##########
@@ -1524,7 +1524,6 @@ Status NewJsonReader::_simdjson_parse_json(bool* is_empty_row, bool* eof) {
}
Status NewJsonReader::_simdjson_parse_json_doc(size_t* size, bool* eof) {
// read a whole message
- SCOPED_TIMER(_file_read_timer);
Review Comment:
If we remove this profile, some info will lost.Maybe we could add it somewhere else
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] airborne12 commented on pull request #20876: [Improvement](load) single partition load optimize
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #20876:
URL: https://github.com/apache/doris/pull/20876#issuecomment-1596171583
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] eldenmoon commented on pull request #20876: [Improvement](load) single partition load optimize
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #20876:
URL: https://github.com/apache/doris/pull/20876#issuecomment-1598022312
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] eldenmoon commented on pull request #20876: [Improvement](load) single partition load optimize
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #20876:
URL: https://github.com/apache/doris/pull/20876#issuecomment-1594316318
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20876: [Improvement](load) single partition load optimize
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20876:
URL: https://github.com/apache/doris/pull/20876#issuecomment-1593286820
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xiaokang commented on pull request #20876: [Improvement](load) single partition load optimize
Posted by "xiaokang (via GitHub)" <gi...@apache.org>.
xiaokang commented on PR #20876:
URL: https://github.com/apache/doris/pull/20876#issuecomment-1596609723
BTW, this opt can be made more widely for normal load to one partition of a table, not just for table with one partition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20876: [Improvement](load) single partition load optimize
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20876:
URL: https://github.com/apache/doris/pull/20876#issuecomment-1594202562
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] qidaye merged pull request #20876: [Improvement](load) single partition load optimize
Posted by "qidaye (via GitHub)" <gi...@apache.org>.
qidaye merged PR #20876:
URL: https://github.com/apache/doris/pull/20876
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] qidaye commented on pull request #20876: [Improvement](load) single partition load optimize
Posted by "qidaye (via GitHub)" <gi...@apache.org>.
qidaye commented on PR #20876:
URL: https://github.com/apache/doris/pull/20876#issuecomment-1598036867
Please add some description for your optimization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zzzxl1993 commented on pull request #20876: [Improvement](load) single partition load optimize
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on PR #20876:
URL: https://github.com/apache/doris/pull/20876#issuecomment-1598031159
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20876: [Improvement](load) single partition load optimize
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20876:
URL: https://github.com/apache/doris/pull/20876#issuecomment-1594149260
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20876: [Improvement](load) single partition load optimize
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20876:
URL: https://github.com/apache/doris/pull/20876#issuecomment-1594166668
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] eldenmoon commented on a diff in pull request #20876: [Improvement](load) single partition load optimize
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on code in PR #20876:
URL: https://github.com/apache/doris/pull/20876#discussion_r1231724457
##########
be/src/vec/sink/vtablet_sink.cpp:
##########
@@ -1276,6 +1278,56 @@ void VOlapTableSink::_generate_row_distribution_payload(
}
}
+Status VOlapTableSink::_single_partition_generate(RuntimeState* state, vectorized::Block* block,
+ ChannelDistributionPayload& channel_to_payload,
+ size_t num_rows, int32_t filtered_rows) {
+ const VOlapTablePartition* partition = nullptr;
+ uint32_t tablet_index = 0;
+ bool stop_processing = false;
+ for (int32_t i = 0; i < num_rows; ++i) {
+ if (UNLIKELY(filtered_rows) > 0 && _filter_bitmap.Get(i)) {
+ continue;
+ }
+ bool is_continue = false;
+ RETURN_IF_ERROR(find_tablet(state, block, i, &partition, tablet_index, stop_processing,
+ is_continue));
+ if (is_continue) {
+ continue;
+ }
+ if (config::enable_lazy_open_partition) {
+ _open_partition(partition);
+ }
+ break;
+ }
+ for (int j = 0; j < partition->indexes.size(); ++j) {
+ auto tid = partition->indexes[j].tablets[tablet_index];
+ auto it = _channels[j]->_channels_by_tablet.find(tid);
+ DCHECK(it != _channels[j]->_channels_by_tablet.end())
+ << "unknown tablet, tablet_id=" << tablet_index;
+ int64_t row_cnt = 0;
+ for (const auto& channel : it->second) {
+ if (channel_to_payload[j].count(channel.get()) < 1) {
+ channel_to_payload[j].insert(
+ {channel.get(), Payload {std::unique_ptr<vectorized::IColumn::Selector>(
+ new vectorized::IColumn::Selector()),
+ std::vector<int64_t>()}});
+ }
+ auto& ptr = channel_to_payload[j][channel.get()].first;
Review Comment:
the `ptr` and `vec` variable is too simple to understand it's meaning
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20876: [Improvement](load) single partition load optimize
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20876:
URL: https://github.com/apache/doris/pull/20876#issuecomment-1594202287
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20876: [Improvement](load) single partition load optimize
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20876:
URL: https://github.com/apache/doris/pull/20876#issuecomment-1594469367
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20876: [Improvement](load) single partition load optimize
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20876:
URL: https://github.com/apache/doris/pull/20876#issuecomment-1597185281
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org