You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/05/12 13:20:19 UTC
[GitHub] [incubator-doris] yixiutt opened a new pull request, #9531: [improvement](load) reduce useless err_msg format in VOlapTableSink send
yixiutt opened a new pull request, #9531:
URL: https://github.com/apache/incubator-doris/pull/9531
error msg format but not used, waste a lot of cpu in fragment thread
in some case of load, just format it when needed
# Proposed changes
Issue Number: close #xxx
## Problem Summary:
improve olap tablet sink by reduce useless string format, thread perf below:
Describe the overview of changes.
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know)
2. Has unit tests been added: (Yes/No/No Need)
3. Has document been added or modified: (Yes/No/No Need)
4. Does it need to update dependencies: (Yes/No)
5. Are there any changes that cannot be rolled back: (Yes/No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yiguolei commented on a diff in pull request #9531: [improvement](load) reduce useless err_msg format in VOlapTableSink send
Posted by GitBox <gi...@apache.org>.
yiguolei commented on code in PR #9531:
URL: https://github.com/apache/incubator-doris/pull/9531#discussion_r871431637
##########
be/src/vec/sink/vtablet_sink.cpp:
##########
@@ -506,27 +506,27 @@ Status VOlapTableSink::_validate_data(RuntimeState* state, vectorized::Block* bl
if (!filter_bitmap->Get(j)) {
auto str_val = column_string->get_data_at(j);
bool invalid = str_val.size > limit;
-
error_msg.clear();
- if (str_val.size > desc->type().len) {
- fmt::format_to(error_msg, "{}",
- "the length of input is too long than schema. ");
- fmt::format_to(error_msg, "column_name: {}; ", desc->col_name());
- fmt::format_to(error_msg, "input str: [{}] ", str_val.to_prefix(10));
- fmt::format_to(error_msg, "schema length: {}; ", desc->type().len);
- fmt::format_to(error_msg, "actual length: {}; ", str_val.size);
- } else if (str_val.size > limit) {
- fmt::format_to(
- error_msg, "{}",
- "the length of input string is too long than vec schema. ");
- fmt::format_to(error_msg, "column_name: {}; ", desc->col_name());
- fmt::format_to(error_msg, "input str: [{}] ", str_val.to_prefix(10));
- fmt::format_to(error_msg, "schema length: {}; ", desc->type().len);
- fmt::format_to(error_msg, "limit length: {}; ", limit);
- fmt::format_to(error_msg, "actual length: {}; ", str_val.size);
- }
-
if (invalid) {
Review Comment:
The modified logic is not the same with previous one.
size_t limit = std::min(config::string_type_length_soft_limit_bytes, desc->type().len);
So, if (desc->type().len > string_type_length_soft_limit_bytes), then the result is wrong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yixiutt commented on a diff in pull request #9531: [improvement](load) reduce useless err_msg format in VOlapTableSink send
Posted by GitBox <gi...@apache.org>.
yixiutt commented on code in PR #9531:
URL: https://github.com/apache/incubator-doris/pull/9531#discussion_r872955673
##########
be/src/vec/sink/vtablet_sink.cpp:
##########
@@ -506,27 +506,27 @@ Status VOlapTableSink::_validate_data(RuntimeState* state, vectorized::Block* bl
if (!filter_bitmap->Get(j)) {
auto str_val = column_string->get_data_at(j);
bool invalid = str_val.size > limit;
-
error_msg.clear();
- if (str_val.size > desc->type().len) {
- fmt::format_to(error_msg, "{}",
- "the length of input is too long than schema. ");
- fmt::format_to(error_msg, "column_name: {}; ", desc->col_name());
- fmt::format_to(error_msg, "input str: [{}] ", str_val.to_prefix(10));
- fmt::format_to(error_msg, "schema length: {}; ", desc->type().len);
- fmt::format_to(error_msg, "actual length: {}; ", str_val.size);
- } else if (str_val.size > limit) {
- fmt::format_to(
- error_msg, "{}",
- "the length of input string is too long than vec schema. ");
- fmt::format_to(error_msg, "column_name: {}; ", desc->col_name());
- fmt::format_to(error_msg, "input str: [{}] ", str_val.to_prefix(10));
- fmt::format_to(error_msg, "schema length: {}; ", desc->type().len);
- fmt::format_to(error_msg, "limit length: {}; ", limit);
- fmt::format_to(error_msg, "actual length: {}; ", str_val.size);
- }
-
if (invalid) {
Review Comment:
set error msg when invalid, no logic changed, i can't understand you logic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #9531: [improvement](load) reduce useless err_msg format in VOlapTableSink send
Posted by GitBox <gi...@apache.org>.
morningman merged PR #9531:
URL: https://github.com/apache/incubator-doris/pull/9531
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9531: [improvement](load) reduce useless err_msg format in VOlapTableSink send
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #9531:
URL: https://github.com/apache/incubator-doris/pull/9531#issuecomment-1125003972
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9531: [improvement](load) reduce useless err_msg format in VOlapTableSink send
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #9531:
URL: https://github.com/apache/incubator-doris/pull/9531#issuecomment-1125003929
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org