You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/12/20 07:09:48 UTC
[GitHub] [doris] jacktengg opened a new pull request, #15199: [fix](scanner scheduler) fix coredump of ScannerScheduler::_scanner_scan
jacktengg opened a new pull request, #15199:
URL: https://github.com/apache/doris/pull/15199
# Proposed changes
Issue Number: close #15198
## Problem summary
`ScannerScheduler::_scanner_scan` missed checking returned Status of `scanner->get_block`, when `scanner->get_block` failed some column is NULL, which caused coredump when calling `block->bytes()`.
The sql mentioned in issue returns `Status::RuntimeError` in `FunctionSubstringIndex:: execute_impl`:
```
class FunctionSubstringIndex : public IFunction {
Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
size_t result, size_t input_rows_count) override {
for (size_t i = 1; i <= 2; i++) {
ColumnPtr columnPtr = remove_nullable(block.get_by_position(arguments[i]).column);
if (!is_column_const(*columnPtr)) {
return Status::RuntimeError("Argument at index {} for function {} must be constant",
i + 1, get_name());
}
}
}
}
```
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15199: [fix](scanner scheduler) fix coredump of ScannerScheduler::_scanner_scan
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15199:
URL: https://github.com/apache/doris/pull/15199#issuecomment-1359076370
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15199: [fix](scanner scheduler) fix coredump of ScannerScheduler::_scanner_scan
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15199:
URL: https://github.com/apache/doris/pull/15199#issuecomment-1360754631
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15199: [fix](scanner scheduler) fix coredump of ScannerScheduler::_scanner_scan
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15199:
URL: https://github.com/apache/doris/pull/15199#issuecomment-1358929891
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15199: [fix](scanner scheduler) fix coredump of ScannerScheduler::_scanner_scan
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15199:
URL: https://github.com/apache/doris/pull/15199#issuecomment-1360802426
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15199: [fix](scanner scheduler) fix coredump of ScannerScheduler::_scanner_scan
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15199:
URL: https://github.com/apache/doris/pull/15199#issuecomment-1359076332
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15199: [fix](scanner scheduler) fix coredump of ScannerScheduler::_scanner_scan
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15199:
URL: https://github.com/apache/doris/pull/15199#issuecomment-1360773321
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15199: [fix](scanner scheduler) fix coredump of ScannerScheduler::_scanner_scan
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15199:
URL: https://github.com/apache/doris/pull/15199#issuecomment-1360789337
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #15199: [fix](scanner scheduler) fix coredump of ScannerScheduler::_scanner_scan
Posted by GitBox <gi...@apache.org>.
yiguolei merged PR #15199:
URL: https://github.com/apache/doris/pull/15199
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15199: [fix](scanner scheduler) fix coredump of ScannerScheduler::_scanner_scan
Posted by GitBox <gi...@apache.org>.
hello-stephen commented on PR #15199:
URL: https://github.com/apache/doris/pull/15199#issuecomment-1358956616
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.18 seconds
load time: 628 seconds
storage size: 17123649045 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221220074500_clickbench_pr_65506.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #15199: [fix](scanner scheduler) fix coredump of ScannerScheduler::_scanner_scan
Posted by GitBox <gi...@apache.org>.
morningman commented on code in PR #15199:
URL: https://github.com/apache/doris/pull/15199#discussion_r1053883362
##########
be/src/vec/exec/scan/scanner_scheduler.cpp:
##########
@@ -249,6 +249,13 @@ void ScannerScheduler::_scanner_scan(ScannerScheduler* scheduler, ScannerContext
eos = true;
}
+ if (UNLIKELY(!status.ok())) {
Review Comment:
I think this logic should be merged with `if` in line 238?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #15199: [fix](scanner scheduler) fix coredump of ScannerScheduler::_scanner_scan
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on code in PR #15199:
URL: https://github.com/apache/doris/pull/15199#discussion_r1053931207
##########
be/src/vec/exec/scan/scanner_scheduler.cpp:
##########
@@ -235,12 +235,16 @@ void ScannerScheduler::_scanner_scan(ScannerScheduler* scheduler, ScannerContext
VLOG_ROW << "VOlapScanNode input rows: " << block->rows() << ", eos: " << eos;
// The VFileScanner for external table may try to open not exist files,
// Because FE file cache for external table may out of date.
- if (!status.ok() && (typeid(*scanner) == typeid(doris::vectorized::VFileScanner) &&
- !status.is<ErrorCode::NOT_FOUND>())) {
- LOG(WARNING) << "Scan thread read VOlapScanner failed: " << status.to_string();
- // Add block ptr in blocks, prevent mem leak in read failed
- blocks.push_back(block);
- break;
+ if (!status.ok()) {
+ auto type_id_scanner = typeid(*scanner);
Review Comment:
warning: calling a private constructor of class 'std::type_info' [clang-diagnostic-error]
```cpp
auto type_id_scanner = typeid(*scanner);
^
```
**/usr/include/c++/11/typeinfo:180:** declared private here
```cpp
type_info(const type_info&);
^
```
##########
be/src/vec/exec/scan/scanner_scheduler.cpp:
##########
@@ -235,12 +235,16 @@
VLOG_ROW << "VOlapScanNode input rows: " << block->rows() << ", eos: " << eos;
// The VFileScanner for external table may try to open not exist files,
// Because FE file cache for external table may out of date.
- if (!status.ok() && (typeid(*scanner) == typeid(doris::vectorized::VFileScanner) &&
- !status.is<ErrorCode::NOT_FOUND>())) {
- LOG(WARNING) << "Scan thread read VOlapScanner failed: " << status.to_string();
- // Add block ptr in blocks, prevent mem leak in read failed
- blocks.push_back(block);
- break;
+ if (!status.ok()) {
+ auto type_id_scanner = typeid(*scanner);
+ auto type_id_file_scanner = typeid(doris::vectorized::VFileScanner);
Review Comment:
warning: calling a private constructor of class 'std::type_info' [clang-diagnostic-error]
```cpp
auto type_id_file_scanner = typeid(doris::vectorized::VFileScanner);
^
```
**/usr/include/c++/11/typeinfo:180:** declared private here
```cpp
type_info(const type_info&);
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org