You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "GoGoWen (via GitHub)" <gi...@apache.org> on 2023/11/14 05:10:03 UTC
[PR] [BugFix](JDBC Catalog) fix jdbc catalog query bitmap may cause be core sometimes [doris]
GoGoWen opened a new pull request, #26933:
URL: https://github.com/apache/doris/pull/26933
## Proposed changes
fix issue that sometimes be will core like below:
Aborted at 1699756702 (unix time) try "date -d @1699756702" if you are using GNU date ***
Current BE git commitID: ca47d75e83 ***
SIGABRT unknown detail explain (@0x25ce1c) received by PID 2477596 (TID 2478098 OR 0x7f3ec817c700) from PID 2477596; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk1/yyq/doris/be/src/common/signal_handler.h:417
1# 0x00007F3FF2C5F090 in /lib/x86_64-linux-gnu/libc.so.6
2# raise at ../sysdeps/unix/sysv/linux/raise.c:51
3# abort at /build/glibc-SzIz7B/glibc-2.31/stdlib/abort.c:81
4# _gnu_cxx::_verbose_terminate_handler() [clone .cold] at ../../../../libstdc+-v3/libsupc+/vterminate.cc:75
5# _cxxabiv1::_terminate(void ()) at ../../../../libstdc+-v3/libsupc+/eh_terminate.cc:48
6# 0x00005612E3DEC791 in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
7# 0x00005612E3DEC8E4 in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
8# 0x00005612E3DECCD6 in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
9# phmap::priv::raw_hash_set<phmap::priv::FlatHashSetPolicy<unsigned long>, phmap::Hash<unsigned long>, phmap::EqualTo<unsigned long>, std::allocator<unsigned long> >::resize(unsigned long) at /mnt/disk1/yyq/doris/thirdparty/installed/include/parallel_hashmap/phmap.h:2014
10# phmap::priv::raw_hash_set<phmap::priv::FlatHashSetPolicy<unsigned long>, phmap::Hash<unsigned long>, phmap::EqualTo<unsigned long>, std::allocator<unsigned long> >::raw_hash_set(phmap::priv::raw_hash_set<phmap::priv::FlatHashSetPolicy<unsigned long>, phmap::Hash<unsigned long>, phmap::EqualTo<unsigned long>, std::allocator<unsigned long> > const&, std::allocator<unsigned long> const&) at /mnt/disk1/yyq/doris/thirdparty/installed/include/parallel_hashmap/phmap.h:1167
11# doris::BitmapValue::BitmapValue(doris::BitmapValue const&) at /mnt/disk1/yyq/doris/be/src/util/bitmap_value.h:1199
12# doris::vectorized::ColumnComplexType<doris::BitmapValue>::insert(doris::vectorized::Field const&) at /mnt/disk1/yyq/doris/be/src/vec/columns/column_complex.h:146
13# doris::vectorized::IDataType::create_column_const(unsigned long, doris::vectorized::Field const&) const at /mnt/disk1/yyq/doris/be/src/vec/data_types/data_type.cpp:75
14# doris::vectorized::IDataType::create_column_const_with_default_value(unsigned long) const in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
15# doris::vectorized::JdbcConnector::_cast_string_to_bitmap(doris::SlotDescriptor const*, doris::vectorized::Block*, int, int) at /mnt/disk1/yyq/doris/be/src/vec/exec/vjdbc_connector.cpp:893
16# doris::vectorized::JdbcConnector::get_next(bool*, std::vector<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>, std::allocator<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> > >&, doris::vectorized::Block*, int) at /mnt/disk1/yyq/doris/be/src/vec/exec/vjdbc_connector.cpp:502
17# doris::vectorized::NewJdbcScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /mnt/disk1/yyq/doris/be/src/vec/exec/scan/new_jdbc_scanner.cpp:192
18# doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
19# doris::vectorized::ScannerScheduler::_scanner_scan(doris::vectorized::ScannerScheduler*, doris::vectorized::ScannerContext*, std::shared_ptr<doris::vectorized::VScanner>) at /mnt/disk1/yyq/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:397
20# std::_Function_handler<void (), doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_1::operator()() const::{lambda()#5}>::_M_invoke(std::_Any_data const&) at /mnt/disk1/yyq/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
21# doris::ThreadPool::dispatch_thread() in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
22# doris::Thread::supervise_thread(void*) at /mnt/disk1/yyq/doris/be/src/util/thread.cpp:495
23# start_thread at /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:478
24# __clone at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [BugFix](JDBC Catalog) fix jdbc catalog query bitmap may cause be core sometimes [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26933:
URL: https://github.com/apache/doris/pull/26933#issuecomment-1809661900
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.36 seconds
stream load tsv: 553 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17098755910 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [BugFix](JDBC Catalog) fix jdbc catalog query bitmap may cause be core sometimes [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26933:
URL: https://github.com/apache/doris/pull/26933#issuecomment-1809651788
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 4ac17e5b3500af7966b388d868968cf3b5aa47c2, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5254 5112 5060 5060
q2 359 155 164 155
q3 2029 2006 1964 1964
q4 1403 1354 1369 1354
q5 3982 3949 4004 3949
q6 252 130 133 130
q7 1475 883 892 883
q8 2766 2771 2759 2759
q9 9858 9648 9401 9401
q10 3464 3537 3553 3537
q11 389 258 250 250
q12 436 287 281 281
q13 4595 4114 4074 4074
q14 319 292 297 292
q15 626 568 563 563
q16 672 593 581 581
q17 1130 1067 1085 1067
q18 8123 7602 7668 7602
q19 1672 1684 1679 1679
q20 560 318 313 313
q21 4695 4330 4368 4330
q22 520 414 415 414
Total cold run time: 54579 ms
Total hot run time: 50638 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5033 4971 5211 4971
q2 332 230 235 230
q3 4162 3979 4035 3979
q4 2795 2785 2772 2772
q5 9706 9728 9702 9702
q6 243 127 124 124
q7 3035 2505 2517 2505
q8 4890 4823 4875 4823
q9 13069 12977 13129 12977
q10 4097 4168 4161 4161
q11 721 657 634 634
q12 978 847 809 809
q13 4320 3872 3830 3830
q14 376 338 367 338
q15 596 543 559 543
q16 762 666 697 666
q17 3836 3821 3975 3821
q18 9687 9410 9529 9410
q19 1807 1778 1765 1765
q20 2399 2074 2036 2036
q21 8977 8882 8902 8882
q22 928 848 865 848
Total cold run time: 82749 ms
Total hot run time: 79826 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [BugFix](JDBC Catalog) fix jdbc catalog query bitmap may cause be core sometimes [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #26933:
URL: https://github.com/apache/doris/pull/26933#discussion_r1392003574
##########
be/src/util/bitmap_value.h:
##########
@@ -1259,10 +1259,9 @@ class BitmapValue {
}
static std::string empty_bitmap() {
- static BitmapValue bitmap;
- std::string buf;
- buf.resize(bitmap.getSizeInBytes());
- bitmap.write_to(buf.data());
+ std::string buf(sizeof(BitmapValue), 0);
Review Comment:
why the original code will core???
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [BugFix](JDBC Catalog) fix jdbc catalog query bitmap may cause be core sometimes [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26933:
URL: https://github.com/apache/doris/pull/26933#issuecomment-1809674786
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [BugFix](JDBC Catalog) fix jdbc catalog query bitmap may cause be core sometimes [doris]
Posted by "GoGoWen (via GitHub)" <gi...@apache.org>.
GoGoWen commented on code in PR #26933:
URL: https://github.com/apache/doris/pull/26933#discussion_r1392060226
##########
be/src/util/bitmap_value.h:
##########
@@ -1259,10 +1259,9 @@ class BitmapValue {
}
static std::string empty_bitmap() {
- static BitmapValue bitmap;
- std::string buf;
- buf.resize(bitmap.getSizeInBytes());
- bitmap.write_to(buf.data());
+ std::string buf(sizeof(BitmapValue), 0);
Review Comment:
BitmapValue::write_to will get a string with size 1 for empty BitmapValue, however the size 1 string will reinterpret to BitmapValue* back in ColumnComplexType::insert:
void insert(const Field& x) override {
const String& s = doris::vectorized::get<const String&>(x);
data.push_back(*reinterpret_cast<const T*>(s.c_str()));
}
in data.push_back will goto BitmapValue copy constructor, as the _type is not first member in BitmapValue, cause access to an unknown memory location.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [BugFix](JDBC Catalog) fix jdbc catalog query bitmap may cause be core sometimes [doris]
Posted by "zy-kkk (via GitHub)" <gi...@apache.org>.
zy-kkk merged PR #26933:
URL: https://github.com/apache/doris/pull/26933
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [BugFix](JDBC Catalog) fix jdbc catalog query bitmap may cause be core sometimes [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26933:
URL: https://github.com/apache/doris/pull/26933#issuecomment-1809630016
TeamCity be ut coverage result:
Function Coverage: 36.74% (8418/22915)
Line Coverage: 29.26% (68387/233686)
Region Coverage: 27.87% (35343/126827)
Branch Coverage: 24.65% (18058/73254)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4ac17e5b3500af7966b388d868968cf3b5aa47c2_4ac17e5b3500af7966b388d868968cf3b5aa47c2/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [BugFix](JDBC Catalog) fix jdbc catalog query bitmap may cause be core sometimes [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26933:
URL: https://github.com/apache/doris/pull/26933#issuecomment-1809674835
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [BugFix](JDBC Catalog) fix jdbc catalog query bitmap may cause be core sometimes [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26933:
URL: https://github.com/apache/doris/pull/26933#issuecomment-1809562299
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [BugFix](JDBC Catalog) fix jdbc catalog query bitmap may cause be core sometimes [doris]
Posted by "GoGoWen (via GitHub)" <gi...@apache.org>.
GoGoWen commented on PR #26933:
URL: https://github.com/apache/doris/pull/26933#issuecomment-1809564964
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org