You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/12/10 04:58:45 UTC
[GitHub] [doris] HappenLee opened a new pull request, #14984: [Bug](DictoryColumn) reverse the _codes.size() replace _reserve_size
HappenLee opened a new pull request, #14984:
URL: https://github.com/apache/doris/pull/14984
# Proposed changes
fix bug of reserve not enough
```
==21902==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7f730c7df000 at pc 0x55918c70fe1f bp 0x7f744e24cde0 sp 0x7f744e24cdd0
WRITE of size 16 at 0x7f730c7df000 thread T138 (_scanner_scan)
#0 0x55918c70fe1e in void doris::vectorized::PODArray<doris::StringValue, 4096ul, Allocator<false, false>, 15ul, 16ul>::push_back_without_reserve<doris::StringValue&>(doris::StringValue&) /mnt/disk1/happen/doris-verion/doris/be/src/vec/common/pod_array.h:424
#1 0x55918c71413e in doris::vectorized::PredicateColumnType<(doris::PrimitiveType)23>::insert_string_value(char const*, unsigned long) /mnt/disk1/happen/doris-verion/doris/be/src/vec/columns/predicate_column.h:191
#2 0x55918c70f992 in doris::vectorized::PredicateColumnType<(doris::PrimitiveType)23>::insert_data(char const*, unsigned long) /mnt/disk1/happen/doris-verion/doris/be/src/vec/columns/predicate_column.h:215
#3 0x55918c7035c2 in doris::vectorized::ColumnDictionary<int>::convert_to_predicate_column_if_dictionary() /mnt/disk1/happen/doris-verion/doris/be/src/vec/columns/column_dictionary.h:286
#4 0x5591913b3d08 in doris::vectorized::ColumnNullable::convert_to_predicate_column_if_dictionary() /mnt/disk1/happen/doris-verion/doris/be/src/vec/columns/column_nullable.h:302
#5 0x55918da54cf3 in doris::segment_v2::BinaryDictPageDecoder::next_batch(unsigned long*, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&) /mnt/disk1/happen/doris-verion/doris/be/src/olap/rowset/segment_v2/binary_dict_page.cpp:242
#6 0x55918d8583bf in doris::segment_v2::FileColumnIterator::next_batch(unsigned long*, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&, bool*) /mnt/disk1/happen/doris-verion/doris/be/src/olap/rowset/segment_v2/column_reader.cpp:805
#7 0x55918daf182f in doris::segment_v2::ColumnIterator::next_batch(unsigned long*, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&) /mnt/disk1/happen/doris-verion/doris/be/src/olap/rowset/segment_v2/column_reader.h:254
#8 0x55918dae41e6 in doris::segment_v2::SegmentIterator::_read_columns(std::vector<unsigned int, std::allocator<unsigned int> > const&, std::vector<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>, std::allocator<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> > >&, unsigned long) /mnt/disk1/happen/doris-verion/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:923
#9 0x55918dae594f in doris::segment_v2::SegmentIterator::_read_columns_by_index(unsigned int, unsigned int&, bool) /mnt/disk1/happen/doris-verion/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:986
#10 0x55918dae8b5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) /mnt/disk1/happen/doris-verion/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1143
#11 0x55919a88cea2 in doris::vectorized::VUnionIterator::next_batch(doris::vectorized::Block*) /mnt/disk1/happen/doris-verion/doris/be/src/vec/olap/vgeneric_iterators.cpp:383
#12 0x55918e690ac5 in doris::BetaRowsetReader::next_block(doris::vectorized::Block*) /mnt/disk1/happen/doris-verion/doris/be/src/olap/rowset/beta_rowset_reader.cpp:298
#13 0x55919a8ad395 in doris::vectorized::VCollectIterator::Level0Iterator::next(doris::vectorized::Block*) /mnt/disk1/happen/doris-verion/doris/be/src/vec/olap/vcollect_iterator.cpp:267
#14 0x55919a8b1f97 in doris::vectorized::VCollectIterator::Level1Iterator::_normal_next(doris::vectorized::Block*) /mnt/disk1/happen/doris-verion/doris/be/src/vec/olap/vcollect_iterator.cpp:533
#15 0x55919a8af001 in doris::vectorized::VCollectIterator::Level1Iterator::next(doris::vectorized::Block*) /mnt/disk1/happen/doris-verion/doris/be/src/vec/olap/vcollect_iterator.cpp:359
#16 0x55919a8ab63e in doris::vectorized::VCollectIterator::next(doris::vectorized::Block*) /mnt/disk1/happen/doris-verion/doris/be/src/vec/olap/vcollect_iterator.cpp:187
#17 0x55919a8c7ed4 in doris::vectorized::BlockReader::_direct_next_block(doris::vectorized::Block*, doris::MemPool*, doris::ObjectPool*, bool*) /mnt/disk1/happen/doris-verion/doris/be/src/vec/olap/block_reader.cpp:167
#18 0x55918d33c620 in doris::vectorized::BlockReader::next_block_with_aggregation(doris::vectorized::Block*, doris::MemPool*, doris::ObjectPool*, bool*) /mnt/disk1/happen/doris-verion/doris/be/src/vec/olap/block_reader.h:45
#19 0x55919b10b053 in doris::vectorized::NewOlapScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) /mnt/disk1/happen/doris-verion/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:315
#20 0x55919afcee80 in doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) /mnt/disk1/happen/doris-verion/doris/be/src/vec/exec/scan/vscanner.cpp:53
#21 0x55919afa77b3 in doris::vectorized::ScannerScheduler::_scanner_scan(doris::vectorized::ScannerScheduler*, doris::vectorized::ScannerContext*, doris::vectorized::VScanner*) /mnt/disk1/happen/doris-verion/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236
#22 0x55919afa5a21 in operator() /mnt/disk1/happen/doris-verion/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:143
#23 0x55919afa9a0b in __invoke_impl<void, doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::<lambda()>&> /mnt/disk1/happen/ldb_toolchain/include/c++/11/bits/invoke.h:61
#24 0x55919afa9683 in __invoke_r<void, doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::<lambda()>&> /mnt/disk1/happen/ldb_toolchain/include/c++/11/bits/invoke.h:111
#25 0x55919afa902b in _M_invoke /mnt/disk1/happen/ldb_toolchain/include/c++/11/bits/std_function.h:291
#26 0x55918e7ebe95 in std::function<void ()>::operator()() const /mnt/disk1/happen/ldb_toolchain/include/c++/11/bits/std_function.h:560
#27 0x55918f34f89f in doris::FunctionRunnable::run() /mnt/disk1/happen/doris-verion/doris/be/src/util/threadpool.cpp:45
#28 0x55918f34a6eb in doris::ThreadPool::dispatch_thread() /mnt/disk1/happen/doris-verion/doris/be/src/util/threadpool.cpp:534
#29 0x55918f36c06d in void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk1/happen/ldb_toolchain/include/c++/11/bits/invoke.h:74
#30 0x55918f36b90c in std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk1/happen/ldb_toolchain/include/c++/11/bits/invoke.h:96
#31 0x55918f36acab in void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /mnt/disk1/happen/ldb_toolchain/include/c++/11/functional:420
#32 0x55918f3697bc in void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::operator()<, void>() /mnt/disk1/happen/ldb_toolchain/include/c++/11/functional:503
#33 0x55918f3663ad in void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) /mnt/disk1/happen/ldb_toolchain/include/c++/11/bits/invoke.h:61
#34 0x55918f363865 in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) /mnt/disk1/happen/ldb_toolchain/include/c++/11/bits/invoke.h:111
#35 0x55918f35eb64 in std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()> >::_M_invoke(std::_Any_data const&) /mnt/disk1/happen/ldb_toolchain/include/c++/11/bits/std_function.h:291
#36 0x55918e7ebe95 in std::function<void ()>::operator()() const /mnt/disk1/happen/ldb_toolchain/include/c++/11/bits/std_function.h:560
#37 0x55918f32a38f in doris::Thread::supervise_thread(void*) /mnt/disk1/happen/doris-verion/doris/be/src/util/thread.cpp:454
#38 0x7f752d075ea4 in start_thread (/lib64/libpthread.so.0+0x7ea4)
#39 0x7f752d3888dc in __clone (/lib64/libc.so.6+0xfe8dc)
0x7f730c7df000 is located 6144 bytes to the left of 617327-byte region [0x7f730c7e0800,0x7f730c87736f)
freed by thread T138 (_scanner_scan) here:
#0 0x55918c35d377 in operator delete[](void*) (/data/doris/be/lib/doris_be+0xf15b377)
#1 0x55918c48b68a in std::enable_if<std::is_convertible<char (*) [], char (*) []>::value, void>::type std::default_delete<char []>::operator()<char>(char*) const /mnt/disk1/happen/ldb_toolchain/include/c++/11/bits/unique_ptr.h:120
#2 0x55918c48b737 in std::__uniq_ptr_impl<char, std::default_delete<char []> >::reset(char*) /mnt/disk1/happen/ldb_toolchain/include/c++/11/bits/unique_ptr.h:182
#3 0x55918d9d4143 in std::__uniq_ptr_impl<char, std::default_delete<char []> >::operator=(std::__uniq_ptr_impl<char, std::default_delete<char []> >&&) /mnt/disk1/happen/ldb_toolchain/include/c++/11/bits/unique_ptr.h:167
#4 0x55918d99220c in std::__uniq_ptr_data<char, std::default_delete<char []>, true, true>::operator=(std::__uniq_ptr_data<char, std::default_delete<char []>, true, true>&&) /mnt/disk1/happen/ldb_toolchain/include/c++/11/bits/unique_ptr.h:212
#5 0x55918d992236 in std::unique_ptr<char [], std::default_delete<char []> >::operator=(std::unique_ptr<char [], std::default_delete<char []> >&&) /mnt/disk1/happen/ldb_toolchain/include/c++/11/bits/unique_ptr.h:623
#6 0x55918da339b7 in doris::segment_v2::PageIO::read_and_decompress_page(doris::segment_v2::PageReadOptions const&, doris::segment_v2::PageHandle*, doris::Slice*, doris::segment_v2::PageFooterPB*) /mnt/disk1/happen/doris-verion/doris/be/src/olap/rowset/segment_v2/page_io.cpp:185
#7 0x55918d849104 in doris::segment_v2::ColumnReader::read_page(doris::segment_v2::ColumnIteratorOptions const&, doris::segment_v2::PagePointer const&, doris::segment_v2::PageHandle*, doris::Slice*, doris::segment_v2::PageFooterPB*, doris::BlockCompressionCodec*) const /mnt/disk1/happen/doris-verion/doris/be/src/olap/rowset/segment_v2/column_reader.cpp:161
#8 0x55918d85af76 in doris::segment_v2::FileColumnIterator::_read_data_page(doris::segment_v2::OrdinalPageIndexIterator const&) /mnt/disk1/happen/doris-verion/doris/be/src/olap/rowset/segment_v2/column_reader.cpp:932
#9 0x55918d85547c in doris::segment_v2::FileColumnIterator::seek_to_ordinal(unsigned long) /mnt/disk1/happen/doris-verion/doris/be/src/olap/rowset/segment_v2/column_reader.cpp:654
#10 0x55918dade6c2 in doris::segment_v2::SegmentIterator::_seek_columns(std::vector<unsigned int, std::allocator<unsigned int> > const&, unsigned int) /mnt/disk1/happen/doris-verion/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:611
#11 0x55918dae5735 in doris::segment_v2::SegmentIterator::_read_columns_by_index(unsigned int, unsigned int&, bool) /mnt/disk1/happen/doris-verion/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:983
#12 0x55918dae8b5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) /mnt/disk1/happen/doris-verion/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1143
#13 0x55919a88cea2 in doris::vectorized::VUnionIterator::next_batch(doris::vectorized::Block*) /mnt/disk1/happen/doris-verion/doris/be/src/vec/olap/vgeneric_iterators.cpp:383
#14 0x55918e690ac5 in doris::BetaRowsetReader::next_block(doris::vectorized::Block*) /mnt/disk1/happen/doris-verion/doris/be/src/olap/rowset/beta_rowset_reader.cpp:298
#15 0x55919a8ad395 in doris::vectorized::VCollectIterator::Level0Iterator::next(doris::vectorized::Block*) /mnt/disk1/happen/doris-verion/doris/be/src/vec/olap/vcollect_iterator.cpp:267
#16 0x55919a8b1f97 in doris::vectorized::VCollectIterator::Level1Iterator::_normal_next(doris::vectorized::Block*) /mnt/disk1/happen/doris-verion/doris/be/src/vec/olap/vcollect_iterator.cpp:533
#17 0x55919a8af001 in doris::vectorized::VCollectIterator::Level1Iterator::next(doris::vectorized::Block*) /mnt/disk1/happen/doris-verion/doris/be/src/vec/olap/vcollect_iterator.cpp:359
#18 0x55919a8ab63e in doris::vectorized::VCollectIterator::next(doris::vectorized::Block*) /mnt/disk1/happen/doris-verion/doris/be/src/vec/olap/vcollect_iterator.cpp:187
#19 0x55919a8c7ed4 in doris::vectorized::BlockReader::_direct_next_block(doris::vectorized::Block*, doris::MemPool*, doris::ObjectPool*, bool*) /mnt/disk1/happen/doris-verion/doris/be/src/vec/olap/block_reader.cpp:167
#20 0x55918d33c620 in doris::vectorized::BlockReader::next_block_with_aggregation(doris::vectorized::Block*, doris::MemPool*, doris::ObjectPool*, bool*) /mnt/disk1/happen/doris-verion/doris/be/src/vec/olap/block_reader.h:45
#21 0x55919b10b053 in doris::vectorized::NewOlapScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) /mnt/disk1/happen/doris-verion/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:315
#22 0x55919afcee80 in doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) /mnt/disk1/happen/doris-verion/doris/be/src/vec/exec/scan/vscanner.cpp:53
#23 0x55919afa77b3 in doris::vectorized::ScannerScheduler::_scanner_scan(doris::vectorized::ScannerScheduler*, doris::vectorized::ScannerContext*, doris::vectorized::VScanner*) /mnt/disk1/happen/doris-verion/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236
#24 0x55919afa5a21 in operator() /mnt/disk1/happen/doris-verion/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:143
#25 0x55919afa9a0b in __invoke_impl<void, doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::<lambda()>&> /mnt/disk1/happen/ldb_toolchain/include/c++/11/bits/invoke.h:61
#26 0x55919afa9683 in __invoke_r<void, doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::<lambda()>&> /mnt/disk1/happen/ldb_toolchain/include/c++/11/bits/invoke.h:111
#27 0x55919afa902b in _M_invoke /mnt/disk1/happen/ldb_toolchain/include/c++/11/bits/std_function.h:291
#28 0x55918e7ebe95 in std::function<void ()>::operator()() const /mnt/disk1/happen/ldb_toolchain/include/c++/11/bits/std_function.h:560
#29 0x55918f34f89f in doris::FunctionRunnable::run() /mnt/disk1/happen/doris-verion/doris/be/src/util/threadpool.cpp:45
```
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14984: [Bug](DictoryColumn) reverse the _codes.size() replace _reserve_size
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #14984:
URL: https://github.com/apache/doris/pull/14984#issuecomment-1345134304
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #14984: [Bug](DictoryColumn) reverse the _codes.size() replace _reserve_size
Posted by GitBox <gi...@apache.org>.
yiguolei merged PR #14984:
URL: https://github.com/apache/doris/pull/14984
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #14984: [Bug](DictoryColumn) reverse the _codes.size() replace _reserve_size
Posted by GitBox <gi...@apache.org>.
hello-stephen commented on PR #14984:
URL: https://github.com/apache/doris/pull/14984#issuecomment-1345142499
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 37.31 seconds
load time: 453 seconds
storage size: 17123356221 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221210052057_clickbench_pr_61265.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org