You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "jacktengg (via GitHub)" <gi...@apache.org> on 2023/11/15 03:41:36 UTC
[PR] [fix](agg) fix coredump of multi distinct of decimal128I [doris]
jacktengg opened a new pull request, #27014:
URL: https://github.com/apache/doris/pull/27014
## Proposed changes
Issue Number: close #xxx
Test case:
```
CREATE TABLE `multi_distinct_agg_tab` (
`k1` bigint(20) NULL,
`k2` varchar(20) NULL,
`d1` DECIMAL(18, 0) NULL,
`d2` DECIMAL(38, 0) NULL
) ENGINE = OLAP DUPLICATE KEY(`k1`) DISTRIBUTED BY HASH(`k1`) BUCKETS 2 PROPERTIES (
"replication_allocation" = "tag.location.default: 1"
);
INSERT INTO
`multi_distinct_agg_tab` (`k1`, `k2`, `d1`, `d2`)
VALUES (1, 'aaa', 123, 356),(2, 'bbb', 123, 789), (3, 'ccc', 246, 789);
select
count(distinct d1),
count(distinct d2)
from
multi_distinct_agg_tab;
```
Coredump:
```
17:02:20 F20231114 16:55:35.946138 156798 assert_cast.h:57] Bad cast from type:doris::vectorized::ColumnDecimal<doris::vectorized::Decimal128I> to doris::vectorized::ColumnDecimal<doris::vectorized::Decimal<__int128> >
17:02:20 *** Check failure stack trace: ***
17:02:20 @ 0x5627046b3216 google::LogMessageFatal::~LogMessageFatal()
17:02:20 @ 0x5626da741024 assert_cast<>()
17:02:20 @ 0x5626e7c242a4 doris::vectorized::AggregateFunctionUniq<>::add_batch_single_place()
17:02:20 @ 0x5626e7c2c030 doris::vectorized::AggregateFunctionNullUnaryInline<>::add_batch_single_place()
17:02:20 @ 0x5626f14e13c8 doris::vectorized::AggFnEvaluator::execute_single_add()
17:02:20 @ 0x562703dde5b7 doris::pipeline::AggSinkLocalState<>::_execute_without_key()
17:02:20 @ 0x5627041df937 std::_Function_handler<>::_M_invoke()
17:02:20 @ 0x562703dcdb00 doris::pipeline::AggSinkOperatorX<>::sink()
17:02:20 @ 0x562704676257 doris::pipeline::PipelineXTask::execute()
17:02:20 @ 0x5627046931e2 doris::pipeline::TaskScheduler::_do_work()
17:02:20 @ 0x5626dae3276d doris::ThreadPool::dispatch_thread()
17:02:20 @ 0x5626dae0e7e9 doris::Thread::supervise_thread()
17:02:20 @ 0x7fa1d23ef609 start_thread
17:02:20 @ 0x7fa1d269c133 clone
17:02:20 @ (nil) (unknown)
17:02:20 *** Query id: 28287cc913164182-928abacb26a33665 ***
17:02:20 *** tablet id: 0 ***
17:02:20 *** Aborted at 1699952136 (unix time) try "date -d @1699952136" if you are using GNU date ***
17:02:20 *** Current BE git commitID: 3ce53c8 ***
17:02:20 *** SIGABRT unknown detail explain (@0x26349) received by PID 156489 (TID 156798 OR 0x7f9fee91a700) from PID 156489; stack trace: ***
17:02:20 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:417
17:02:20 1# 0x00007FA1D25C0090 in /lib/x86_64-linux-gnu/libc.so.6
17:02:20 2# raise at ../sysdeps/unix/sysv/linux/raise.c:51
17:02:20 3# abort at /build/glibc-SzIz7B/glibc-2.31/stdlib/abort.c:81
17:02:20 4# 0x00005627046BA11D in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
17:02:20 5# google::LogMessage::SendToLog() in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
17:02:20 6# google::LogMessage::Flush() in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
17:02:20 7# google::LogMessageFatal::~LogMessageFatal() in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
17:02:20 8# 0x00005626DA741024 in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
17:02:20 9# doris::vectorized::AggregateFunctionUniq<doris::vectorized::Decimal<__int128>, doris::vectorized::AggregateFunctionUniqExactData<__int128> >::add_batch_single_place(unsigned long, char*, doris::vectorized::IColumn const**, doris::vectorized::Arena*) const at /root/doris/be/src/vec/aggregate_functions/aggregate_function_uniq.h:176
17:02:20 10# doris::vectorized::AggregateFunctionNullUnaryInline<doris::vectorized::AggregateFunctionUniq<doris::vectorized::Decimal<__int128>, doris::vectorized::AggregateFunctionUniqExactData<__int128> >, false>::add_batch_single_place(unsigned long, char*, doris::vectorized::IColumn const**, doris::vectorized::Arena*) const in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
17:02:20 11# doris::vectorized::AggFnEvaluator::execute_single_add(doris::vectorized::Block*, char*, doris::vectorized::Arena*) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
17:02:20 12# doris::pipeline::AggSinkLocalState<doris::pipeline::AggDependency, doris::pipeline::BlockingAggSinkLocalState>::_execute_without_key(doris::vectorized::Block*) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
17:02:20 13# std::_Function_handler<doris::Status (doris::vectorized::Block*), std::_Bind_result<doris::Status, doris::Status (doris::pipeline::AggSinkLocalState<doris::pipeline::AggDependency, doris::pipeline::BlockingAggSinkLocalState>::*(doris::pipeline::AggSinkLocalState<doris::pipeline::AggDependency, doris::pipeline::BlockingAggSinkLocalState>*, std::_Placeholder<1>))(doris::vectorized::Block*)> >::_M_invoke(std::_Any_data const&, doris::vectorized::Block*&&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
17:02:20 14# doris::pipeline::AggSinkOperatorX<doris::pipeline::BlockingAggSinkLocalState>::sink(doris::RuntimeState*, doris::vectorized::Block*, doris::pipeline::SourceState) at /root/doris/be/src/pipeline/exec/aggregation_sink_operator.cpp:791
17:02:20 15# doris::pipeline::PipelineXTask::execute(bool*) at /root/doris/be/src/pipeline/pipeline_x/pipeline_x_task.cpp:261
17:02:20 16# doris::pipeline::TaskScheduler::_do_work(unsigned long) at /root/doris/be/src/pipeline/task_scheduler.cpp:262
```
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](agg) fix coredump of multi distinct of decimal128I [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27014:
URL: https://github.com/apache/doris/pull/27014#issuecomment-1811815228
TeamCity be ut coverage result:
Function Coverage: 36.80% (8419/22879)
Line Coverage: 29.28% (68406/233610)
Region Coverage: 27.88% (35352/126819)
Branch Coverage: 24.65% (18069/73296)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c1dc23d5228bfeb539aa46d3709f989e6da5b75a_c1dc23d5228bfeb539aa46d3709f989e6da5b75a/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](agg) fix coredump of multi distinct of decimal128I [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #27014:
URL: https://github.com/apache/doris/pull/27014#discussion_r1393718106
##########
be/src/vec/aggregate_functions/aggregate_function_uniq.cpp:
##########
@@ -48,6 +48,9 @@ AggregateFunctionPtr create_aggregate_function_uniq(const std::string& name,
} else if (which.is_decimal64()) {
return creator_without_type::create<AggregateFunctionUniq<Decimal64, Data<Int64>>>(
argument_types, result_is_nullable);
+ } else if (which.is_decimal128i()) {
Review Comment:
what about decimal256?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](agg) fix coredump of multi distinct of decimal128I [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27014:
URL: https://github.com/apache/doris/pull/27014#issuecomment-1811816724
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](agg) fix coredump of multi distinct of decimal128I [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27014:
URL: https://github.com/apache/doris/pull/27014#issuecomment-1811764644
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](agg) fix coredump of multi distinct of decimal128I [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27014:
URL: https://github.com/apache/doris/pull/27014#issuecomment-1811895070
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](agg) fix coredump of multi distinct of decimal128I [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27014:
URL: https://github.com/apache/doris/pull/27014#issuecomment-1811764690
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](agg) fix coredump of multi distinct of decimal128I [doris]
Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on PR #27014:
URL: https://github.com/apache/doris/pull/27014#issuecomment-1811811758
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](agg) fix coredump of multi distinct of decimal128I [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #27014:
URL: https://github.com/apache/doris/pull/27014
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](agg) fix coredump of multi distinct of decimal128I [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27014:
URL: https://github.com/apache/doris/pull/27014#issuecomment-1811895058
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](agg) fix coredump of multi distinct of decimal128I [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27014:
URL: https://github.com/apache/doris/pull/27014#issuecomment-1811910054
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.7 seconds
stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17096159419 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](agg) fix coredump of multi distinct of decimal128I [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27014:
URL: https://github.com/apache/doris/pull/27014#issuecomment-1811835377
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.3 seconds
stream load tsv: 553 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.5 seconds inserted 10000000 Rows, about 350K ops/s
storage size: 17097341188 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](agg) fix coredump of multi distinct of decimal128I [doris]
Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on PR #27014:
URL: https://github.com/apache/doris/pull/27014#issuecomment-1811761328
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](agg) fix coredump of multi distinct of decimal128I [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27014:
URL: https://github.com/apache/doris/pull/27014#issuecomment-1811874300
TeamCity be ut coverage result:
Function Coverage: 36.79% (8418/22879)
Line Coverage: 29.28% (68390/233610)
Region Coverage: 27.87% (35349/126819)
Branch Coverage: 24.65% (18065/73296)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9fb9160aa2178914164079216554b71a9b9139cb_9fb9160aa2178914164079216554b71a9b9139cb/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](agg) fix coredump of multi distinct of decimal128I [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27014:
URL: https://github.com/apache/doris/pull/27014#issuecomment-1811898839
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 9fb9160aa2178914164079216554b71a9b9139cb, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5249 5079 5129 5079
q2 372 145 159 145
q3 2036 2055 2003 2003
q4 1423 1383 1355 1355
q5 4015 3985 4025 3985
q6 258 134 130 130
q7 1462 884 911 884
q8 2789 2805 2781 2781
q9 9712 9647 9551 9551
q10 3450 3538 3534 3534
q11 384 250 259 250
q12 429 284 284 284
q13 4586 4178 4143 4143
q14 329 272 283 272
q15 599 558 545 545
q16 673 592 583 583
q17 1138 1092 1093 1092
q18 8036 7601 7581 7581
q19 1692 1699 1703 1699
q20 585 335 300 300
q21 4679 4346 4364 4346
q22 519 410 422 410
Total cold run time: 54415 ms
Total hot run time: 50952 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4988 5018 5012 5012
q2 346 224 234 224
q3 4072 4039 3980 3980
q4 2800 2801 2795 2795
q5 9621 9705 9632 9632
q6 248 122 127 122
q7 3103 2468 2533 2468
q8 4840 4819 4855 4819
q9 13244 13127 13028 13028
q10 4064 4155 4154 4154
q11 702 643 651 643
q12 1022 809 811 809
q13 4276 3903 3869 3869
q14 375 354 349 349
q15 620 549 549 549
q16 764 690 663 663
q17 3895 3872 3903 3872
q18 9755 9360 9555 9360
q19 1837 1773 1783 1773
q20 2416 2065 2054 2054
q21 8880 8719 8721 8719
q22 959 817 908 817
Total cold run time: 82827 ms
Total hot run time: 79711 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](agg) fix coredump of multi distinct of decimal128I [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27014:
URL: https://github.com/apache/doris/pull/27014#issuecomment-1811895018
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org