You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "zhiqiang-hhhh (via GitHub)" <gi...@apache.org> on 2024/06/14 09:10:49 UTC

[PR] [fix](memory allocate) Fix reinitialization of BlockReader [doris]

zhiqiang-hhhh opened a new pull request, #36314:
URL: https://github.com/apache/doris/pull/36314

   `ASSIGN_STATUS_IF_CATCH_EXCEPTION` will convert error code from `MEM_ALLOC_FAILED` to `MEM_LIMIT_EXCEEDED`, this will lead to problem like below:
   
   ```text
   F20240614 16:07:18.529981 3400047 delete_handler.cpp:388] Check failed: !_is_inited reinitialize delete handler.
   *** Check failure stack trace: ***
       @     0x561cdc597926  google::LogMessage::SendToLog()
       @     0x561cdc594370  google::LogMessage::Flush()
       @     0x561cdc598169  google::LogMessageFatal::~LogMessageFatal()
       @     0x561cacf6ba70  doris::DeleteHandler::init()
       @     0x561caf6e9298  doris::TabletReader::_init_delete_condition()
       @     0x561caf6e257c  doris::TabletReader::_init_params()
       @     0x561caf6e1a09  doris::TabletReader::init()
       @     0x561cd847917f  doris::vectorized::BlockReader::init()
       @     0x561cdbc466a4  doris::vectorized::NewOlapScanner::open()
       @     0x561cc3a1b623  doris::vectorized::ScannerScheduler::_scanner_scan()
       @     0x561cc3a1fae1  _ZZZZN5doris10vectorized16ScannerScheduler6submitESt10shared_ptrINS0_14ScannerContextEES2_INS0_8ScanTaskEEENK3$_1clEvENKUlvE_clEvENKUlvE_clEv
       @     0x561cc3a1f554  _ZZZN5doris10vectorized16ScannerScheduler6submitESt10shared_ptrINS0_14ScannerContextEES2_INS0_8ScanTaskEEENK3$_1clEvENKUlvE_clEv
       @     0x561cc3a1f3c5  _ZSt13__invoke_implIvRZZN5doris10vectorized16ScannerScheduler6submitESt10shared_ptrINS1_14ScannerContextEES3_INS1_8ScanTaskEEENK3$_1clEvEUlvE_JEET_St14__invoke_otherOT0_DpOT1_
       @     0x561cc3a1f365  _ZSt10__invoke_rIvRZZN5doris10vectorized16ScannerScheduler6submitESt10shared_ptrINS1_14ScannerContextEES3_INS1_8ScanTaskEEENK3$_1clEvEUlvE_JEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESC_E4typeEOSD_DpOSE_
       @     0x561cc3a1f18d  _ZNSt17_Function_handlerIFvvEZZN5doris10vectorized16ScannerScheduler6submitESt10shared_ptrINS2_14ScannerContextEES4_INS2_8ScanTaskEEENK3$_1clEvEUlvE_E9_M_invokeERKSt9_Any_data
       @     0x561cac9f8023  std::function<>::operator()()
       @     0x561cc3a333b5  _ZZN5doris10vectorized23SimplifiedScanScheduler16submit_scan_taskENS0_18SimplifiedScanTaskEENKUlvE_clEv
       @     0x561cc3a33395  _ZSt13__invoke_implIvRZN5doris10vectorized23SimplifiedScanScheduler16submit_scan_taskENS1_18SimplifiedScanTaskEEUlvE_JEET_St14__invoke_otherOT0_DpOT1_
       @     0x561cc3a33335  _ZSt10__invoke_rIvRZN5doris10vectorized23SimplifiedScanScheduler16submit_scan_taskENS1_18SimplifiedScanTaskEEUlvE_JEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EES7_E4typeEOS8_DpOS9_
       @     0x561cc3a3312d  _ZNSt17_Function_handlerIFvvEZN5doris10vectorized23SimplifiedScanScheduler16submit_scan_taskENS2_18SimplifiedScanTaskEEUlvE_E9_M_invokeERKSt9_Any_data
       @     0x561cac9f8023  std::function<>::operator()()
       @     0x561cb095d4e9  doris::FunctionRunnable::run()
       @     0x561cb094913e  doris::ThreadPool::dispatch_thread()
       @     0x561cb0970564  std::__invoke_impl<>()
       @     0x561cb097043d  std::__invoke<>()
       @     0x561cb09703c5  _ZNSt5_BindIFMN5doris10ThreadPoolEFvvEPS1_EE6__callIvJEJLm0EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
       @     0x561cb097026e  std::_Bind<>::operator()<>()
       @     0x561cb0970185  std::__invoke_impl<>()
       @     0x561cb0970125  _ZSt10__invoke_rIvRSt5_BindIFMN5doris10ThreadPoolEFvvEPS2_EEJEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESA_E4typeEOSB_DpOSC_
       @     0x561cb096fdcd  std::_Function_handler<>::_M_invoke()
       @     0x561cac9f8023  std::function<>::operator()()
       @     0x561cb091666c  doris::Thread::supervise_thread()
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](memory allocate) Fix reinitialization of TabletReader [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36314:
URL: https://github.com/apache/doris/pull/36314#issuecomment-2167927355

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](memory allocate) Fix reinitialization of TabletReader [doris]

Posted by "zhiqiang-hhhh (via GitHub)" <gi...@apache.org>.
zhiqiang-hhhh commented on PR #36314:
URL: https://github.com/apache/doris/pull/36314#issuecomment-2167603726

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](memory allocate) Fix reinitialization of TabletReader [doris]

Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #36314:
URL: https://github.com/apache/doris/pull/36314


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](memory allocate) Fix reinitialization of TabletReader [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36314:
URL: https://github.com/apache/doris/pull/36314#issuecomment-2167688265

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 173785 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 1d078db7ff5656e784d89d2f429c4af5d444e749, data reload: false
   
   query1	921	382	385	382
   query2	6458	2359	2417	2359
   query3	6645	205	204	204
   query4	19787	17274	17263	17263
   query5	3595	458	466	458
   query6	243	152	151	151
   query7	4573	299	287	287
   query8	320	298	293	293
   query9	8481	2403	2387	2387
   query10	582	292	293	292
   query11	10701	10101	9973	9973
   query12	117	85	82	82
   query13	1631	357	360	357
   query14	9998	7674	7648	7648
   query15	243	186	189	186
   query16	7846	258	255	255
   query17	1948	530	517	517
   query18	1994	267	288	267
   query19	199	155	148	148
   query20	92	82	81	81
   query21	211	132	126	126
   query22	4530	4214	4185	4185
   query23	33705	33502	33619	33502
   query24	11222	3055	2944	2944
   query25	611	365	377	365
   query26	925	155	158	155
   query27	2385	332	323	323
   query28	6714	2119	2121	2119
   query29	896	635	632	632
   query30	226	152	151	151
   query31	946	789	760	760
   query32	93	58	58	58
   query33	749	285	274	274
   query34	1091	485	477	477
   query35	724	638	645	638
   query36	1120	949	964	949
   query37	146	74	72	72
   query38	2919	2824	2809	2809
   query39	924	844	847	844
   query40	220	129	131	129
   query41	59	60	58	58
   query42	115	106	103	103
   query43	592	583	531	531
   query44	1260	729	733	729
   query45	208	162	173	162
   query46	1086	731	730	730
   query47	1868	1782	1790	1782
   query48	371	299	306	299
   query49	868	415	421	415
   query50	765	394	399	394
   query51	6788	6662	6645	6645
   query52	111	90	95	90
   query53	355	299	292	292
   query54	911	442	447	442
   query55	75	77	71	71
   query56	288	267	280	267
   query57	1102	1062	1047	1047
   query58	255	253	248	248
   query59	3255	3288	2982	2982
   query60	309	283	306	283
   query61	112	110	105	105
   query62	615	444	446	444
   query63	321	292	287	287
   query64	8932	2356	1809	1809
   query65	3193	3093	3119	3093
   query66	783	335	327	327
   query67	15489	14893	14747	14747
   query68	7811	537	531	531
   query69	687	466	388	388
   query70	1169	1119	1025	1025
   query71	523	261	267	261
   query72	8923	5275	5419	5275
   query73	814	322	319	319
   query74	5871	5412	5432	5412
   query75	5014	2655	2659	2655
   query76	4303	943	885	885
   query77	765	293	286	286
   query78	10459	10050	9712	9712
   query79	4038	514	519	514
   query80	1016	464	464	464
   query81	557	219	224	219
   query82	782	105	105	105
   query83	380	170	170	170
   query84	276	88	95	88
   query85	1258	269	269	269
   query86	439	323	303	303
   query87	3239	3037	3094	3037
   query88	4643	2356	2330	2330
   query89	497	383	384	383
   query90	1913	196	189	189
   query91	128	99	99	99
   query92	65	49	49	49
   query93	4852	501	500	500
   query94	1184	185	193	185
   query95	406	309	315	309
   query96	606	271	261	261
   query97	3165	3085	3050	3050
   query98	223	198	194	194
   query99	1205	834	851	834
   Total cold run time: 284028 ms
   Total hot run time: 173785 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](memory allocate) Fix reinitialization of TabletReader [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36314:
URL: https://github.com/apache/doris/pull/36314#issuecomment-2167697510

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 29.88 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 1d078db7ff5656e784d89d2f429c4af5d444e749, data reload: false
   
   query1	0.04	0.03	0.04
   query2	0.08	0.04	0.03
   query3	0.24	0.05	0.04
   query4	1.68	0.06	0.07
   query5	0.51	0.47	0.48
   query6	1.15	0.74	0.72
   query7	0.02	0.02	0.01
   query8	0.05	0.04	0.04
   query9	0.54	0.50	0.50
   query10	0.53	0.55	0.54
   query11	0.15	0.11	0.11
   query12	0.15	0.12	0.12
   query13	0.60	0.60	0.61
   query14	0.78	0.78	0.81
   query15	0.85	0.81	0.81
   query16	0.37	0.34	0.36
   query17	0.99	1.01	1.03
   query18	0.21	0.24	0.27
   query19	1.77	1.67	1.69
   query20	0.01	0.01	0.01
   query21	15.41	0.66	0.65
   query22	4.67	7.98	1.31
   query23	18.28	1.41	1.34
   query24	2.18	0.21	0.23
   query25	0.16	0.07	0.08
   query26	0.25	0.18	0.17
   query27	0.08	0.08	0.08
   query28	13.24	1.00	0.98
   query29	12.63	3.26	3.26
   query30	0.25	0.07	0.06
   query31	2.86	0.39	0.38
   query32	3.30	0.46	0.46
   query33	2.83	2.90	2.94
   query34	17.19	4.39	4.40
   query35	4.48	4.45	4.49
   query36	0.63	0.48	0.46
   query37	0.18	0.15	0.15
   query38	0.15	0.15	0.14
   query39	0.05	0.04	0.04
   query40	0.18	0.13	0.14
   query41	0.10	0.04	0.05
   query42	0.05	0.05	0.04
   query43	0.04	0.04	0.04
   Total cold run time: 109.91 s
   Total hot run time: 29.88 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](memory allocate) Fix reinitialization of BlockReader [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36314:
URL: https://github.com/apache/doris/pull/36314#issuecomment-2167598483

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
   
   Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
   See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](memory allocate) Fix reinitialization of TabletReader [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36314:
URL: https://github.com/apache/doris/pull/36314#issuecomment-2167669425

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 39503 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 1d078db7ff5656e784d89d2f429c4af5d444e749, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17641	4337	4289	4289
   q2	2029	194	189	189
   q3	10445	1102	1189	1102
   q4	10190	786	749	749
   q5	7567	2619	2639	2619
   q6	217	135	134	134
   q7	986	595	592	592
   q8	9213	2041	2069	2041
   q9	8978	6452	6482	6452
   q10	8944	3796	3706	3706
   q11	448	242	242	242
   q12	459	233	220	220
   q13	17761	2990	2976	2976
   q14	258	220	217	217
   q15	524	477	494	477
   q16	523	370	381	370
   q17	958	697	673	673
   q18	7989	7298	7388	7298
   q19	4702	1567	1456	1456
   q20	654	321	322	321
   q21	4910	3106	3051	3051
   q22	383	329	330	329
   Total cold run time: 115779 ms
   Total hot run time: 39503 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4402	4218	4206	4206
   q2	377	260	257	257
   q3	2979	2719	2969	2719
   q4	1999	1723	1746	1723
   q5	5594	5489	5517	5489
   q6	230	127	129	127
   q7	2165	1801	1856	1801
   q8	3260	3428	3419	3419
   q9	8674	8623	8782	8623
   q10	4097	3924	3764	3764
   q11	609	475	467	467
   q12	795	635	642	635
   q13	17040	3194	3144	3144
   q14	301	269	285	269
   q15	545	472	493	472
   q16	495	420	436	420
   q17	1804	1517	1480	1480
   q18	7935	8070	7715	7715
   q19	1809	1665	1566	1566
   q20	2061	1904	1881	1881
   q21	5098	4938	4760	4760
   q22	625	549	515	515
   Total cold run time: 72894 ms
   Total hot run time: 55452 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](memory allocate) Fix reinitialization of TabletReader [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36314:
URL: https://github.com/apache/doris/pull/36314#issuecomment-2167927253

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](memory allocate) Fix reinitialization of TabletReader [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36314:
URL: https://github.com/apache/doris/pull/36314#issuecomment-2167649335

   TeamCity be ut coverage result:
    Function Coverage: 36.45% (8988/24661) 
    Line Coverage: 28.02% (73677/262936)
    Region Coverage: 27.49% (38276/139213)
    Branch Coverage: 24.19% (19515/80660)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/1d078db7ff5656e784d89d2f429c4af5d444e749_1d078db7ff5656e784d89d2f429c4af5d444e749/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](memory allocate) Fix reinitialization of TabletReader [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36314:
URL: https://github.com/apache/doris/pull/36314#issuecomment-2167608491

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org