You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "hust-hhb (via GitHub)" <gi...@apache.org> on 2024/03/01 04:27:57 UTC

[PR] [improvement](group_comit) Add bvar to monitor the total wal count on disk [doris]

hust-hhb opened a new pull request, #31646:
URL: https://github.com/apache/doris/pull/31646

   ## Proposed changes
   Use a bvar to monitor the total wal count on disk.
   
   Issue Number: close #xxx
   
   <!--Describe your changes.-->
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](group_comit) Add bvar to monitor the total wal count on disk [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31646:
URL: https://github.com/apache/doris/pull/31646#issuecomment-1972485814

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](group_comit) Add bvar to monitor the total wal count on disk [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31646:
URL: https://github.com/apache/doris/pull/31646#issuecomment-1975557956

   TeamCity be ut coverage result:
    Function Coverage: 35.43% (8555/24146) 
    Line Coverage: 27.20% (69427/255230)
    Region Coverage: 26.41% (36010/136337)
    Branch Coverage: 23.25% (18394/79100)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/06caa9c4594cdd9df9468c456eaecf885312d3a3_06caa9c4594cdd9df9468c456eaecf885312d3a3/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](group_comit) Add bvar to monitor the total wal count on disk [doris]

Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #31646:
URL: https://github.com/apache/doris/pull/31646#issuecomment-2009082779

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](group_comit) Add bvar to monitor the total wal count on disk [doris]

Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #31646:
URL: https://github.com/apache/doris/pull/31646


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](group_comit) Add bvar to monitor the total wal count on disk [doris]

Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on code in PR #31646:
URL: https://github.com/apache/doris/pull/31646#discussion_r1508713110


##########
be/src/olap/wal/wal_manager.cpp:
##########
@@ -380,6 +386,8 @@ Status WalManager::_replay_background() {
         if (_first_replay.compare_exchange_strong(expected, false)) {
             RETURN_IF_ERROR(_load_wals());
         }
+        auto _total_wal_size = get_wal_queue_size(-1);
+        g_wal_total_count.set_value(_total_wal_size);

Review Comment:
   @mymeiyi already change



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](group_comit) Add bvar to monitor the total wal count on disk [doris]

Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #31646:
URL: https://github.com/apache/doris/pull/31646#issuecomment-2008721657

   run p0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](group_comit) Add bvar to monitor the total wal count on disk [doris]

Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #31646:
URL: https://github.com/apache/doris/pull/31646#issuecomment-2009023731

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](group_comit) Add bvar to monitor the total wal count on disk [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31646:
URL: https://github.com/apache/doris/pull/31646#issuecomment-1972540347

   TeamCity be ut coverage result:
    Function Coverage: 35.64% (8550/23988) 
    Line Coverage: 27.45% (69397/252802)
    Region Coverage: 26.62% (36014/135265)
    Branch Coverage: 23.44% (18397/78498)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/e024be77ac0e23072b660138ad9e7ec3e565d450_e024be77ac0e23072b660138ad9e7ec3e565d450/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](group_comit) Add bvar to monitor the total wal count on disk [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31646:
URL: https://github.com/apache/doris/pull/31646#issuecomment-1972856513

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](group_comit) Add bvar to monitor the total wal count on disk [doris]

Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #31646:
URL: https://github.com/apache/doris/pull/31646#issuecomment-2006493506

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](group_comit) Add bvar to monitor the total wal count on disk [doris]

Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on code in PR #31646:
URL: https://github.com/apache/doris/pull/31646#discussion_r1508697219


##########
be/src/olap/wal/wal_manager.cpp:
##########
@@ -380,6 +386,8 @@ Status WalManager::_replay_background() {
         if (_first_replay.compare_exchange_strong(expected, false)) {
             RETURN_IF_ERROR(_load_wals());
         }
+        auto _total_wal_size = get_wal_queue_size(-1);
+        g_wal_total_count.set_value(_total_wal_size);

Review Comment:
   can write to one line:
   g_wal_total_count.set_value(get_wal_queue_size(-1));



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](group_comit) Add bvar to monitor the total wal count on disk [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31646:
URL: https://github.com/apache/doris/pull/31646#issuecomment-2006328782

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](group_comit) Add bvar to monitor the total wal count on disk [doris]

Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #31646:
URL: https://github.com/apache/doris/pull/31646#issuecomment-1975538908

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](group_comit) Add bvar to monitor the total wal count on disk [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31646:
URL: https://github.com/apache/doris/pull/31646#issuecomment-2009111398

   TeamCity be ut coverage result:
    Function Coverage: 35.31% (8731/24730) 
    Line Coverage: 27.10% (71423/263531)
    Region Coverage: 26.35% (37071/140705)
    Branch Coverage: 23.25% (18951/81512)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/06caa9c4594cdd9df9468c456eaecf885312d3a3_06caa9c4594cdd9df9468c456eaecf885312d3a3/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](group_comit) Add bvar to monitor the total wal count on disk [doris]

Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #31646:
URL: https://github.com/apache/doris/pull/31646#issuecomment-2008564257

   run p0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](group_comit) Add bvar to monitor the total wal count on disk [doris]

Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #31646:
URL: https://github.com/apache/doris/pull/31646#issuecomment-1972485929

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](group_comit) Add bvar to monitor the total wal count on disk [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31646:
URL: https://github.com/apache/doris/pull/31646#issuecomment-2009061474

   TeamCity be ut coverage result:
    Function Coverage: 35.31% (8731/24730) 
    Line Coverage: 27.10% (71419/263531)
    Region Coverage: 26.35% (37070/140705)
    Branch Coverage: 23.25% (18952/81512)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/06caa9c4594cdd9df9468c456eaecf885312d3a3_06caa9c4594cdd9df9468c456eaecf885312d3a3/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](group_comit) Add bvar to monitor the total wal count on disk [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31646:
URL: https://github.com/apache/doris/pull/31646#issuecomment-2006589453

   TeamCity be ut coverage result:
    Function Coverage: 35.32% (8727/24710) 
    Line Coverage: 27.13% (71385/263144)
    Region Coverage: 26.38% (37036/140410)
    Branch Coverage: 23.27% (18935/81360)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/06caa9c4594cdd9df9468c456eaecf885312d3a3_06caa9c4594cdd9df9468c456eaecf885312d3a3/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](group_comit) Add bvar to monitor the total wal count on disk [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31646:
URL: https://github.com/apache/doris/pull/31646#issuecomment-1972835213

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](group_comit) Add bvar to monitor the total wal count on disk [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31646:
URL: https://github.com/apache/doris/pull/31646#issuecomment-1972492512

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](group_comit) Add bvar to monitor the total wal count on disk [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31646:
URL: https://github.com/apache/doris/pull/31646#issuecomment-1972519479

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 38165 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit e024be77ac0e23072b660138ad9e7ec3e565d450, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17662	4302	4053	4053
   q2	2052	150	141	141
   q3	10675	1006	958	958
   q4	4686	955	952	952
   q5	7624	2931	3608	2931
   q6	183	126	124	124
   q7	1307	842	824	824
   q8	9572	2109	2109	2109
   q9	7471	6590	6496	6496
   q10	8258	2593	2627	2593
   q11	421	220	216	216
   q12	794	316	318	316
   q13	17995	2989	2937	2937
   q14	276	254	250	250
   q15	476	443	432	432
   q16	472	392	393	392
   q17	955	878	842	842
   q18	6825	5815	5907	5815
   q19	1571	1535	1537	1535
   q20	559	302	283	283
   q21	7569	3676	3696	3676
   q22	812	290	295	290
   Total cold run time: 108215 ms
   Total hot run time: 38165 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	3991	4013	3971	3971
   q2	308	218	222	218
   q3	2975	2904	2931	2904
   q4	1831	1806	1798	1798
   q5	5240	5251	5248	5248
   q6	201	115	117	115
   q7	2242	1797	1808	1797
   q8	3215	3268	3295	3268
   q9	8496	8531	8566	8531
   q10	6126	3763	3740	3740
   q11	535	432	441	432
   q12	686	542	537	537
   q13	7109	2821	2789	2789
   q14	286	263	247	247
   q15	473	447	440	440
   q16	454	426	399	399
   q17	1691	1676	1671	1671
   q18	7926	7533	7243	7243
   q19	3206	1638	1625	1625
   q20	1946	1769	1731	1731
   q21	5060	4810	4715	4715
   q22	529	477	467	467
   Total cold run time: 64526 ms
   Total hot run time: 53886 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org