You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "CalvinKirs (via GitHub)" <gi...@apache.org> on 2023/12/14 08:57:01 UTC
[PR] [Chore](Job)Add the configuration of the maximum number of persistence tasks for the job [doris]
CalvinKirs opened a new pull request, #28411:
URL: https://github.com/apache/doris/pull/28411
…
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Chore](Job)Add the configuration of the maximum number of persistence tasks for the job [doris]
Posted by "CalvinKirs (via GitHub)" <gi...@apache.org>.
CalvinKirs commented on PR #28411:
URL: https://github.com/apache/doris/pull/28411#issuecomment-1855443760
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Chore](Job)Add the configuration of the maximum number of persistence tasks for the job [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba merged PR #28411:
URL: https://github.com/apache/doris/pull/28411
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Chore](Job)Add the configuration of the maximum number of persistence tasks for the job [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28411:
URL: https://github.com/apache/doris/pull/28411#issuecomment-1855540089
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit adf2043424c82dd34296978e1dd5520db21c1b29, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4704 4430 4444 4430
q2 362 162 158 158
q3 1459 1264 1263 1263
q4 1116 886 891 886
q5 3136 3188 3163 3163
q6 249 127 129 127
q7 981 498 476 476
q8 2204 2245 2207 2207
q9 6695 6639 6657 6639
q10 3221 3267 3296 3267
q11 325 204 200 200
q12 351 209 213 209
q13 4561 3791 3791 3791
q14 239 217 217 217
q15 564 522 515 515
q16 437 377 384 377
q17 1011 612 545 545
q18 7173 6902 7013 6902
q19 1526 1403 1445 1403
q20 551 304 303 303
q21 3143 2644 2609 2609
q22 353 283 286 283
Total cold run time: 44361 ms
Total hot run time: 39970 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4405 4375 4387 4375
q2 269 161 174 161
q3 3534 3528 3517 3517
q4 2387 2377 2377 2377
q5 5710 5725 5726 5725
q6 239 119 124 119
q7 2374 1872 1870 1870
q8 3514 3525 3526 3525
q9 9013 8933 8970 8933
q10 3911 3962 3987 3962
q11 511 381 396 381
q12 759 600 596 596
q13 4276 3515 3556 3515
q14 287 253 252 252
q15 573 528 514 514
q16 511 431 474 431
q17 1876 1863 1860 1860
q18 8722 8334 8266 8266
q19 1716 1725 1737 1725
q20 2270 1960 1959 1959
q21 6532 6175 6156 6156
q22 511 419 442 419
Total cold run time: 63900 ms
Total hot run time: 60638 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Chore](Job)Add the configuration of the maximum number of persistence tasks for the job [doris]
Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr commented on code in PR #28411:
URL: https://github.com/apache/doris/pull/28411#discussion_r1426428553
##########
fe/fe-common/src/main/java/org/apache/doris/common/Config.java:
##########
@@ -1585,7 +1585,10 @@ public class Config extends ConfigBase {
*/
@ConfField(description = {"任务堆积时用于存放定时任务的队列大小", "The number of timer jobs that can be queued."})
public static int job_dispatch_timer_job_queue_size = 1024;
-
+ @ConfField(description = {"一个 Job 的 task 最大的持久化数量，超过这个限制将会丢弃旧的 task 记录, 如果值 < 1, 将不会持久化。",
+ "Maximum number of persistence allowed per task in a job,exceeding which old tasks will be discarded，"
+ + "If the value is less than 1, it will not be persisted." })
+ public static int max_persistence_task_count = 100;
Review Comment:
what will happen,if set to -1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Chore](Job)Add the configuration of the maximum number of persistence tasks for the job [doris]
Posted by "CalvinKirs (via GitHub)" <gi...@apache.org>.
CalvinKirs commented on PR #28411:
URL: https://github.com/apache/doris/pull/28411#issuecomment-1857161205
run feut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Chore](Job)Add the configuration of the maximum number of persistence tasks for the job [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28411:
URL: https://github.com/apache/doris/pull/28411#issuecomment-1857142328
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Chore](Job)Add the configuration of the maximum number of persistence tasks for the job [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28411:
URL: https://github.com/apache/doris/pull/28411#issuecomment-1855494700
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.81 seconds
stream load tsv: 581 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17216986784 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Chore](Job)Add the configuration of the maximum number of persistence tasks for the job [doris]
Posted by "CalvinKirs (via GitHub)" <gi...@apache.org>.
CalvinKirs commented on code in PR #28411:
URL: https://github.com/apache/doris/pull/28411#discussion_r1426430867
##########
fe/fe-common/src/main/java/org/apache/doris/common/Config.java:
##########
@@ -1585,7 +1585,10 @@ public class Config extends ConfigBase {
*/
@ConfField(description = {"任务堆积时用于存放定时任务的队列大小", "The number of timer jobs that can be queued."})
public static int job_dispatch_timer_job_queue_size = 1024;
-
+ @ConfField(description = {"一个 Job 的 task 最大的持久化数量，超过这个限制将会丢弃旧的 task 记录, 如果值 < 1, 将不会持久化。",
+ "Maximum number of persistence allowed per task in a job,exceeding which old tasks will be discarded，"
+ + "If the value is less than 1, it will not be persisted." })
+ public static int max_persistence_task_count = 100;
Review Comment:
If the value is less than 1, it will not be persisted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Chore](Job)Add the configuration of the maximum number of persistence tasks for the job [doris]
Posted by "CalvinKirs (via GitHub)" <gi...@apache.org>.
CalvinKirs commented on PR #28411:
URL: https://github.com/apache/doris/pull/28411#issuecomment-1855452384
If persistence will be unified in the job later, I think it might be better to add a new method of persisting association records.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Chore](Job)Add the configuration of the maximum number of persistence tasks for the job [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28411:
URL: https://github.com/apache/doris/pull/28411#issuecomment-1857142290
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org