You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/01/17 08:19:19 UTC
[GitHub] [incubator-doris] morningman commented on a change in pull request #5205: [Load] Load job should not begin transaction when task queue in loadingLoadTaskScheduler is full to avoid txn timeout
morningman commented on a change in pull request #5205:
URL: https://github.com/apache/incubator-doris/pull/5205#discussion_r559094294
##########
File path: fe/fe-core/src/main/java/org/apache/doris/common/Config.java
##########
@@ -593,12 +593,19 @@
public static int max_running_txn_num_per_db = 100;
/**
- * The load task executor pool size. This pool size limits the max running load tasks.
- * Currently, it only limits the load task of broker load, pending and loading phases.
+ * The pending_load task executor pool size. This pool size limits the max running pending_load tasks.
+ * Currently, it only limits the pending_load task of broker load and spark load.
* It should be less than 'max_running_txn_num_per_db'
*/
@ConfField(mutable = false, masterOnly = true)
- public static int async_load_task_pool_size = 10;
+ public static int async_pending_load_task_pool_size = 5;
Review comment:
Better to keep the config name. Or made some compatibility.
Some other users may modify this config to enhance their load ability.
If the name changed, their config will be disable and fall back to default value, this will cause problem
##########
File path: fe/fe-core/src/main/java/org/apache/doris/load/loadv2/LoadJobScheduler.java
##########
@@ -63,12 +64,18 @@ protected void runAfterCatalogReady() {
private void process() throws InterruptedException {
while (true) {
- // take one load job from queue
- LoadJob loadJob = needScheduleJobs.poll();
- if (loadJob == null) {
+ if (!needScheduleJobs.isEmpty()) {
+ if (needScheduleJobs.peek() instanceof BrokerLoadJob && Catalog.getCurrentCatalog().getLoadingLoadTaskScheduler().isTaskQueueFull()) {
Review comment:
Broker or Spark Load？
##########
File path: fe/fe-core/src/main/java/org/apache/doris/common/Config.java
##########
@@ -593,12 +593,19 @@
public static int max_running_txn_num_per_db = 100;
/**
- * The load task executor pool size. This pool size limits the max running load tasks.
- * Currently, it only limits the load task of broker load, pending and loading phases.
+ * The pending_load task executor pool size. This pool size limits the max running pending_load tasks.
+ * Currently, it only limits the pending_load task of broker load and spark load.
* It should be less than 'max_running_txn_num_per_db'
*/
@ConfField(mutable = false, masterOnly = true)
- public static int async_load_task_pool_size = 10;
+ public static int async_pending_load_task_pool_size = 5;
Review comment:
And the `desired_max_waiting_jobs` is useless?
##########
File path: fe/fe-core/src/main/java/org/apache/doris/common/Config.java
##########
@@ -593,12 +593,19 @@
public static int max_running_txn_num_per_db = 100;
/**
- * The load task executor pool size. This pool size limits the max running load tasks.
- * Currently, it only limits the load task of broker load, pending and loading phases.
+ * The pending_load task executor pool size. This pool size limits the max running pending_load tasks.
+ * Currently, it only limits the pending_load task of broker load and spark load.
* It should be less than 'max_running_txn_num_per_db'
*/
@ConfField(mutable = false, masterOnly = true)
- public static int async_load_task_pool_size = 10;
+ public static int async_pending_load_task_pool_size = 5;
+
+ /**
+ * The loading_load task executor pool size. This pool size limits the max running loading_load tasks.
+ * Currently, it only limits the loading_load task of broker load.
+ */
+ @ConfField(mutable = false, masterOnly = true)
+ public static int async_loading_load_task_pool_size = 20;
Review comment:
I think pending number should be larger loading number？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org