You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/01/07 10:30:36 UTC
[GitHub] [incubator-doris] caiconghui opened a new pull request #5205: [Load] Load job should not begin transaction when task queue in loadingLoadTaskScheduler is full to avoid txn timeout
caiconghui opened a new pull request #5205:
URL: https://github.com/apache/incubator-doris/pull/5205
## Proposed changes
Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue.
## Types of changes
What types of changes does your code introduce to Doris?
_Put an `x` in the boxes that apply_
- [] Bugfix (non-breaking change which fixes an issue)
- [] New feature (non-breaking change which adds functionality)
- [] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [] Documentation Update (if none of the other choices apply)
- [] Code refactor (Modify the code structure, format the code, etc...)
## Checklist
_Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._
- [x] I have create an issue on (Fix #5204), and have described the bug/feature there in detail
- [] Compiling and unit tests pass locally with my changes
- [] I have added tests that prove my fix is effective or that my feature works
- [] If this change need a document change, I have updated the document
- [] Any dependent changes have been merged
## Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on a change in pull request #5205: [Load] Load job should not begin transaction when task queue in loadingLoadTaskScheduler is full to avoid txn timeout
Posted by GitBox <gi...@apache.org>.
caiconghui commented on a change in pull request #5205:
URL: https://github.com/apache/incubator-doris/pull/5205#discussion_r559307133
##########
File path: fe/fe-core/src/main/java/org/apache/doris/load/loadv2/LoadJobScheduler.java
##########
@@ -63,12 +64,18 @@ protected void runAfterCatalogReady() {
private void process() throws InterruptedException {
while (true) {
- // take one load job from queue
- LoadJob loadJob = needScheduleJobs.poll();
- if (loadJob == null) {
+ if (!needScheduleJobs.isEmpty()) {
+ if (needScheduleJobs.peek() instanceof BrokerLoadJob && Catalog.getCurrentCatalog().getLoadingLoadTaskScheduler().isTaskQueueFull()) {
Review comment:
Spark Load not use async loading task pool
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #5205: [Load] Load job should not begin transaction when task queue in loadingLoadTaskScheduler is full to avoid txn timeout
Posted by GitBox <gi...@apache.org>.
morningman merged pull request #5205:
URL: https://github.com/apache/incubator-doris/pull/5205
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on a change in pull request #5205: [Load] Load job should not begin transaction when task queue in loadingLoadTaskScheduler is full to avoid txn timeout
Posted by GitBox <gi...@apache.org>.
caiconghui commented on a change in pull request #5205:
URL: https://github.com/apache/incubator-doris/pull/5205#discussion_r559345871
##########
File path: fe/fe-core/src/main/java/org/apache/doris/common/Config.java
##########
@@ -593,12 +593,19 @@
public static int max_running_txn_num_per_db = 100;
/**
- * The load task executor pool size. This pool size limits the max running load tasks.
- * Currently, it only limits the load task of broker load, pending and loading phases.
+ * The pending_load task executor pool size. This pool size limits the max running pending_load tasks.
+ * Currently, it only limits the pending_load task of broker load and spark load.
* It should be less than 'max_running_txn_num_per_db'
*/
@ConfField(mutable = false, masterOnly = true)
- public static int async_load_task_pool_size = 10;
+ public static int async_pending_load_task_pool_size = 5;
Review comment:
> And the `desired_max_waiting_jobs` is useless?
desired_max_waiting_jobs is used to limit blocking queue size
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #5205: [Load] Load job should not begin transaction when task queue in loadingLoadTaskScheduler is full to avoid txn timeout
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #5205:
URL: https://github.com/apache/incubator-doris/pull/5205#discussion_r559094294
##########
File path: fe/fe-core/src/main/java/org/apache/doris/common/Config.java
##########
@@ -593,12 +593,19 @@
public static int max_running_txn_num_per_db = 100;
/**
- * The load task executor pool size. This pool size limits the max running load tasks.
- * Currently, it only limits the load task of broker load, pending and loading phases.
+ * The pending_load task executor pool size. This pool size limits the max running pending_load tasks.
+ * Currently, it only limits the pending_load task of broker load and spark load.
* It should be less than 'max_running_txn_num_per_db'
*/
@ConfField(mutable = false, masterOnly = true)
- public static int async_load_task_pool_size = 10;
+ public static int async_pending_load_task_pool_size = 5;
Review comment:
Better to keep the config name. Or made some compatibility.
Some other users may modify this config to enhance their load ability.
If the name changed, their config will be disable and fall back to default value, this will cause problem
##########
File path: fe/fe-core/src/main/java/org/apache/doris/load/loadv2/LoadJobScheduler.java
##########
@@ -63,12 +64,18 @@ protected void runAfterCatalogReady() {
private void process() throws InterruptedException {
while (true) {
- // take one load job from queue
- LoadJob loadJob = needScheduleJobs.poll();
- if (loadJob == null) {
+ if (!needScheduleJobs.isEmpty()) {
+ if (needScheduleJobs.peek() instanceof BrokerLoadJob && Catalog.getCurrentCatalog().getLoadingLoadTaskScheduler().isTaskQueueFull()) {
Review comment:
Broker or Spark Load？
##########
File path: fe/fe-core/src/main/java/org/apache/doris/common/Config.java
##########
@@ -593,12 +593,19 @@
public static int max_running_txn_num_per_db = 100;
/**
- * The load task executor pool size. This pool size limits the max running load tasks.
- * Currently, it only limits the load task of broker load, pending and loading phases.
+ * The pending_load task executor pool size. This pool size limits the max running pending_load tasks.
+ * Currently, it only limits the pending_load task of broker load and spark load.
* It should be less than 'max_running_txn_num_per_db'
*/
@ConfField(mutable = false, masterOnly = true)
- public static int async_load_task_pool_size = 10;
+ public static int async_pending_load_task_pool_size = 5;
Review comment:
And the `desired_max_waiting_jobs` is useless?
##########
File path: fe/fe-core/src/main/java/org/apache/doris/common/Config.java
##########
@@ -593,12 +593,19 @@
public static int max_running_txn_num_per_db = 100;
/**
- * The load task executor pool size. This pool size limits the max running load tasks.
- * Currently, it only limits the load task of broker load, pending and loading phases.
+ * The pending_load task executor pool size. This pool size limits the max running pending_load tasks.
+ * Currently, it only limits the pending_load task of broker load and spark load.
* It should be less than 'max_running_txn_num_per_db'
*/
@ConfField(mutable = false, masterOnly = true)
- public static int async_load_task_pool_size = 10;
+ public static int async_pending_load_task_pool_size = 5;
+
+ /**
+ * The loading_load task executor pool size. This pool size limits the max running loading_load tasks.
+ * Currently, it only limits the loading_load task of broker load.
+ */
+ @ConfField(mutable = false, masterOnly = true)
+ public static int async_loading_load_task_pool_size = 20;
Review comment:
I think pending number should be larger loading number？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on a change in pull request #5205: [Load] Load job should not begin transaction when task queue in loadingLoadTaskScheduler is full to avoid txn timeout
Posted by GitBox <gi...@apache.org>.
caiconghui commented on a change in pull request #5205:
URL: https://github.com/apache/incubator-doris/pull/5205#discussion_r559345635
##########
File path: fe/fe-core/src/main/java/org/apache/doris/common/Config.java
##########
@@ -593,12 +593,19 @@
public static int max_running_txn_num_per_db = 100;
/**
- * The load task executor pool size. This pool size limits the max running load tasks.
- * Currently, it only limits the load task of broker load, pending and loading phases.
+ * The pending_load task executor pool size. This pool size limits the max running pending_load tasks.
+ * Currently, it only limits the pending_load task of broker load and spark load.
* It should be less than 'max_running_txn_num_per_db'
*/
@ConfField(mutable = false, masterOnly = true)
- public static int async_load_task_pool_size = 10;
+ public static int async_pending_load_task_pool_size = 5;
+
+ /**
+ * The loading_load task executor pool size. This pool size limits the max running loading_load tasks.
+ * Currently, it only limits the loading_load task of broker load.
+ */
+ @ConfField(mutable = false, masterOnly = true)
+ public static int async_loading_load_task_pool_size = 20;
Review comment:
pending task is used to collect some schema info or preparement work， loading task is used to execute loading job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on a change in pull request #5205: [Load] Load job should not begin transaction when task queue in loadingLoadTaskScheduler is full to avoid txn timeout
Posted by GitBox <gi...@apache.org>.
caiconghui commented on a change in pull request #5205:
URL: https://github.com/apache/incubator-doris/pull/5205#discussion_r559307315
##########
File path: fe/fe-core/src/main/java/org/apache/doris/common/Config.java
##########
@@ -593,12 +593,19 @@
public static int max_running_txn_num_per_db = 100;
/**
- * The load task executor pool size. This pool size limits the max running load tasks.
- * Currently, it only limits the load task of broker load, pending and loading phases.
+ * The pending_load task executor pool size. This pool size limits the max running pending_load tasks.
+ * Currently, it only limits the pending_load task of broker load and spark load.
* It should be less than 'max_running_txn_num_per_db'
*/
@ConfField(mutable = false, masterOnly = true)
- public static int async_load_task_pool_size = 10;
+ public static int async_pending_load_task_pool_size = 5;
+
+ /**
+ * The loading_load task executor pool size. This pool size limits the max running loading_load tasks.
+ * Currently, it only limits the loading_load task of broker load.
+ */
+ @ConfField(mutable = false, masterOnly = true)
+ public static int async_loading_load_task_pool_size = 20;
Review comment:
pending task not execute heavy task，while loading task cost much time
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org