You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/04/22 03:28:32 UTC
[GitHub] [incubator-doris] xy720 opened a new pull request, #9164: [Bug] Fix bug that new coming routine load tasks are rejected all the time and report TOO_MANY_TASK error
xy720 opened a new pull request, #9164:
URL: https://github.com/apache/incubator-doris/pull/9164
# Proposed changes
Issue Number: close #9162
## Problem Summary:
This bug is caused by the data consumer group get by this routine load task.
Sometimes, the kafka's begin offsets is not set, so that 0 data consumer was assign in this data consumer group.
Then the routine load task will be stuck in data consumer group's start_all function, and the fragment corresponding to this task will never be closed.
As the result, this routine load task will never be removed from task pool so that the task pool will full soon and reject any new coming routine load task.
## Checklist(Required)
1. Does it affect the original behavior: (No)
2. Has unit tests been added: (No Need)
3. Has document been added or modified: (No Need)
4. Does it need to update dependencies: (No)
5. Are there any changes that cannot be rolled back: (No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] pengxiangyu commented on a diff in pull request #9164: [Bug] Fix bug that new coming routine load tasks are rejected all the time and report TOO_MANY_TASK error
Posted by GitBox <gi...@apache.org>.
pengxiangyu commented on code in PR #9164:
URL: https://github.com/apache/incubator-doris/pull/9164#discussion_r859414107
##########
be/src/runtime/routine_load/data_consumer_pool.cpp:
##########
@@ -68,11 +68,17 @@ Status DataConsumerPool::get_consumer_grp(StreamLoadContext* ctx,
}
DCHECK(ctx->kafka_info);
+ if (ctx->kafka_info->begin_offset.size() == 0) {
+ return Status::InternalError(
Review Comment:
This is not an error, if a pipe has not data for a long time, it will write a lot of error logs.
Check it before call get_consumer_grp maybe better. Otherwise check return value and don't write warning log each time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9164: [Bug] Fix bug that new coming routine load tasks are rejected all the time and report TOO_MANY_TASK error
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #9164:
URL: https://github.com/apache/incubator-doris/pull/9164#issuecomment-1110523538
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9164: [Bug] Fix bug that new coming routine load tasks are rejected all the time and report TOO_MANY_TASK error
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #9164:
URL: https://github.com/apache/incubator-doris/pull/9164#issuecomment-1106164713
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9164: [Bug] Fix bug that new coming routine load tasks are rejected all the time and report TOO_MANY_TASK error
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #9164:
URL: https://github.com/apache/incubator-doris/pull/9164#issuecomment-1109772071
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xy720 commented on a diff in pull request #9164: [Bug] Fix bug that new coming routine load tasks are rejected all the time and report TOO_MANY_TASK error
Posted by GitBox <gi...@apache.org>.
xy720 commented on code in PR #9164:
URL: https://github.com/apache/incubator-doris/pull/9164#discussion_r858567367
##########
be/src/runtime/routine_load/data_consumer_pool.cpp:
##########
@@ -67,12 +67,18 @@ Status DataConsumerPool::get_consumer_grp(StreamLoadContext* ctx,
"PAUSE: Currently only support consumer group for Kafka data source");
}
DCHECK(ctx->kafka_info);
+ DCHECK(ctx->kafka_info->begin_offset.size() > 0);
Review Comment:
Ok. Now the task will fail in get_consumer_grp method if the size of begin_offset is equal to 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9164: [Bug] Fix bug that new coming routine load tasks are rejected all the time and report TOO_MANY_TASK error
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #9164:
URL: https://github.com/apache/incubator-doris/pull/9164#issuecomment-1106164601
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a diff in pull request #9164: [Bug] Fix bug that new coming routine load tasks are rejected all the time and report TOO_MANY_TASK error
Posted by GitBox <gi...@apache.org>.
morningman commented on code in PR #9164:
URL: https://github.com/apache/incubator-doris/pull/9164#discussion_r858454251
##########
be/src/runtime/routine_load/data_consumer_pool.cpp:
##########
@@ -67,12 +67,18 @@ Status DataConsumerPool::get_consumer_grp(StreamLoadContext* ctx,
"PAUSE: Currently only support consumer group for Kafka data source");
}
DCHECK(ctx->kafka_info);
+ DCHECK(ctx->kafka_info->begin_offset.size() > 0);
Review Comment:
I think we should let the task fail directly if `ctx->kafka_info->begin_offset.size() == 0`, so that it can be easier to find error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xy720 commented on a diff in pull request #9164: [Bug] Fix bug that new coming routine load tasks are rejected all the time and report TOO_MANY_TASK error
Posted by GitBox <gi...@apache.org>.
xy720 commented on code in PR #9164:
URL: https://github.com/apache/incubator-doris/pull/9164#discussion_r857919153
##########
be/src/runtime/routine_load/data_consumer_pool.cpp:
##########
@@ -67,12 +67,18 @@ Status DataConsumerPool::get_consumer_grp(StreamLoadContext* ctx,
"PAUSE: Currently only support consumer group for Kafka data source");
}
DCHECK(ctx->kafka_info);
+ DCHECK(ctx->kafka_info->begin_offset.size() > 0);
Review Comment:
In the create statement example below, you can see
The user didn't specify the custom partitions.
So that 1、Fe will get all kafka partitions from server in routine load's scheduler.
The user set the default offset by datetime.
So that 2、Fe will get kafka offset by time from server in routine load's scheduler.
When 1 is success, meanwhile 2 is failed, the progress of this routine load may not contains any partitions and offsets.
Nevertheless, since newCurrentKafkaPartition which is get by kafka server may be always equal to currentKafkaPartitions, the wrong progress will never be updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a diff in pull request #9164: [Bug] Fix bug that new coming routine load tasks are rejected all the time and report TOO_MANY_TASK error
Posted by GitBox <gi...@apache.org>.
morningman commented on code in PR #9164:
URL: https://github.com/apache/incubator-doris/pull/9164#discussion_r855928956
##########
be/src/runtime/routine_load/data_consumer_pool.cpp:
##########
@@ -67,12 +67,18 @@ Status DataConsumerPool::get_consumer_grp(StreamLoadContext* ctx,
"PAUSE: Currently only support consumer group for Kafka data source");
}
DCHECK(ctx->kafka_info);
+ DCHECK(ctx->kafka_info->begin_offset.size() > 0);
Review Comment:
`DCHECK` means it must `> 0`;
So there should not be `consumer_num < 1`.
Why the begin offset is empty? Is it normal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #9164: [Bug] Fix bug that new coming routine load tasks are rejected all the time and report TOO_MANY_TASK error
Posted by GitBox <gi...@apache.org>.
morningman merged PR #9164:
URL: https://github.com/apache/incubator-doris/pull/9164
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org