You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/06/12 10:26:07 UTC
[GitHub] [incubator-doris] phenixmzy opened a new issue #3853: Kafka-Topic长时间没有新消息到达,Routine Load任务会被卡住,没有重新被调度
phenixmzy opened a new issue #3853:
URL: https://github.com/apache/incubator-doris/issues/3853
**Describe the bug**
当Kafka-Topic长时间没有新消息到达时,Routine Load任务会被卡住, routine load任务一直处于waiting_task.
一旦kafka-topic有新消息到达后,routine load 任务仍然一直waiting_task:1,
在UI上的Progress列offset 也一直是{"0": OFFSET_END}, 登录mysql client 也没有查到新数据.
只有重启FE后, 卡住的routine load 任务才恢复runnning_task状态.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 commented on issue #3853: Kafka-Topic长时间没有新消息到达,Routine Load任务会被卡住,没有重新被调度
Posted by GitBox <gi...@apache.org>.
EmmyMiao87 commented on issue #3853:
URL: https://github.com/apache/incubator-doris/issues/3853#issuecomment-644653527
一开始创建routine load的时候,不要指定end offset 而是指定一个确定的值。这样就没问题了
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #3853: Kafka-Topic长时间没有新消息到达,Routine Load任务会被卡住,没有重新被调度
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #3853:
URL: https://github.com/apache/incubator-doris/issues/3853#issuecomment-643254185
你是否观察到 在 show routine Load 的结果中, abortedTaskNum 数量在上升?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #3853: Kafka-Topic长时间没有新消息到达,Routine Load任务会被卡住,没有重新被调度
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #3853:
URL: https://github.com/apache/incubator-doris/issues/3853#issuecomment-644640252
> > 你是否观察到 在 show routine Load 的结果中, abortedTaskNum 数量在上升?
>
> 有,基本每隔1-5秒就上升一次
>
> ## {"receivedBytes":0,"errorRows":0,"committedTaskNum":0,"loadedRows":0,"loadRowsRate":0,"abortedTaskNum":8759,"totalRows":0,"unselectedRows":0,"receivedBytesRate":0,"taskExecuteTimeMs":1}
你可以尝试手动设置一个有效的offset,而不是使用 OFFSET_END,然后观察是否开始消费
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] phenixmzy commented on issue #3853: Kafka-Topic长时间没有新消息到达,Routine Load任务会被卡住,没有重新被调度
Posted by GitBox <gi...@apache.org>.
phenixmzy commented on issue #3853:
URL: https://github.com/apache/incubator-doris/issues/3853#issuecomment-644638252
> 你是否观察到 在 show routine Load 的结果中, abortedTaskNum 数量在上升?
有,基本每隔1-5秒就上升一次
{"receivedBytes":0,"errorRows":0,"committedTaskNum":0,"loadedRows":0,"loadRowsRate":0,"abortedTaskNum":8759,"totalRows":0,"unselectedRows":0,"receivedBytesRate":0,"taskExecuteTimeMs":1}
--
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] phenixmzy commented on issue #3853: Kafka-Topic长时间没有新消息到达,Routine Load任务会被卡住,没有重新被调度
Posted by GitBox <gi...@apache.org>.
phenixmzy commented on issue #3853:
URL: https://github.com/apache/incubator-doris/issues/3853#issuecomment-652729975
thanks @EmmyMiao87
总结一下,这个问题主要是没有配置好,正确的操作应该是
FROM KAFKA
(
"kafka_broker_list" = "$kafka_host:9092",
"kafka_topic" = "kafka-topic-name",
"kafka_partitions" = "0,1,2",
"kafka_offsets" = "partition_0_offset, partition_1_offset_num, partition_2_offset_num",
"property.group.id" = "my.gid",
"property.client.id" = "my.cid"
);
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org