[GitHub] [incubator-doris] xonze opened a new issue #1757: kafka routine load kafka consume failed Broker: Offset out of range

[GitBox <gi...@apache.org>]

xonze opened a new issue #1757: kafka routine load kafka consume failed Broker: Offset out of range
URL: https://github.com/apache/incubator-doris/issues/1757
 
 
   **使用kafka routine load数据过程中的Offset out of range问题**
   
   可能由于大批量导入历史数据的原因，中间出现过个别机器有commit失败的情况，时间久了就会出现各个BE所分配的partition消费offset不均衡差异过大，最终累计时间久了kafka中旧的数据会过期自动清理，这个时候就会出现个别BE对应partition消费的当前的offset已经没有数据了（就是BE消费太慢，kafka已经过期的数据的offset它才在消费），这个时候就会报以下错误，并且会暂停routine load
   
   `kafka consume failed: b2489f7e1dc83663-8a71e3ae834b3f9a, msg: Broker: Offset out of range`
   
   **这里有两个问题：**
   
   1. routine load过程中没有动态的考虑offset均衡问题，每个BE始终分配的那几个固定的partition，
   2. 消费kafka中已经过期的offset对应的数据的时候应该跳过并继续往后消费，而不是报`Offset out of range`并暂停routine load
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org