You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by "Vanshul.Chawla" <Va...@target.com> on 2016/05/17 23:41:17 UTC
Kafka consumer issue in Camel 2.17
Hello,
We are getting the below error in Kafka consumer for Camel 2.17 with Kafka 0.9.0.1
2016-05-17 14:43:06,269 [nsfeed-invoice]] INFO AbstractCoordinator - Marking the coordinator 2147483155 dead.
2016-05-17 14:43:06,274 [nsfeed-invoice]] INFO AbstractCoordinator - Marking the coordinator 2147483155 dead.
2016-05-17 14:43:06,279 [nsfeed-invoice]] INFO AbstractCoordinator - Marking the coordinator 2147483155 dead.
2016-05-17 14:43:06,283 [nsfeed-invoice]] INFO AbstractCoordinator - Marking the coordinator 2147483155 dead.
2016-05-17 14:43:06,290 [nsfeed-invoice]] INFO AbstractCoordinator - Marking the coordinator 2147483155 dead.
2016-05-17 14:43:06,295 [nsfeed-invoice]] INFO AbstractCoordinator - Marking the coordinator 2147483155 dead.
2016-05-17 14:43:06,301 [nsfeed-invoice]] INFO AbstractCoordinator - Marking the coordinator 2147483155 dead.
2016-05-17 14:43:06,306 [nsfeed-invoice]] INFO AbstractCoordinator - Marking the coordinator 2147483155 dead.
2016-05-17 14:43:06,311 [nsfeed-invoice]] INFO AbstractCoordinator - Marking the coordinator 2147483155 dead.
2016-05-17 14:43:06,570 [nsfeed-invoice]] ERROR ConsumerCoordinator - Error UNKNOWN_MEMBER_ID occurred while committing offsets for group ilCamelInvoiceSub
2016-05-17 14:43:06,570 [nsfeed-invoice]] WARN ConsumerCoordinator - Auto offset commit failed: Commit cannot be completed due to group rebalance
2016-05-17 14:43:06,582 [nsfeed-invoice]] ERROR ConsumerCoordinator - Error UNKNOWN_MEMBER_ID occurred while committing offsets for group ilCamelInvoiceSub
2016-05-17 14:43:06,582 [nsfeed-invoice]] WARN ConsumerCoordinator - Auto offset commit failed:
Show less
We have 4 consumers , each one for different topic.
3 works fine but 4th one gives this error and as offset is not commited, if consumer is started , it begins reading from starting.
Route config is as below:
<from
uri="kafka:{{financial.kafka.host}}?topic={{financial.order.kafka.topic}}&groupId=ilCamelOrderSub&autoOffsetReset=earliest&securityProtocol=SSL&serializerClass=org.apache.kafka.common.serialization.StringSerializer&sslTruststoreLocation={{kafka.truststorelocation}}&sslTruststorePassword={{kafka.truststorepassword}}&sslKeystoreLocation={{kafka.keystorelocation}}&sslKeystorePassword={{kafka.keystorepassword}}&consumersCount={{kafka.order.consumersCount}}" />
Please could someone help us here if they have faced same issue.
Vanshul
Re: Kafka consumer issue in Camel 2.17
Posted by Gerard Klijs <ge...@dizzit.com>.
Could be different things. If I understand correctly consuming with 3
threads, each with their own topic, but probably with the same groupId goed
fine, but when reading with 4 threads, each with their own topic you see an
error? How long before you see the error? Maybe you have a time-out because
either the client or the broker can't keep up?
It may be better to check if a consumer is still alive before trying to
commit the offset, as that seems to be te problem, but I'm not sure.
On Wed, May 18, 2016 at 1:48 AM Vanshul.Chawla <Va...@target.com>
wrote:
> Hello,
>
> We are getting the below error in Kafka consumer for Camel 2.17 with Kafka
> 0.9.0.1
>
> 2016-05-17 14:43:06,269 [nsfeed-invoice]] INFO AbstractCoordinator -
> Marking the coordinator 2147483155 dead.
> 2016-05-17 14:43:06,274 [nsfeed-invoice]] INFO AbstractCoordinator -
> Marking the coordinator 2147483155 dead.
> 2016-05-17 14:43:06,279 [nsfeed-invoice]] INFO AbstractCoordinator -
> Marking the coordinator 2147483155 dead.
> 2016-05-17 14:43:06,283 [nsfeed-invoice]] INFO AbstractCoordinator -
> Marking the coordinator 2147483155 dead.
> 2016-05-17 14:43:06,290 [nsfeed-invoice]] INFO AbstractCoordinator -
> Marking the coordinator 2147483155 dead.
> 2016-05-17 14:43:06,295 [nsfeed-invoice]] INFO AbstractCoordinator -
> Marking the coordinator 2147483155 dead.
> 2016-05-17 14:43:06,301 [nsfeed-invoice]] INFO AbstractCoordinator -
> Marking the coordinator 2147483155 dead.
> 2016-05-17 14:43:06,306 [nsfeed-invoice]] INFO AbstractCoordinator -
> Marking the coordinator 2147483155 dead.
> 2016-05-17 14:43:06,311 [nsfeed-invoice]] INFO AbstractCoordinator -
> Marking the coordinator 2147483155 dead.
> 2016-05-17 14:43:06,570 [nsfeed-invoice]] ERROR ConsumerCoordinator -
> Error UNKNOWN_MEMBER_ID occurred while committing offsets for group
> ilCamelInvoiceSub
> 2016-05-17 14:43:06,570 [nsfeed-invoice]] WARN ConsumerCoordinator - Auto
> offset commit failed: Commit cannot be completed due to group rebalance
> 2016-05-17 14:43:06,582 [nsfeed-invoice]] ERROR ConsumerCoordinator -
> Error UNKNOWN_MEMBER_ID occurred while committing offsets for group
> ilCamelInvoiceSub
> 2016-05-17 14:43:06,582 [nsfeed-invoice]] WARN ConsumerCoordinator - Auto
> offset commit failed:
> Show less
>
>
> We have 4 consumers , each one for different topic.
>
> 3 works fine but 4th one gives this error and as offset is not commited,
> if consumer is started , it begins reading from starting.
>
> Route config is as below:
>
> <from
>
> uri="kafka:{{financial.kafka.host}}?topic={{financial.order.kafka.topic}}&groupId=ilCamelOrderSub&autoOffsetReset=earliest&securityProtocol=SSL&serializerClass=org.apache.kafka.common.serialization.StringSerializer&sslTruststoreLocation={{kafka.truststorelocation}}&sslTruststorePassword={{kafka.truststorepassword}}&sslKeystoreLocation={{kafka.keystorelocation}}&sslKeystorePassword={{kafka.keystorepassword}}&consumersCount={{kafka.order.consumersCount}}"
> />
>
> Please could someone help us here if they have faced same issue.
>
> Vanshul
>
>