Kafka consumers freeze

["Sullivan, Tim" <TI...@homedepot.com>]

            Is there a way I can proactively check my consumers to see if they are consuming? Periodically some or all of my consumers stop consuming. The only way I am made aware of this is when my down stream feeds folks alert me that their data isn’t flowing into Kafka. My normal solution is to bump the kafka servers and then they begin to consume.

            Any help will be greatly appreciated.


Tim Sullivan
Be Well

Sr. Software Engineer
Supply Chain – IT | Data & Analytics
The Home Depot
2250 Newmarket Parkway | Atlanta, GA 30339

Work Cell 470-455-8346
Personal Cell 678-525-2583
Home Phone:  770-945-3315

[signature_1339376142]



________________________________

The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.

Re: Kafka consumers freeze

["M. Manna" <ma...@gmail.com>]

Hey Tim

On Fri, 31 Jan 2020 at 13:06, Sullivan, Tim <TI...@homedepot.com>
wrote:

>
>
>             Is there a way I can proactively check my consumers to see if
> they are consuming? Periodically some or all of my consumers stop
> consuming. The only way I am made aware of this is when my down stream
> feeds folks alert me that their data isn’t flowing into Kafka. My normal
> solution is to bump the kafka servers and then they begin to consume.
>
>
>
>             Any help will be greatly appreciated.
>

 Does using Burrow <https://github.com/linkedin/Burrow> help you?

Also, using cruise-control <https://github.com/linkedin/cruise-control> you
could dynamically rebalance the workload, if some consumers are not in
steady state.

Regards,


>
>
>
> Tim Sullivan
>
> Be Well
>
>
>
> Sr. Software Engineer
>
> Supply Chain – IT | Data & Analytics
>
> The Home Depot
>
> 2250 Newmarket Parkway | Atlanta, GA 30339
>
>
>
> Work Cell 470-455-8346
>
> Personal Cell 678-525-2583
>
> Home Phone:  770-945-3315
>
>
>
> [image: signature_1339376142]
>
>
>
>
>
> ------------------------------
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>