What should my Consumer connect to Kafka/Zookeeper

[Jesse Whitham <je...@loyalty.co.nz>]

Hi Everyone,


I am trying to get some information around what my consumer should connect to. According to the docs sending messages should be directly to the Kafka broker and consuming messages should be from Zookeeper? I am really after a reason this is the case as I am aware that you can just consume directly from the Kafka broker.


Thanks in advance for any help!


Jesse Whitham

Devops Engineer

e. jesse.whitham@loyalty.co.nz<ma...@loyalty.co.nz>

w. loyalty.co.nz<http://www.loyalty.co.nz/>

[Loyalty NZ]

Loyalty New Zealand Ltd. PO Box 3451, New Zealand 6011

Level 3, NZX Building, 11 Cable Street, Wellington

Re: What should my Consumer connect to Kafka/Zookeeper

[David Esposito <da...@bettercloud.com.INVALID>]

Your consumers/producers will push/pull messages directly from kafka. More
specifically, to/from the broker that is responsibile for the partition
assigned to the consumer or corresponding to the published message key.

Others should correct me if I am wrong, but zookeeper was used to store
information about topics, consumers, and offsets (version 0.8.*-ish). In
newer versions of kafka (0.9.* and 0.10.*) those values are instead written
to internal topics for Kafka to manage consumers, topics, and offsets.

So much (possibly all?) of the functionality that zookeeper provided is
being deprecated in favor of storing that info in kafka.

Another thing to consider is your serialization strategy and contract
between producers and consumers. You should consider predefining schemas
for your topics. Confluent.io
<http://docs.confluent.io/2.0.1/quickstart.html> provides steps to start
using a schema registry
<http://docs.confluent.io/2.0.0/schema-registry/docs/intro.html> to enforce
producers. There are also libraries that leverage this schema registry to
provide fast and efficient serialization and deserialization. Check out Apache
Avro <https://cwiki.apache.org/confluence/display/AVRO/Index>.

Also checkout https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem


On Tue, Aug 30, 2016 at 7:16 PM, Jesse Whitham <je...@loyalty.co.nz>
wrote:

> Hi Everyone,
>
>
> I am trying to get some information around what my consumer should connect
> to. According to the docs sending messages should be directly to the Kafka
> broker and consuming messages should be from Zookeeper? I am really after a
> reason this is the case as I am aware that you can just consume directly
> from the Kafka broker.
>
>
> Thanks in advance for any help!
>
>
> Jesse Whitham
>
> Devops Engineer
>
> e. jesse.whitham@loyalty.co.nz<ma...@loyalty.co.nz>
>
> w. loyalty.co.nz<http://www.loyalty.co.nz/>
>
> [Loyalty NZ]
>
> Loyalty New Zealand Ltd. PO Box 3451, New Zealand 6011
>
> Level 3, NZX Building, 11 Cable Street, Wellington
>



-- 
*David Esposito* | Product Developer | m. 678-653-1225
3405 Piedmont Rd. NE, Suite 325, Atlanta, GA 30305
<http://www.bettercloud.com>
"We could do that better. We’ve got the right technology."
<https://www.bettercloud.com/monitor/meet-brandon-williams/?utm_source=bettercloud_email&utm_medium=email_signature&utm_campaign=workplace_innovators>
-
Meet Brandon Williams, our newest Modern Workplace Innovator