kafka producers and consumers on different machine

[Dimitris Pappas <dv...@gmail.com>]

Dear sir/madame

My name is Dimitris Pappas and i am a research assistant in Athena R.C. 
in athens.

I downloaded the standalone version of kafka and tried to create 
consumers and producers on other machines.

The producers and consumers built, which were listening to 'localhost' 
worked like a charm.

On the contrary, consumers and producers built on another machine,
trying to send data or consume data from the ip of the machine running
the kafka server and the zookeeper, did not perform as expected.
Is a there a simple guide to set up such a grid of machines?

the error that i get is the following on the kafka server:

[2016-04-20 17:34:10,244] ERROR [KafkaApi-0] error when handling request 
null (kafka.server.KafkaApis)
kafka.admin.AdminOperationException: replication factor: 3 larger than 
available brokers: 1
         at 
kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:77)
         at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:236)
         at kafka.server.KafkaApis$$anonfun$20.apply(KafkaApis.scala:565)
         at kafka.server.KafkaApis$$anonfun$20.apply(KafkaApis.scala:555)
         at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
         at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
         at scala.collection.immutable.Set$Set1.foreach(Set.scala:79)
         at 
scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
         at 
scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)
         at scala.collection.SetLike$class.map(SetLike.scala:92)
         at scala.collection.AbstractSet.map(Set.scala:47)
         at kafka.server.KafkaApis.getTopicMetadata(KafkaApis.scala:555)
         at 
kafka.server.KafkaApis.handleGroupCoordinatorRequest(KafkaApis.scala:702)
         at kafka.server.KafkaApis.handle(KafkaApis.scala:78)
         at 
kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
         at java.lang.Thread.run(Thread.java:745)

I would be grateful if you could help me with the forth-mentioned problem.


Thank you in advance
Dimitris Pappas

---
Αυτό το e-mail ελέγχθηκε για ιούς από το πρόγραμμα Avast antivirus.
https://www.avast.com/antivirus

Re: kafka producers and consumers on different machine

[Marko Bonaći <ma...@sematext.com>]

I'm assuming that you created a topic with replication factor 3, while
having only a single broker.
Try with replication factor 1 or add additional brokers.

Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>

On Wed, Apr 20, 2016 at 4:35 PM, Dimitris Pappas <dv...@gmail.com>
wrote:

> Dear sir/madame
>
> My name is Dimitris Pappas and i am a research assistant in Athena R.C. in
> athens.
>
> I downloaded the standalone version of kafka and tried to create consumers
> and producers on other machines.
>
> The producers and consumers built, which were listening to 'localhost'
> worked like a charm.
>
> On the contrary, consumers and producers built on another machine,
> trying to send data or consume data from the ip of the machine running
> the kafka server and the zookeeper, did not perform as expected.
> Is a there a simple guide to set up such a grid of machines?
>
> the error that i get is the following on the kafka server:
>
> [2016-04-20 17:34:10,244] ERROR [KafkaApi-0] error when handling request
> null (kafka.server.KafkaApis)
> kafka.admin.AdminOperationException: replication factor: 3 larger than
> available brokers: 1
>         at
> kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:77)
>         at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:236)
>         at kafka.server.KafkaApis$$anonfun$20.apply(KafkaApis.scala:565)
>         at kafka.server.KafkaApis$$anonfun$20.apply(KafkaApis.scala:555)
>         at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
>         at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
>         at scala.collection.immutable.Set$Set1.foreach(Set.scala:79)
>         at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
>         at
> scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)
>         at scala.collection.SetLike$class.map(SetLike.scala:92)
>         at scala.collection.AbstractSet.map(Set.scala:47)
>         at kafka.server.KafkaApis.getTopicMetadata(KafkaApis.scala:555)
>         at
> kafka.server.KafkaApis.handleGroupCoordinatorRequest(KafkaApis.scala:702)
>         at kafka.server.KafkaApis.handle(KafkaApis.scala:78)
>         at
> kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
>         at java.lang.Thread.run(Thread.java:745)
>
> I would be grateful if you could help me with the forth-mentioned problem.
>
>
> Thank you in advance
> Dimitris Pappas
>
> ---
> Αυτό το e-mail ελέγχθηκε για ιούς από το πρόγραμμα Avast antivirus.
> https://www.avast.com/antivirus
>
>