You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Milind Parikh <mi...@gmail.com> on 2012/02/19 08:35:02 UTC
Kafka and Countandra
I am experimenting with Kafka to see if it can serve as a event buffer for
Countandra (www.countandra.org) and beyond that, into cascading counters. I
have setup Kafka on a local server Ubuntu 10.04, openjdk1.6 with default
installation of 0.7.
*1. While the default console-producer, console-consumer paradigm works
great, when I try modiying the batch size
*
bin/kafka-console-producer.sh --batch-size 300 --zookeeper localhost:2181
--topic test1
it gives me a
Exception in thread "main" java.lang.NumberFormatException: null
at java.lang.Integer.parseInt(Integer.java:443)
at java.lang.Integer.parseInt(Integer.java:514)
at
scala.collection.immutable.StringLike$class.toInt(StringLike.scala:207)
at scala.collection.immutable.StringOps.toInt(StringOps.scala:31)
at kafka.utils.Utils$.getIntInRange(Utils.scala:189)
at kafka.utils.Utils$.getInt(Utils.scala:174)
at
kafka.producer.async.AsyncProducerConfigShared$class.$init$(AsyncProducerConfig.scala:45)
at kafka.producer.ProducerConfig.<init>(ProducerConfig.scala:25)
at kafka.producer.ConsoleProducer$.main(ConsoleProducer.scala:108)
at kafka.producer.ConsoleProducer.main(ConsoleProducer.scala)
I have looked at the code and can't figure out what's wrong
*2. When I do bin/kafka-console-producer.sh --timeout 30000 --zookeeper
localhost:2181 --topic test1 *
I would think that console-producer would wait for 30s if the batch size
(default 200) is not full. It doesn't. It takes the same time without the
timeout parameter (default 1000) and dumps whatever the batch size.
I am probably missing something here. Any pointers would be appreciated.
Thanks
Milind
Re: Kafka and Countandra
Posted by Milind Parikh <mi...@gmail.com>.
Thanks
https://issues.apache.org/jira/browse/KAFKA-279
Regards
Milind
On Sun, Feb 19, 2012 at 10:55 AM, Jun Rao <ju...@gmail.com> wrote:
> Milind,
>
> You hit a couple of bugs in ConsoleProducer.
>
> 1. The code does the following to set batch size
> props.put("batch.size", batchSize)
> Instead, it should do
> props.put("batch.size", batchSize.toString)
>
> 2. It sets the wrong property name for timeout. Instead of doing
> props.put("queue.enqueueTimeout.ms", sendTimeout.toString)
> it should do
> props.put("queue.time", sendTimeout.toString)
>
> Could you file a jira for that?
>
> Thanks,
>
> Jun
>
>
> On Sat, Feb 18, 2012 at 11:35 PM, Milind Parikh <milindparikh@gmail.com
> >wrote:
>
> > I am experimenting with Kafka to see if it can serve as a event buffer
> for
> > Countandra (www.countandra.org) and beyond that, into cascading
> counters.
> > I
> > have setup Kafka on a local server Ubuntu 10.04, openjdk1.6 with default
> > installation of 0.7.
> >
> > *1. While the default console-producer, console-consumer paradigm works
> > great, when I try modiying the batch size
> > *
> > bin/kafka-console-producer.sh --batch-size 300 --zookeeper
> localhost:2181
> > --topic test1
> >
> > it gives me a
> >
> > Exception in thread "main" java.lang.NumberFormatException: null
> > at java.lang.Integer.parseInt(Integer.java:443)
> > at java.lang.Integer.parseInt(Integer.java:514)
> > at
> > scala.collection.immutable.StringLike$class.toInt(StringLike.scala:207)
> > at scala.collection.immutable.StringOps.toInt(StringOps.scala:31)
> > at kafka.utils.Utils$.getIntInRange(Utils.scala:189)
> > at kafka.utils.Utils$.getInt(Utils.scala:174)
> > at
> >
> >
> kafka.producer.async.AsyncProducerConfigShared$class.$init$(AsyncProducerConfig.scala:45)
> > at kafka.producer.ProducerConfig.<init>(ProducerConfig.scala:25)
> > at kafka.producer.ConsoleProducer$.main(ConsoleProducer.scala:108)
> > at kafka.producer.ConsoleProducer.main(ConsoleProducer.scala)
> >
> > I have looked at the code and can't figure out what's wrong
> >
> > *2. When I do bin/kafka-console-producer.sh --timeout 30000 --zookeeper
> > localhost:2181 --topic test1 *
> >
> > I would think that console-producer would wait for 30s if the batch size
> > (default 200) is not full. It doesn't. It takes the same time without the
> > timeout parameter (default 1000) and dumps whatever the batch size.
> >
> > I am probably missing something here. Any pointers would be appreciated.
> >
> > Thanks
> > Milind
> >
>
Re: Kafka and Countandra
Posted by Jun Rao <ju...@gmail.com>.
Milind,
You hit a couple of bugs in ConsoleProducer.
1. The code does the following to set batch size
props.put("batch.size", batchSize)
Instead, it should do
props.put("batch.size", batchSize.toString)
2. It sets the wrong property name for timeout. Instead of doing
props.put("queue.enqueueTimeout.ms", sendTimeout.toString)
it should do
props.put("queue.time", sendTimeout.toString)
Could you file a jira for that?
Thanks,
Jun
On Sat, Feb 18, 2012 at 11:35 PM, Milind Parikh <mi...@gmail.com>wrote:
> I am experimenting with Kafka to see if it can serve as a event buffer for
> Countandra (www.countandra.org) and beyond that, into cascading counters.
> I
> have setup Kafka on a local server Ubuntu 10.04, openjdk1.6 with default
> installation of 0.7.
>
> *1. While the default console-producer, console-consumer paradigm works
> great, when I try modiying the batch size
> *
> bin/kafka-console-producer.sh --batch-size 300 --zookeeper localhost:2181
> --topic test1
>
> it gives me a
>
> Exception in thread "main" java.lang.NumberFormatException: null
> at java.lang.Integer.parseInt(Integer.java:443)
> at java.lang.Integer.parseInt(Integer.java:514)
> at
> scala.collection.immutable.StringLike$class.toInt(StringLike.scala:207)
> at scala.collection.immutable.StringOps.toInt(StringOps.scala:31)
> at kafka.utils.Utils$.getIntInRange(Utils.scala:189)
> at kafka.utils.Utils$.getInt(Utils.scala:174)
> at
>
> kafka.producer.async.AsyncProducerConfigShared$class.$init$(AsyncProducerConfig.scala:45)
> at kafka.producer.ProducerConfig.<init>(ProducerConfig.scala:25)
> at kafka.producer.ConsoleProducer$.main(ConsoleProducer.scala:108)
> at kafka.producer.ConsoleProducer.main(ConsoleProducer.scala)
>
> I have looked at the code and can't figure out what's wrong
>
> *2. When I do bin/kafka-console-producer.sh --timeout 30000 --zookeeper
> localhost:2181 --topic test1 *
>
> I would think that console-producer would wait for 30s if the batch size
> (default 200) is not full. It doesn't. It takes the same time without the
> timeout parameter (default 1000) and dumps whatever the batch size.
>
> I am probably missing something here. Any pointers would be appreciated.
>
> Thanks
> Milind
>
Re: Kafka and Countandra
Posted by Milind Parikh <mi...@gmail.com>.
Interesting question to my question..:). I'm the primary committer of
Countandra. I released the first version (0.5)about a month ago.
Back to my original question, am I doing something wrong in the invocation
of console -producer and/or is my expectations about --timeout misguided?
Thanks
Milind
/***********************
sent from my android...please pardon occasional typos as I respond @ the
speed of thought
************************/
On Feb 19, 2012 8:30 AM, "Evan chan" <ev...@ooyala.com> wrote:
Milind, out of curiosity, how much experience do you have with Countandra,
and how stable is it?
-Evan
Carry your candle, run to the darkness
Seek out the helpless, deceived and poor
Hold out your candle for all to see it
Take your candle, and go light your world
On Feb 18, 2012, at 11:35 PM, Milind Parikh <mi...@gmail.com> wrote:
> I am experimentin...
> *1. While the default console-producer, console-consumer paradigm works
> great, when I try modiying the batch size
> *
> bin/kafka-console-producer.sh --batch-size 300 -...
> *2. When I do bin/kafka-console-producer.sh --timeout 30000 --zookeeper
> localhost:2181 --topic test1 *
>
> I would think that console-producer would wait for 30s if the batch size
> (default 200) is not...
Re: Kafka and Countandra
Posted by Evan chan <ev...@ooyala.com>.
Milind, out of curiosity, how much experience do you have with Countandra, and how stable is it?
-Evan
Carry your candle, run to the darkness
Seek out the helpless, deceived and poor
Hold out your candle for all to see it
Take your candle, and go light your world
On Feb 18, 2012, at 11:35 PM, Milind Parikh <mi...@gmail.com> wrote:
> I am experimenting with Kafka to see if it can serve as a event buffer for
> Countandra (www.countandra.org) and beyond that, into cascading counters. I
> have setup Kafka on a local server Ubuntu 10.04, openjdk1.6 with default
> installation of 0.7.
>
> *1. While the default console-producer, console-consumer paradigm works
> great, when I try modiying the batch size
> *
> bin/kafka-console-producer.sh --batch-size 300 --zookeeper localhost:2181
> --topic test1
>
> it gives me a
>
> Exception in thread "main" java.lang.NumberFormatException: null
> at java.lang.Integer.parseInt(Integer.java:443)
> at java.lang.Integer.parseInt(Integer.java:514)
> at
> scala.collection.immutable.StringLike$class.toInt(StringLike.scala:207)
> at scala.collection.immutable.StringOps.toInt(StringOps.scala:31)
> at kafka.utils.Utils$.getIntInRange(Utils.scala:189)
> at kafka.utils.Utils$.getInt(Utils.scala:174)
> at
> kafka.producer.async.AsyncProducerConfigShared$class.$init$(AsyncProducerConfig.scala:45)
> at kafka.producer.ProducerConfig.<init>(ProducerConfig.scala:25)
> at kafka.producer.ConsoleProducer$.main(ConsoleProducer.scala:108)
> at kafka.producer.ConsoleProducer.main(ConsoleProducer.scala)
>
> I have looked at the code and can't figure out what's wrong
>
> *2. When I do bin/kafka-console-producer.sh --timeout 30000 --zookeeper
> localhost:2181 --topic test1 *
>
> I would think that console-producer would wait for 30s if the batch size
> (default 200) is not full. It doesn't. It takes the same time without the
> timeout parameter (default 1000) and dumps whatever the batch size.
>
> I am probably missing something here. Any pointers would be appreciated.
>
> Thanks
> Milind