You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Ian Friedman <ia...@flurry.com> on 2014/04/10 01:13:16 UTC
console producer question
Hey guys. We recently migrated our production cluster from 0.7.2 to 0.8.1. One of the tools in the 0.7 distribution was something called Producer Shell, which we used in some cron jobs to do some manual addition of messages (for messages that got dropped for various reasons). So in 0.8 that is gone, and I have tried replacing it with the console Producer. It seems like the console producer only produces to a single partition whereas the producer shell used to randomly distribute the messages. Is this behavior configurable? AM I missing something obvious?
Thanks in advance
--Ian
Re: console producer question
Posted by Ian Friedman <ia...@flurry.com>.
Thanks Magnus! I will definitely check this out
—Ian
On Apr 9, 2014, at 8:39 PM, Magnus Edenhill <ma...@edenhill.se> wrote:
> Hey Ian,
>
> this is where a tool like kafkacat comes in handy, it will use a random
> partitioner by default (without the need for defining a key):
>
> tail -f /my/log | kafkacat -b mybroker -t mytopic
>
> See
> https://github.com/edenhill/kafkacat
>
>
>
> 2014-04-10 6:13 GMT+07:00 Ian Friedman <ia...@flurry.com>:
>
>> Hey guys. We recently migrated our production cluster from 0.7.2 to 0.8.1.
>> One of the tools in the 0.7 distribution was something called Producer
>> Shell, which we used in some cron jobs to do some manual addition of
>> messages (for messages that got dropped for various reasons). So in 0.8
>> that is gone, and I have tried replacing it with the console Producer. It
>> seems like the console producer only produces to a single partition whereas
>> the producer shell used to randomly distribute the messages. Is this
>> behavior configurable? AM I missing something obvious?
>>
>> Thanks in advance
>> --Ian
Re: console producer question
Posted by Joe Stein <jo...@stealth.ly>.
Thanks Magnus, very cool!!!! I added it to the client page
https://cwiki.apache.org/confluence/display/KAFKA/Clients
cheers!
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Thu, Apr 10, 2014 at 5:23 PM, Ian Friedman <ia...@flurry.com> wrote:
> Magnus this worked in our scripts perfectly, thanks a bunch!!
>
> --Ian
>
> On Apr 9, 2014, at 8:39 PM, Magnus Edenhill <ma...@edenhill.se> wrote:
>
> > Hey Ian,
> >
> > this is where a tool like kafkacat comes in handy, it will use a random
> > partitioner by default (without the need for defining a key):
> >
> > tail -f /my/log | kafkacat -b mybroker -t mytopic
> >
> > See
> > https://github.com/edenhill/kafkacat
> >
> >
> >
> > 2014-04-10 6:13 GMT+07:00 Ian Friedman <ia...@flurry.com>:
> >
> >> Hey guys. We recently migrated our production cluster from 0.7.2 to
> 0.8.1.
> >> One of the tools in the 0.7 distribution was something called Producer
> >> Shell, which we used in some cron jobs to do some manual addition of
> >> messages (for messages that got dropped for various reasons). So in 0.8
> >> that is gone, and I have tried replacing it with the console Producer.
> It
> >> seems like the console producer only produces to a single partition
> whereas
> >> the producer shell used to randomly distribute the messages. Is this
> >> behavior configurable? AM I missing something obvious?
> >>
> >> Thanks in advance
> >> --Ian
>
>
Re: console producer question
Posted by Ian Friedman <ia...@flurry.com>.
Magnus this worked in our scripts perfectly, thanks a bunch!!
--Ian
On Apr 9, 2014, at 8:39 PM, Magnus Edenhill <ma...@edenhill.se> wrote:
> Hey Ian,
>
> this is where a tool like kafkacat comes in handy, it will use a random
> partitioner by default (without the need for defining a key):
>
> tail -f /my/log | kafkacat -b mybroker -t mytopic
>
> See
> https://github.com/edenhill/kafkacat
>
>
>
> 2014-04-10 6:13 GMT+07:00 Ian Friedman <ia...@flurry.com>:
>
>> Hey guys. We recently migrated our production cluster from 0.7.2 to 0.8.1.
>> One of the tools in the 0.7 distribution was something called Producer
>> Shell, which we used in some cron jobs to do some manual addition of
>> messages (for messages that got dropped for various reasons). So in 0.8
>> that is gone, and I have tried replacing it with the console Producer. It
>> seems like the console producer only produces to a single partition whereas
>> the producer shell used to randomly distribute the messages. Is this
>> behavior configurable? AM I missing something obvious?
>>
>> Thanks in advance
>> --Ian
Re: console producer question
Posted by Magnus Edenhill <ma...@edenhill.se>.
Hey Ian,
this is where a tool like kafkacat comes in handy, it will use a random
partitioner by default (without the need for defining a key):
tail -f /my/log | kafkacat -b mybroker -t mytopic
See
https://github.com/edenhill/kafkacat
2014-04-10 6:13 GMT+07:00 Ian Friedman <ia...@flurry.com>:
> Hey guys. We recently migrated our production cluster from 0.7.2 to 0.8.1.
> One of the tools in the 0.7 distribution was something called Producer
> Shell, which we used in some cron jobs to do some manual addition of
> messages (for messages that got dropped for various reasons). So in 0.8
> that is gone, and I have tried replacing it with the console Producer. It
> seems like the console producer only produces to a single partition whereas
> the producer shell used to randomly distribute the messages. Is this
> behavior configurable? AM I missing something obvious?
>
> Thanks in advance
> --Ian
Re: console producer question
Posted by Joel Koshy <jj...@gmail.com>.
This may be because the 0.8 producer sticks to a partition during
metadata refresh intervals. You can get around that by specifying a
key:
--property parse.key=true --property key.separator=###
Each line would then be:
KEY###MESSAGE
The key is used for partitioning but will also be stored with the
message but that shouldn't break your consumers if they extract just
the message. We have a partition key which is thrown away after
partitioning but that has not been integrated with the console
producer.
Thanks,
Joel
On Wed, Apr 09, 2014 at 07:13:16PM -0400, Ian Friedman wrote:
> Hey guys. We recently migrated our production cluster from 0.7.2 to 0.8.1. One of the tools in the 0.7 distribution was something called Producer Shell, which we used in some cron jobs to do some manual addition of messages (for messages that got dropped for various reasons). So in 0.8 that is gone, and I have tried replacing it with the console Producer. It seems like the console producer only produces to a single partition whereas the producer shell used to randomly distribute the messages. Is this behavior configurable? AM I missing something obvious?
>
> Thanks in advance
> --Ian