You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Bhavesh Mistry <mi...@gmail.com> on 2014/06/25 01:37:02 UTC
Request New Feature On Kafka 0.8.1
Hi Kafka Dev Teeam,
I would like to request following features in Kafka Async Producers:
1) Ability to Inject the implementation of the Blocking Queue similar to
serialize and partition class etc…
I would like to inject following LIB for queue implementation. Of course
wrapper upon Queue interface..
eg http://lmax-exchange.github.io/disruptor/
2) Can we have configuration to send the data in Parallel to Brokers using
Pool Executor Service …etc or ability to inject Default Handlers in here
which will run on back ground threads
https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/producer/async/ProducerSendThread.scala
eg: Our event contains a timestamp so it is ok to have data out of order
into partition due to parallel write to Brokers per partition...
We are looking to dump the messages parallel not build up queue while it
data is being send to brokers and the send thread will take time to
complete entire batch if we have round robin on Partitions ( eg message
count % number Of partitions).
Please let me know if there is any alternative..
Thanks,
Bhavesh
Re: Request New Feature On Kafka 0.8.1
Posted by Bhavesh Mistry <mi...@gmail.com>.
Hi Kefka Dev,
Please let me know if you guys agree with use case features presented on
previous email.
Thanks,
Bhavesh
On Fri, Jun 27, 2014 at 11:45 AM, Bhavesh Mistry <mistry.p.bhavesh@gmail.com
> wrote:
> HI Steve,
>
> I am referring to version 0.8.0 and I would like to try in dev environment
> unbounded queue to try to figure upper bound of the application during the
> stress test and through put we want to achieve.
>
> Also, additional features I would like to add At Run-time on Producer side
> :
>
> 1) Ability to change the tuning parameters at Run-time (eg Batch Size,
> Queue size, enqueue time etc) because our app run on various private
> cloud , physical box (Same app) so it would be great If Kafka allow tuning
> parameter to be reset at runtime.
>
> 2) During the batch Async send, occasionally we get
> MessageSizeTooLargeException, is it possible ensure from the producer side
> that batch can not exceed configure limit, and if it does exceed the limit,
> divide the batch into 1/2 and send it transparently unless single message
> exceed the limit.
>
> Our use case is "transferring log lines", we can configure batch size for
> optimal throughput but we do not want to loose data when batch size exceed
> configured limit (2MB). It there any way we can introduce this batch
> split strategy logic.
>
> 3) Purpose of injecting the Queue implementation, we want to preserve a
> latest message in queue and remove the oldest ( Priority Queue or drop
> message oldest message). Because by configuring the enqueue timeout, it
> drops latest message.
>
> eg. we have some monitoring message that take greater priority than
> regular "log Lines" etc...
>
> I hope this clarify the use cases.
>
> Thanks,
>
> Bhavesh
>
>
>
> On Wed, Jun 25, 2014 at 12:41 PM, Steve Morin <st...@stevemorin.com>
> wrote:
>
>> Bhavesh,
>> On item 1) you didn't say why you want to be able to change the queue
>> implementation?
>> -Steve
>>
>>
>> On Tue, Jun 24, 2014 at 4:37 PM, Bhavesh Mistry <
>> mistry.p.bhavesh@gmail.com>
>> wrote:
>>
>> > Hi Kafka Dev Teeam,
>> >
>> >
>> >
>> >
>> >
>> > I would like to request following features in Kafka Async Producers:
>> >
>> >
>> >
>> > 1) Ability to Inject the implementation of the Blocking Queue similar
>> to
>> > serialize and partition class etc…
>> >
>> >
>> >
>> > I would like to inject following LIB for queue implementation. Of
>> course
>> > wrapper upon Queue interface..
>> >
>> > eg http://lmax-exchange.github.io/disruptor/
>> >
>> >
>> >
>> > 2) Can we have configuration to send the data in Parallel to Brokers
>> using
>> > Pool Executor Service …etc or ability to inject Default Handlers in here
>> > which will run on back ground threads
>> >
>> >
>> >
>> >
>> >
>> https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/producer/async/ProducerSendThread.scala
>> >
>> >
>> >
>> > eg: Our event contains a timestamp so it is ok to have data out of
>> order
>> > into partition due to parallel write to Brokers per partition...
>> >
>> >
>> > We are looking to dump the messages parallel not build up queue while it
>> > data is being send to brokers and the send thread will take time to
>> > complete entire batch if we have round robin on Partitions ( eg message
>> > count % number Of partitions).
>> >
>> > Please let me know if there is any alternative..
>> >
>> >
>> > Thanks,
>> > Bhavesh
>> >
>>
>
>
Re: Request New Feature On Kafka 0.8.1
Posted by Bhavesh Mistry <mi...@gmail.com>.
HI Steve,
I am referring to version 0.8.0 and I would like to try in dev environment
unbounded queue to try to figure upper bound of the application during the
stress test and through put we want to achieve.
Also, additional features I would like to add At Run-time on Producer side :
1) Ability to change the tuning parameters at Run-time (eg Batch Size,
Queue size, enqueue time etc) because our app run on various private
cloud , physical box (Same app) so it would be great If Kafka allow tuning
parameter to be reset at runtime.
2) During the batch Async send, occasionally we get
MessageSizeTooLargeException, is it possible ensure from the producer side
that batch can not exceed configure limit, and if it does exceed the limit,
divide the batch into 1/2 and send it transparently unless single message
exceed the limit.
Our use case is "transferring log lines", we can configure batch size for
optimal throughput but we do not want to loose data when batch size exceed
configured limit (2MB). It there any way we can introduce this batch
split strategy logic.
3) Purpose of injecting the Queue implementation, we want to preserve a
latest message in queue and remove the oldest ( Priority Queue or drop
message oldest message). Because by configuring the enqueue timeout, it
drops latest message.
eg. we have some monitoring message that take greater priority than
regular "log Lines" etc...
I hope this clarify the use cases.
Thanks,
Bhavesh
On Wed, Jun 25, 2014 at 12:41 PM, Steve Morin <st...@stevemorin.com> wrote:
> Bhavesh,
> On item 1) you didn't say why you want to be able to change the queue
> implementation?
> -Steve
>
>
> On Tue, Jun 24, 2014 at 4:37 PM, Bhavesh Mistry <
> mistry.p.bhavesh@gmail.com>
> wrote:
>
> > Hi Kafka Dev Teeam,
> >
> >
> >
> >
> >
> > I would like to request following features in Kafka Async Producers:
> >
> >
> >
> > 1) Ability to Inject the implementation of the Blocking Queue similar
> to
> > serialize and partition class etc…
> >
> >
> >
> > I would like to inject following LIB for queue implementation. Of course
> > wrapper upon Queue interface..
> >
> > eg http://lmax-exchange.github.io/disruptor/
> >
> >
> >
> > 2) Can we have configuration to send the data in Parallel to Brokers
> using
> > Pool Executor Service …etc or ability to inject Default Handlers in here
> > which will run on back ground threads
> >
> >
> >
> >
> >
> https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/producer/async/ProducerSendThread.scala
> >
> >
> >
> > eg: Our event contains a timestamp so it is ok to have data out of order
> > into partition due to parallel write to Brokers per partition...
> >
> >
> > We are looking to dump the messages parallel not build up queue while it
> > data is being send to brokers and the send thread will take time to
> > complete entire batch if we have round robin on Partitions ( eg message
> > count % number Of partitions).
> >
> > Please let me know if there is any alternative..
> >
> >
> > Thanks,
> > Bhavesh
> >
>
Re: Request New Feature On Kafka 0.8.1
Posted by Steve Morin <st...@stevemorin.com>.
Bhavesh,
On item 1) you didn't say why you want to be able to change the queue
implementation?
-Steve
On Tue, Jun 24, 2014 at 4:37 PM, Bhavesh Mistry <mi...@gmail.com>
wrote:
> Hi Kafka Dev Teeam,
>
>
>
>
>
> I would like to request following features in Kafka Async Producers:
>
>
>
> 1) Ability to Inject the implementation of the Blocking Queue similar to
> serialize and partition class etc…
>
>
>
> I would like to inject following LIB for queue implementation. Of course
> wrapper upon Queue interface..
>
> eg http://lmax-exchange.github.io/disruptor/
>
>
>
> 2) Can we have configuration to send the data in Parallel to Brokers using
> Pool Executor Service …etc or ability to inject Default Handlers in here
> which will run on back ground threads
>
>
>
>
> https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/producer/async/ProducerSendThread.scala
>
>
>
> eg: Our event contains a timestamp so it is ok to have data out of order
> into partition due to parallel write to Brokers per partition...
>
>
> We are looking to dump the messages parallel not build up queue while it
> data is being send to brokers and the send thread will take time to
> complete entire batch if we have round robin on Partitions ( eg message
> count % number Of partitions).
>
> Please let me know if there is any alternative..
>
>
> Thanks,
> Bhavesh
>
Re: Request New Feature On Kafka 0.8.1
Posted by Neha Narkhede <ne...@gmail.com>.
The concerns you expressed are fixed in the new producer implementation
available on trunk. You can give it a try at
org.apache.kafka.clients.producer.
On Tue, Jun 24, 2014 at 4:37 PM, Bhavesh Mistry <mi...@gmail.com>
wrote:
> Hi Kafka Dev Teeam,
>
>
>
>
>
> I would like to request following features in Kafka Async Producers:
>
>
>
> 1) Ability to Inject the implementation of the Blocking Queue similar to
> serialize and partition class etc…
>
>
>
> I would like to inject following LIB for queue implementation. Of course
> wrapper upon Queue interface..
>
> eg http://lmax-exchange.github.io/disruptor/
>
>
>
> 2) Can we have configuration to send the data in Parallel to Brokers using
> Pool Executor Service …etc or ability to inject Default Handlers in here
> which will run on back ground threads
>
>
>
>
> https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/producer/async/ProducerSendThread.scala
>
>
>
> eg: Our event contains a timestamp so it is ok to have data out of order
> into partition due to parallel write to Brokers per partition...
>
>
> We are looking to dump the messages parallel not build up queue while it
> data is being send to brokers and the send thread will take time to
> complete entire batch if we have round robin on Partitions ( eg message
> count % number Of partitions).
>
> Please let me know if there is any alternative..
>
>
> Thanks,
> Bhavesh
>