You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by R S <my...@gmail.com> on 2012/06/20 19:02:05 UTC
question about broker
Hi,
When producer sends messages to broker on a particular topic and at the
sametime suppose consumer is fetching messages from the same topic. Will
broker be sending message to consumer only after writing the message to
disk or would writing to disk will happen in a asynchronous way.
regards,
rks
Re: question about broker
Posted by Chris Burroughs <ch...@gmail.com>.
This is pretty fundamental to the contract the the broker provides.
Without changing code you could use some LD_PRELOAD hackery to disable
fsync and then set log.flush.interval=1 to one. But that's really not
recommended.
On 06/20/2012 01:11 PM, R S wrote:
> Thank you Neha ,
>
> Just to clarify , when you say high watermark , my understanding of that is
> the maximum offset which is available to consumer .
> The reason iam asking about this is , we want to use kafka for ultra fast
> messaging to do budget updates in shortest possible time , hence we just
> want to send messages to our aggregation service as fast as possible so was
> curious if there is a way we can just skip write to the disk and just
> support Pub-Sub.
>
> regards,
> rks
>
> On Wed, Jun 20, 2012 at 10:34 PM, Neha Narkhede <ne...@gmail.com>wrote:
>
>> The broker maintains a high watermark for each topic partition log.
>> This high watermark moves only after the messages are written to disk.
>> And, the consumer can only consume data exposed by this high
>> watermark.
>>
>> Thanks,
>> Neha
>>
>> On Wed, Jun 20, 2012 at 10:02 AM, R S <my...@gmail.com> wrote:
>>> Hi,
>>>
>>> When producer sends messages to broker on a particular topic and at the
>>> sametime suppose consumer is fetching messages from the same topic. Will
>>> broker be sending message to consumer only after writing the message to
>>> disk or would writing to disk will happen in a asynchronous way.
>>>
>>>
>>> regards,
>>> rks
>>
>
Re: question about broker
Posted by R S <my...@gmail.com>.
Thank you Neha ,
Just to clarify , when you say high watermark , my understanding of that is
the maximum offset which is available to consumer .
The reason iam asking about this is , we want to use kafka for ultra fast
messaging to do budget updates in shortest possible time , hence we just
want to send messages to our aggregation service as fast as possible so was
curious if there is a way we can just skip write to the disk and just
support Pub-Sub.
regards,
rks
On Wed, Jun 20, 2012 at 10:34 PM, Neha Narkhede <ne...@gmail.com>wrote:
> The broker maintains a high watermark for each topic partition log.
> This high watermark moves only after the messages are written to disk.
> And, the consumer can only consume data exposed by this high
> watermark.
>
> Thanks,
> Neha
>
> On Wed, Jun 20, 2012 at 10:02 AM, R S <my...@gmail.com> wrote:
> > Hi,
> >
> > When producer sends messages to broker on a particular topic and at the
> > sametime suppose consumer is fetching messages from the same topic. Will
> > broker be sending message to consumer only after writing the message to
> > disk or would writing to disk will happen in a asynchronous way.
> >
> >
> > regards,
> > rks
>
Re: question about broker
Posted by Neha Narkhede <ne...@gmail.com>.
The broker maintains a high watermark for each topic partition log.
This high watermark moves only after the messages are written to disk.
And, the consumer can only consume data exposed by this high
watermark.
Thanks,
Neha
On Wed, Jun 20, 2012 at 10:02 AM, R S <my...@gmail.com> wrote:
> Hi,
>
> When producer sends messages to broker on a particular topic and at the
> sametime suppose consumer is fetching messages from the same topic. Will
> broker be sending message to consumer only after writing the message to
> disk or would writing to disk will happen in a asynchronous way.
>
>
> regards,
> rks