Kafka Broker - In Memory Topics & Messages

[Pankaj Misra <pa...@impetus.co.in>]

Hi All,

I am working on using Kafka for building a highly scalable system. As I understand and have seen, Kafka broker has a very impressive and scalable file handling mechanisms to provide guaranteed delivery. However in one of the scenarios, I am facing a different challenge.

The scenario is such that the message payload is buffered and guaranteed for delivery by an external system, wherein there is no compelling need for guaranteed delivery from Kafka, but there is a need to parallel process the message streams. This made me wonder, if there is some way in Kafka, wherein I can avoid creation of files and instead stream the messages in-memory as they come and still take advantage of Kafka message streams, avoiding the small overhead of file management (avoid some more disk level IOPS).

Would greatly appreciate community's response.

Thanks & Regards
Pankaj Misra


________________________________






NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.

Re: Kafka Broker - In Memory Topics & Messages

[Jun Rao <ju...@gmail.com>]

For real time consumers, the overhead from the file system should be small
since the requested data is likely in pagecache and we use zero-copy
transfer.

Thanks,

Jun


On Mon, Apr 22, 2013 at 11:07 PM, Pankaj Misra
<pa...@impetus.co.in>wrote:

> Hi All,
>
> I am working on using Kafka for building a highly scalable system. As I
> understand and have seen, Kafka broker has a very impressive and scalable
> file handling mechanisms to provide guaranteed delivery. However in one of
> the scenarios, I am facing a different challenge.
>
> The scenario is such that the message payload is buffered and guaranteed
> for delivery by an external system, wherein there is no compelling need for
> guaranteed delivery from Kafka, but there is a need to parallel process the
> message streams. This made me wonder, if there is some way in Kafka,
> wherein I can avoid creation of files and instead stream the messages
> in-memory as they come and still take advantage of Kafka message streams,
> avoiding the small overhead of file management (avoid some more disk level
> IOPS).
>
> Would greatly appreciate community's response.
>
> Thanks & Regards
> Pankaj Misra
>
>
> ________________________________
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>

Re: Kafka Broker - In Memory Topics & Messages

[David Arthur <mu...@gmail.com>]

Sounds like you want something like zeromq?

http://zguide.zeromq.org/page:all#Divide-and-Conquer

-David

On 4/23/13 2:07 AM, Pankaj Misra wrote:
> Hi All,
>
> I am working on using Kafka for building a highly scalable system. As I understand and have seen, Kafka broker has a very impressive and scalable file handling mechanisms to provide guaranteed delivery. However in one of the scenarios, I am facing a different challenge.
>
> The scenario is such that the message payload is buffered and guaranteed for delivery by an external system, wherein there is no compelling need for guaranteed delivery from Kafka, but there is a need to parallel process the message streams. This made me wonder, if there is some way in Kafka, wherein I can avoid creation of files and instead stream the messages in-memory as they come and still take advantage of Kafka message streams, avoiding the small overhead of file management (avoid some more disk level IOPS).
>
> Would greatly appreciate community's response.
>
> Thanks & Regards
> Pankaj Misra
>
>
> ________________________________
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
>