Kafka for passing large documents

[Otis Gospodnetic <ot...@yahoo.com>]

Hi,

We're considering using Kafka for transport of potentially large "documents" (think documents for full-text indexing, docs that can be as small as tweets or as large as PDF files, say 5MB).

I'm wondering if Kafka is suitable for transporting potentially large documents or if there is something inherent in Kafka that makes it a poor choice for this use case?

Thanks,
Otis
----
Performance Monitoring for Solr / ElasticSearch / HBase - http://sematext.com/spm 

Re: Kafka for passing large documents

[Neha Narkhede <ne...@gmail.com>]

Kafka could be a good fit for transporting large documents, given that
you use compression and tune the consumer fetch buffer correctly.
Since Kafka does not stream read large messages, you need at least n
MB memory available to read a n MB large message. Typically you would
need more memory than that, since multiple consumer threads run on the
same machine. To protect the consumer from running out of memory, we
are adding a server side config that controls the largest message size
that the server will accept.

Thanks,
Neha

On Tue, Oct 16, 2012 at 7:51 AM, Otis Gospodnetic
<ot...@yahoo.com> wrote:
> Hi,
>
> We're considering using Kafka for transport of potentially large "documents" (think documents for full-text indexing, docs that can be as small as tweets or as large as PDF files, say 5MB).
>
> I'm wondering if Kafka is suitable for transporting potentially large documents or if there is something inherent in Kafka that makes it a poor choice for this use case?
>
> Thanks,
> Otis
> ----
> Performance Monitoring for Solr / ElasticSearch / HBase - http://sematext.com/spm