How to get size (metadata) of a message without fetch request in Kafka through SimpleConsumer?

[Hutashan Chandrakar <hu...@gmail.com>]

Hello All,

Requirement :- Process/Read (Consumer) data only if kafka message size
greater then specified size (configured).


I am using SimpleConsumer and trying to pull size wise (GB) message using
spark(kafka RDD).I am able to get earliest and latest offset using metadata
request, but not able to get any size related metadata information.


Is there way to get size of a message without fetchrequest in Kafka through
SimpleConsumer/highlevel api ? Is there any API/method to get size related
information in metadata request/response in kafka api.



Thanks,
Hutashan Chandrakar

Re: How to get size (metadata) of a message without fetch request in Kafka through SimpleConsumer?

[Ashish Singh <as...@cloudera.com>]

Hey,

IIRC, Kafka does not provide any size related metadata. IMHO, there isn't
any need of such metadata as well. If you want to process data only after
you have certain amount of data to process, you can always consume records
from Kafka into some buffer and process the buffered data once it is of
required size. If your concern is what if your process, owning the buffer,
crashes, how will you know where to start consuming from, Kafka lets you
control when messages are considered consumed. As you are already using
simple consumer, you have this flexibility. Hope this helps.

On Fri, Jul 17, 2015 at 2:21 PM, Hutashan Chandrakar <hu...@gmail.com>
wrote:

> Hello All,
>
> Requirement :- Process/Read (Consumer) data only if kafka message size
> greater then specified size (configured).
>
>
> I am using SimpleConsumer and trying to pull size wise (GB) message using
> spark(kafka RDD).I am able to get earliest and latest offset using metadata
> request, but not able to get any size related metadata information.
>
>
> Is there way to get size of a message without fetchrequest in Kafka through
> SimpleConsumer/highlevel api ? Is there any API/method to get size related
> information in metadata request/response in kafka api.
>
>
>
> Thanks,
> Hutashan Chandrakar
>



-- 

Regards,
Ashish