Storm KafkaConfig questions

[Raphael Hsieh <ra...@gmail.com>]

I'm confused about what the values fetchSizeBytes and bufferSizeBytes
relate to in my storm topology.

My TridentKafkaConfig seems to be completely separated from my StormConfig.
In my TridentKafkaConfig I can set my fetchSizeBytes and bufferSizeBytes
(what's the difference?) to a value, but in my StormConfig, I can set my
max batch size as well.

if I do:

tridentKafkaConfig.fetchSizeBytes = 1024;

then later do

config.put(RichSpoutBatchExecutor.MAX_BATCH_SIZE_CONF, 1024*1024*10)

doesn't that mean that my maximum batch size will be 1024 regardless ?

-- 
Raphael Hsieh

Re: Storm KafkaConfig questions

[Sam Goodwin <sa...@gmail.com>]

I think fetchSizeBytes is related to Kafka, not Storm. So if you look at
the Kafka documentation http://kafka.apache.org/08/configuration.html
you'll see that both producers and consumers have max mesage size in bytes.
Your KafkaConfig max fetch size should match the max message size of the
producers.

On Tue, Sep 30, 2014 at 3:00 PM, Raphael Hsieh <ra...@gmail.com> wrote:

> I'm confused about what the values fetchSizeBytes and bufferSizeBytes
> relate to in my storm topology.
>
> My TridentKafkaConfig seems to be completely separated from my StormConfig.
> In my TridentKafkaConfig I can set my fetchSizeBytes and bufferSizeBytes
> (what's the difference?) to a value, but in my StormConfig, I can set my
> max batch size as well.
>
> if I do:
>
> tridentKafkaConfig.fetchSizeBytes = 1024;
>
> then later do
>
> config.put(RichSpoutBatchExecutor.MAX_BATCH_SIZE_CONF, 1024*1024*10)
>
> doesn't that mean that my maximum batch size will be 1024 regardless ?
>
> --
> Raphael Hsieh
>
>
>
>