You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Madhu Gmail <ma...@gmail.com> on 2013/04/16 23:16:47 UTC
Max events per RPC call from across sink to Avro source
Hello,
I have flume agent configuration with sink batch size as 900.
Also I have another flume agent working as collector.
Even though I have 900 as batch-size on agent sink, I can only see 100 event batch when the sink gets the data from file channel and sends it to collector.
I also see below comment in the code where it says max is 100 per RPC call
<td><tt>batch-size</tt></td>
* <td>The maximum number of events to send per RPC.</td>
* <td>events (int)</td>
* <td>100</td>
Madhu Munagala
(214)679-2872
Re: Max events per RPC call from across sink to Avro source
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
> Only for Avro sink, it should be batch-size. But every where else it is batchSize.
Hmm, indeed. I filed a jira in your behalf (https://issues.apache.org/jira/browse/FLUME-2006) , we're happy to fix this small issue.
- Alex
On Apr 17, 2013, at 1:17 AM, Madhu Gmail <ma...@gmail.com> wrote:
> I found the issue with my configuration.
>
> I had batchSize instead of batch-size
>
> It was the property name for batch size.
>
> Only for Avro sink, it should be batch-size. But every where else it is batchSize.
>
>
>
> Madhu Munagala
> (214)679-2872
>
> On Apr 16, 2013, at 4:45 PM, Chris Neal <cw...@gmail.com> wrote:
>
>> I believe that comment is indicating the default value, not the max. Perhaps your config is not correct?
>>
>>
>> On Tue, Apr 16, 2013 at 4:16 PM, Madhu Gmail <ma...@gmail.com> wrote:
>> Hello,
>>
>> I have flume agent configuration with sink batch size as 900.
>>
>> Also I have another flume agent working as collector.
>>
>> Even though I have 900 as batch-size on agent sink, I can only see 100 event batch when the sink gets the data from file channel and sends it to collector.
>>
>> I also see below comment in the code where it says max is 100 per RPC call
>>
>> <td><tt>batch-size</tt></td>
>> * <td>The maximum number of events to send per RPC.</td>
>> * <td>events (int)</td>
>> * <td>100</td>
>>
>> Madhu Munagala
>> (214)679-2872
>>
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Re: Max events per RPC call from across sink to Avro source
Posted by Madhu Gmail <ma...@gmail.com>.
I found the issue with my configuration.
I had batchSize instead of batch-size
It was the property name for batch size.
Only for Avro sink, it should be batch-size. But every where else it is batchSize.
Madhu Munagala
(214)679-2872
On Apr 16, 2013, at 4:45 PM, Chris Neal <cw...@gmail.com> wrote:
> I believe that comment is indicating the default value, not the max. Perhaps your config is not correct?
>
>
> On Tue, Apr 16, 2013 at 4:16 PM, Madhu Gmail <ma...@gmail.com> wrote:
>> Hello,
>>
>> I have flume agent configuration with sink batch size as 900.
>>
>> Also I have another flume agent working as collector.
>>
>> Even though I have 900 as batch-size on agent sink, I can only see 100 event batch when the sink gets the data from file channel and sends it to collector.
>>
>> I also see below comment in the code where it says max is 100 per RPC call
>>
>> <td><tt>batch-size</tt></td>
>> * <td>The maximum number of events to send per RPC.</td>
>> * <td>events (int)</td>
>> * <td>100</td>
>>
>> Madhu Munagala
>> (214)679-2872
>
Re: Max events per RPC call from across sink to Avro source
Posted by Chris Neal <cw...@gmail.com>.
I believe that comment is indicating the default value, not the max.
Perhaps your config is not correct?
On Tue, Apr 16, 2013 at 4:16 PM, Madhu Gmail <ma...@gmail.com>wrote:
> Hello,
>
> I have flume agent configuration with sink batch size as 900.
>
> Also I have another flume agent working as collector.
>
> Even though I have 900 as batch-size on agent sink, I can only see 100
> event batch when the sink gets the data from file channel and sends it to
> collector.
>
> I also see below comment in the code where it says max is 100 per RPC call
>
> <td><tt>batch-size</tt></td>
> * <td>The maximum number of events to send per RPC.</td>
> * <td>events (int)</td>
> * <td>100</td>
>
> Madhu Munagala
> (214)679-2872
Re: Max events per RPC call from across sink to Avro source
Posted by Madhu Gmail <ma...@gmail.com>.
May be it is taking the default batch size of 100 instead of 900 what I have for Avro sink.
Any clue why it is not taking 900 as batch size ?
Thanks
Madhu
On Apr 16, 2013, at 4:16 PM, Madhu Gmail <ma...@gmail.com> wrote:
> Hello,
>
> I have flume agent configuration with sink batch size as 900.
>
> Also I have another flume agent working as collector.
>
> Even though I have 900 as batch-size on agent sink, I can only see 100 event batch when the sink gets the data from file channel and sends it to collector.
>
> I also see below comment in the code where it says max is 100 per RPC call
>
> <td><tt>batch-size</tt></td>
> * <td>The maximum number of events to send per RPC.</td>
> * <td>events (int)</td>
> * <td>100</td>
>
> Madhu Munagala
> (214)679-2872