Re: flume spooldir performance

[manish jaiswal <ma...@gmail.com>]

Hi,

how to increase flume performance to read one million files from spool dir.
Can we read through multiple thread?



regards

On Sun, Feb 7, 2016 at 10:11 PM, manish jaiswal <ma...@gmail.com>
wrote:

> Hi,
>
> how to increase flume performance to read one million files from spool dir.
> Can we read through multiple thread?
>
>
>
> regards
> manish
>

Re: flume spooldir performance

[iain wright <ia...@gmail.com>]

IIRC the spool dir source is single threaded.

You could move your source files into 10 directories with 100k each, and
have 10 spooldir sources.

You will need to have fast channels, and likely multiple sinks or fan out,
to keep up with the source most likely.


-- 
Iain Wright

This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, do
not disclose or disseminate the message to anyone except the intended
recipient. If you have received this message in error, or are not the named
recipient(s), please immediately notify the sender by return email, and
delete all copies of this message.

On Sun, Feb 7, 2016 at 10:09 PM, manish jaiswal <ma...@gmail.com>
wrote:

> Hi,
>
> how to increase flume performance to read one million files from spool dir.
> Can we read through multiple thread?
>
>
>
> regards
>
> On Sun, Feb 7, 2016 at 10:11 PM, manish jaiswal <ma...@gmail.com>
> wrote:
>
>> Hi,
>>
>> how to increase flume performance to read one million files from spool
>> dir.
>> Can we read through multiple thread?
>>
>>
>>
>> regards
>> manish
>>
>
>