You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@apex.apache.org by "Kottapalli, Venkatesh" <VK...@DIRECTV.com> on 2016/01/27 18:06:25 UTC

Reg. file splitter blocks

Hi,

                Is there a way to limit the number of blocks that filesplitter emits per window?

                I have a custom reader that would read data from the blocks, I am looking for a way to adjust the speed of filesplitter to that of the reader so that reader would require less resources to process the blocks.

Please suggest.

-Venkatesh.

Re: Reg. file splitter blocks

Posted by Munagala Ramanath <ra...@datatorrent.com>.

http://docs.datatorrent.com/operators/file_splitter/

Also documented in the Configuration section of the above document.

Ram

On Wed, Jan 27, 2016 at 9:14 AM, Sandeep Deshmukh <sa...@datatorrent.com>
wrote:

> You can use blocksThreshold for this purpose. Here is the relevant java doc
>
> /**
> * This is a threshold on the no. of blocks emitted per window. A lot of
> blocks emitted
> * per window can overwhelm the downstream operators. This setting helps to
> control that.
> */
> @Min(1)
> protected int blocksThreshold;
>
> Source:
>
> https://github.com/apache/incubator-apex-malhar/blob/master/library/src/main/java/com/datatorrent/lib/io/fs/FileSplitter.java
>
> Regards,
> Sandeep
>
> On Wed, Jan 27, 2016 at 10:36 PM, Kottapalli, Venkatesh <
> VKottapalli@directv.com> wrote:
>
> > Hi,
> >
> >                 Is there a way to limit the number of blocks that
> > filesplitter emits per window?
> >
> >                 I have a custom reader that would read data from the
> > blocks, I am looking for a way to adjust the speed of filesplitter to
> that
> > of the reader so that reader would require less resources to process the
> > blocks.
> >
> > Please suggest.
> >
> > -Venkatesh.
> >
>

Re: Reg. file splitter blocks

Posted by Sandeep Deshmukh <sa...@datatorrent.com>.

You can use blocksThreshold for this purpose. Here is the relevant java doc

/**
* This is a threshold on the no. of blocks emitted per window. A lot of
blocks emitted
* per window can overwhelm the downstream operators. This setting helps to
control that.
*/
@Min(1)
protected int blocksThreshold;

Source:
https://github.com/apache/incubator-apex-malhar/blob/master/library/src/main/java/com/datatorrent/lib/io/fs/FileSplitter.java

Regards,
Sandeep

On Wed, Jan 27, 2016 at 10:36 PM, Kottapalli, Venkatesh <
VKottapalli@directv.com> wrote:

> Hi,
>
>                 Is there a way to limit the number of blocks that
> filesplitter emits per window?
>
>                 I have a custom reader that would read data from the
> blocks, I am looking for a way to adjust the speed of filesplitter to that
> of the reader so that reader would require less resources to process the
> blocks.
>
> Please suggest.
>
> -Venkatesh.
>