Re: FileSplitter/BlockReader

[Thomas Weise <th...@datatorrent.com>]

That's one option, though stats listener may provide better context for
decision making.

This is needed for better batch support.

--
sent from mobile
On Feb 23, 2016 8:36 PM, "Sandesh Hegde" <sa...@datatorrent.com> wrote:

> Experimental, Can we use iteration to send the feedback and based on that
> adjust the bandwidth?
>
> On Tue, Feb 23, 2016 at 8:32 PM Thomas Weise <th...@datatorrent.com>
> wrote:
>
>> Priyanka, that's right. In this case we want to process as fast as
>> possible without overwhelming the downstream operators.
>>
>> --
>> sent from mobile
>> On Feb 23, 2016 8:25 PM, "Priyanka Gugale" <pr...@datatorrent.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I have seen that code in Ingestion, and as I remember it had some issues
>>> and wasn't working.
>>> I don't have exactly what Thomas asked, but we can use Bandwidth limit
>>> feature, where splitter itself emits based on configured limit, but yes it
>>> isn't based on load on readers.
>>>
>>> -Priyanka
>>>
>>> On Wed, Feb 24, 2016 at 5:33 AM, Chandni Singh <ch...@datatorrent.com>
>>> wrote:
>>>
>>>> Hi Thomas,
>>>>
>>>> I had written a StatsListener which when set on FileSplitter and
>>>> BlockReader would automatically throttle block emission rate. The stats
>>>> listener will send a setProperty command to FileSplitter however if I
>>>> recall correctly, this changed the property only on the logical operator
>>>> and wasn't propagated to the physical operator.
>>>>
>>>> It was a while back so I am not sure if that was still the issue or
>>>> there was something else wrong my implementation.
>>>>
>>>> Thanks,
>>>> Chandni
>>>>
>>>> On Tue, Feb 23, 2016 at 3:48 PM, Thomas Weise <th...@datatorrent.com>
>>>> wrote:
>>>>
>>>>> Is there any support for the splitter to automatically throttle block
>>>>> emission rate based on the readers throughput?
>>>>>
>>>>> --
>>>>> sent from mobile
>>>>>
>>>>
>>>>
>>>

Re: FileSplitter/BlockReader

[Chandni Singh <ch...@datatorrent.com>]

Hi,

Here is a Stats Listener for doing this. Haven't tested it. Need some
sample test data on hdfs to test. This is based on the input port queue
size of Block reader.

https://github.com/chandnisingh/Malhar/blob/examples/apps/spitter/src/main/java/com/datatorrent/splitter/SplitterThresholdRegulator.java

On Tue, Feb 23, 2016 at 8:45 PM, Thomas Weise <th...@datatorrent.com>
wrote:

> That's one option, though stats listener may provide better context for
> decision making.
>
> This is needed for better batch support.
>
> --
> sent from mobile
> On Feb 23, 2016 8:36 PM, "Sandesh Hegde" <sa...@datatorrent.com> wrote:
>
> > Experimental, Can we use iteration to send the feedback and based on that
> > adjust the bandwidth?
> >
> > On Tue, Feb 23, 2016 at 8:32 PM Thomas Weise <th...@datatorrent.com>
> > wrote:
> >
> >> Priyanka, that's right. In this case we want to process as fast as
> >> possible without overwhelming the downstream operators.
> >>
> >> --
> >> sent from mobile
> >> On Feb 23, 2016 8:25 PM, "Priyanka Gugale" <pr...@datatorrent.com>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> I have seen that code in Ingestion, and as I remember it had some
> issues
> >>> and wasn't working.
> >>> I don't have exactly what Thomas asked, but we can use Bandwidth limit
> >>> feature, where splitter itself emits based on configured limit, but
> yes it
> >>> isn't based on load on readers.
> >>>
> >>> -Priyanka
> >>>
> >>> On Wed, Feb 24, 2016 at 5:33 AM, Chandni Singh <
> chandni@datatorrent.com>
> >>> wrote:
> >>>
> >>>> Hi Thomas,
> >>>>
> >>>> I had written a StatsListener which when set on FileSplitter and
> >>>> BlockReader would automatically throttle block emission rate. The
> stats
> >>>> listener will send a setProperty command to FileSplitter however if I
> >>>> recall correctly, this changed the property only on the logical
> operator
> >>>> and wasn't propagated to the physical operator.
> >>>>
> >>>> It was a while back so I am not sure if that was still the issue or
> >>>> there was something else wrong my implementation.
> >>>>
> >>>> Thanks,
> >>>> Chandni
> >>>>
> >>>> On Tue, Feb 23, 2016 at 3:48 PM, Thomas Weise <thomas@datatorrent.com
> >
> >>>> wrote:
> >>>>
> >>>>> Is there any support for the splitter to automatically throttle block
> >>>>> emission rate based on the readers throughput?
> >>>>>
> >>>>> --
> >>>>> sent from mobile
> >>>>>
> >>>>
> >>>>
> >>>
>