You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by Ambarish Pande <am...@gmail.com> on 2017/04/03 07:05:30 UTC
Restricting emit speed of AbstractFileInputOperator.
Hi,
How can i make the AbstractFileInputOperator emit only 'x' number of lines
per window. Is there a hook for that. Or i have to do it manually?
Thank You.
Re: Restricting emit speed of AbstractFileInputOperator.
Posted by Vlad Rozov <v....@datatorrent.com>.
If it takes t to emit one tuple and t*x < 0.5 sec, emitBatchSize can be
set to x. This will help with CPU utilization as platform will put the
operator thread to sleep if it does not emit and does not have
handleIdleTime() callback defined.
Thank you,
Vlad
On 4/7/17 18:53, Bhupesh Chawda wrote:
> I think the understanding is wrong.
>
> Platform calls emitTuples multiple times in a window. This number is
> unknown; it depends on the window time. We can limit this to x.
>
> emitBatchSize controls the number of tuples emitted in one such call.
> Set this to 1.
>
> This should result in at most x tuples per window. Note that it can be
> less than x as well.
>
> ~ Bhupesh
>
>
>
>
> On Apr 8, 2017 01:56, "Ambarish Pande" <ambarish.pande2011@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hello Bhupesh Sir,
> But does that mean I am emitting only 'x' lines from the file?.
> Because from what I understood, emitTuples() emits multiple lines
> in a single call and emitBatchSize controls number of times
> emitTuples is called in a window. Am I right?. I inferred this
> from the following
>
> The platform invokes the |emitTuples()| callback multiple time
> in each streaming window; within a single such call, if a
> large number of tuples are emitted,there is some risk that
> they may overwhelm the downstream operators especially if they
> are performing some compute intensive operation.
>
>
> Thank You.
>
> On Fri, Apr 7, 2017 at 1:44 PM, Bhupesh Chawda
> <bhupesh@datatorrent.com <ma...@datatorrent.com>> wrote:
>
> You can set emitBatchSize to 1 and make sure emitTuples is
> called just 'x' times within a window. You can do this
> manually by keeping a count and resetting it in beginWindow().
>
> ~ Bhupesh
>
> _______________________________________________________
>
> Bhupesh Chawda
>
> E: bhupesh@datatorrent.com <ma...@datatorrent.com>|
> Twitter: @bhupeshsc
>
> www.datatorrent.com <http://www.datatorrent.com> |
> apex.apache.org <http://apex.apache.org>
>
> On Fri, Apr 7, 2017 at 1:38 PM, Ambarish Pande
> <ambarish.pande2011@gmail.com
> <ma...@gmail.com>> wrote:
>
> Yes i tried. That just gives me control on how many times
> emitTuples is called. I want control on number of tuples
> emitted.
> Thank you.
> Sent from my iPhone
> On 07-Apr-2017, at 8:08 AM, Yogi Devendra
> <devendra.vyavahare@gmail.com
> <ma...@gmail.com>> wrote:
>> Have you tried /emitBatchSize /as mentioned
>> https://apex.apache.org/docs/malhar/operators/fsInputOperator/
>> <https://apex.apache.org/docs/malhar/operators/fsInputOperator/>
>> ~ Yogi
>> On 3 April 2017 at 00:05, Ambarish Pande
>> <ambarish.pande2011@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> Hi,
>> How can i make the AbstractFileInputOperator emit
>> only 'x' number of lines per window. Is there a hook
>> for that. Or i have to do it manually?
>> Thank You.
>>
Re: Restricting emit speed of AbstractFileInputOperator.
Posted by Ambarish Pande <am...@gmail.com>.
Oh Okay. i couldn't find this mentioned clearly in the docs, that is why I
misinterpreted it.
Thank You.
On Sat, Apr 8, 2017 at 7:23 AM, Bhupesh Chawda <bh...@datatorrent.com>
wrote:
> I think the understanding is wrong.
>
> Platform calls emitTuples multiple times in a window. This number is
> unknown; it depends on the window time. We can limit this to x.
>
> emitBatchSize controls the number of tuples emitted in one such call. Set
> this to 1.
>
> This should result in at most x tuples per window. Note that it can be
> less than x as well.
>
> ~ Bhupesh
>
>
>
>
> On Apr 8, 2017 01:56, "Ambarish Pande" <am...@gmail.com>
> wrote:
>
> Hello Bhupesh Sir,
> But does that mean I am emitting only 'x' lines from the file?. Because
> from what I understood, emitTuples() emits multiple lines in a single call
> and emitBatchSize controls number of times emitTuples is called in a
> window. Am I right?. I inferred this from the following
>
> The platform invokes the emitTuples() callback multiple time in each
>> streaming window; within a single such call, if a large number of tuples
>> are emitted, there is some risk that they may overwhelm the downstream
>> operators especially if they are performing some compute intensive
>> operation.
>
>
> Thank You.
>
> On Fri, Apr 7, 2017 at 1:44 PM, Bhupesh Chawda <bh...@datatorrent.com>
> wrote:
>
>> You can set emitBatchSize to 1 and make sure emitTuples is called just
>> 'x' times within a window. You can do this manually by keeping a count and
>> resetting it in beginWindow().
>>
>> ~ Bhupesh
>>
>>
>> _______________________________________________________
>>
>> Bhupesh Chawda
>>
>> E: bhupesh@datatorrent.com | Twitter: @bhupeshsc
>>
>> www.datatorrent.com | apex.apache.org
>>
>>
>>
>> On Fri, Apr 7, 2017 at 1:38 PM, Ambarish Pande <
>> ambarish.pande2011@gmail.com> wrote:
>>
>>> Yes i tried. That just gives me control on how many times emitTuples is
>>> called. I want control on number of tuples emitted.
>>>
>>> Thank you.
>>>
>>> Sent from my iPhone
>>>
>>> On 07-Apr-2017, at 8:08 AM, Yogi Devendra <de...@gmail.com>
>>> wrote:
>>>
>>> Have you tried *emitBatchSize *as mentioned https://apex.apache.
>>> org/docs/malhar/operators/fsInputOperator/
>>>
>>> ~ Yogi
>>>
>>> On 3 April 2017 at 00:05, Ambarish Pande <am...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> How can i make the AbstractFileInputOperator emit only 'x' number of
>>>> lines per window. Is there a hook for that. Or i have to do it manually?
>>>>
>>>> Thank You.
>>>>
>>>
>>>
>>
>
>
Re: Restricting emit speed of AbstractFileInputOperator.
Posted by Bhupesh Chawda <bh...@datatorrent.com>.
I think the understanding is wrong.
Platform calls emitTuples multiple times in a window. This number is
unknown; it depends on the window time. We can limit this to x.
emitBatchSize controls the number of tuples emitted in one such call. Set
this to 1.
This should result in at most x tuples per window. Note that it can be less
than x as well.
~ Bhupesh
On Apr 8, 2017 01:56, "Ambarish Pande" <am...@gmail.com> wrote:
Hello Bhupesh Sir,
But does that mean I am emitting only 'x' lines from the file?. Because
from what I understood, emitTuples() emits multiple lines in a single call
and emitBatchSize controls number of times emitTuples is called in a
window. Am I right?. I inferred this from the following
The platform invokes the emitTuples() callback multiple time in each
> streaming window; within a single such call, if a large number of tuples
> are emitted, there is some risk that they may overwhelm the downstream
> operators especially if they are performing some compute intensive
> operation.
Thank You.
On Fri, Apr 7, 2017 at 1:44 PM, Bhupesh Chawda <bh...@datatorrent.com>
wrote:
> You can set emitBatchSize to 1 and make sure emitTuples is called just 'x'
> times within a window. You can do this manually by keeping a count and
> resetting it in beginWindow().
>
> ~ Bhupesh
>
>
> _______________________________________________________
>
> Bhupesh Chawda
>
> E: bhupesh@datatorrent.com | Twitter: @bhupeshsc
>
> www.datatorrent.com | apex.apache.org
>
>
>
> On Fri, Apr 7, 2017 at 1:38 PM, Ambarish Pande <
> ambarish.pande2011@gmail.com> wrote:
>
>> Yes i tried. That just gives me control on how many times emitTuples is
>> called. I want control on number of tuples emitted.
>>
>> Thank you.
>>
>> Sent from my iPhone
>>
>> On 07-Apr-2017, at 8:08 AM, Yogi Devendra <de...@gmail.com>
>> wrote:
>>
>> Have you tried *emitBatchSize *as mentioned https://apex.apache.
>> org/docs/malhar/operators/fsInputOperator/
>>
>> ~ Yogi
>>
>> On 3 April 2017 at 00:05, Ambarish Pande <am...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> How can i make the AbstractFileInputOperator emit only 'x' number of
>>> lines per window. Is there a hook for that. Or i have to do it manually?
>>>
>>> Thank You.
>>>
>>
>>
>
Re: Restricting emit speed of AbstractFileInputOperator.
Posted by Ambarish Pande <am...@gmail.com>.
Hello Bhupesh Sir,
But does that mean I am emitting only 'x' lines from the file?. Because
from what I understood, emitTuples() emits multiple lines in a single call
and emitBatchSize controls number of times emitTuples is called in a
window. Am I right?. I inferred this from the following
The platform invokes the emitTuples() callback multiple time in each
> streaming window; within a single such call, if a large number of tuples
> are emitted, there is some risk that they may overwhelm the downstream
> operators especially if they are performing some compute intensive
> operation.
Thank You.
On Fri, Apr 7, 2017 at 1:44 PM, Bhupesh Chawda <bh...@datatorrent.com>
wrote:
> You can set emitBatchSize to 1 and make sure emitTuples is called just 'x'
> times within a window. You can do this manually by keeping a count and
> resetting it in beginWindow().
>
> ~ Bhupesh
>
>
> _______________________________________________________
>
> Bhupesh Chawda
>
> E: bhupesh@datatorrent.com | Twitter: @bhupeshsc
>
> www.datatorrent.com | apex.apache.org
>
>
>
> On Fri, Apr 7, 2017 at 1:38 PM, Ambarish Pande <
> ambarish.pande2011@gmail.com> wrote:
>
>> Yes i tried. That just gives me control on how many times emitTuples is
>> called. I want control on number of tuples emitted.
>>
>> Thank you.
>>
>> Sent from my iPhone
>>
>> On 07-Apr-2017, at 8:08 AM, Yogi Devendra <de...@gmail.com>
>> wrote:
>>
>> Have you tried *emitBatchSize *as mentioned https://apex.apache.
>> org/docs/malhar/operators/fsInputOperator/
>>
>> ~ Yogi
>>
>> On 3 April 2017 at 00:05, Ambarish Pande <am...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> How can i make the AbstractFileInputOperator emit only 'x' number of
>>> lines per window. Is there a hook for that. Or i have to do it manually?
>>>
>>> Thank You.
>>>
>>
>>
>
Re: Restricting emit speed of AbstractFileInputOperator.
Posted by Ambarish Pande <am...@gmail.com>.
Sure Sir. I'll put up a feature request.
On Sat, Apr 8, 2017 at 1:13 AM, Yogi Devendra <yo...@apache.org>
wrote:
> Ambarish,
>
> What you are asking for would be useful for others as well.
>
> Would you mind contributing this change to the community?
> If you are interested, please create JIRA on https://issues.apache.org/
> jira/browse/APEXMALHAR for your request.
>
> For any dev discussions, please use dev@apex mailing list.
>
> ~ Yogi
>
> On 7 April 2017 at 01:14, Bhupesh Chawda <bh...@datatorrent.com> wrote:
>
>> You can set emitBatchSize to 1 and make sure emitTuples is called just
>> 'x' times within a window. You can do this manually by keeping a count and
>> resetting it in beginWindow().
>>
>> ~ Bhupesh
>>
>>
>> _______________________________________________________
>>
>> Bhupesh Chawda
>>
>> E: bhupesh@datatorrent.com | Twitter: @bhupeshsc
>>
>> www.datatorrent.com | apex.apache.org
>>
>>
>>
>> On Fri, Apr 7, 2017 at 1:38 PM, Ambarish Pande <
>> ambarish.pande2011@gmail.com> wrote:
>>
>>> Yes i tried. That just gives me control on how many times emitTuples is
>>> called. I want control on number of tuples emitted.
>>>
>>> Thank you.
>>>
>>> Sent from my iPhone
>>>
>>> On 07-Apr-2017, at 8:08 AM, Yogi Devendra <de...@gmail.com>
>>> wrote:
>>>
>>> Have you tried *emitBatchSize *as mentioned https://apex.apache.
>>> org/docs/malhar/operators/fsInputOperator/
>>>
>>> ~ Yogi
>>>
>>> On 3 April 2017 at 00:05, Ambarish Pande <am...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> How can i make the AbstractFileInputOperator emit only 'x' number of
>>>> lines per window. Is there a hook for that. Or i have to do it manually?
>>>>
>>>> Thank You.
>>>>
>>>
>>>
>>
>
Re: Restricting emit speed of AbstractFileInputOperator.
Posted by Yogi Devendra <yo...@apache.org>.
Ambarish,
What you are asking for would be useful for others as well.
Would you mind contributing this change to the community?
If you are interested, please create JIRA on
https://issues.apache.org/jira/browse/APEXMALHAR for your request.
For any dev discussions, please use dev@apex mailing list.
~ Yogi
On 7 April 2017 at 01:14, Bhupesh Chawda <bh...@datatorrent.com> wrote:
> You can set emitBatchSize to 1 and make sure emitTuples is called just 'x'
> times within a window. You can do this manually by keeping a count and
> resetting it in beginWindow().
>
> ~ Bhupesh
>
>
> _______________________________________________________
>
> Bhupesh Chawda
>
> E: bhupesh@datatorrent.com | Twitter: @bhupeshsc
>
> www.datatorrent.com | apex.apache.org
>
>
>
> On Fri, Apr 7, 2017 at 1:38 PM, Ambarish Pande <
> ambarish.pande2011@gmail.com> wrote:
>
>> Yes i tried. That just gives me control on how many times emitTuples is
>> called. I want control on number of tuples emitted.
>>
>> Thank you.
>>
>> Sent from my iPhone
>>
>> On 07-Apr-2017, at 8:08 AM, Yogi Devendra <de...@gmail.com>
>> wrote:
>>
>> Have you tried *emitBatchSize *as mentioned https://apex.apache.
>> org/docs/malhar/operators/fsInputOperator/
>>
>> ~ Yogi
>>
>> On 3 April 2017 at 00:05, Ambarish Pande <am...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> How can i make the AbstractFileInputOperator emit only 'x' number of
>>> lines per window. Is there a hook for that. Or i have to do it manually?
>>>
>>> Thank You.
>>>
>>
>>
>
Re: Restricting emit speed of AbstractFileInputOperator.
Posted by Bhupesh Chawda <bh...@datatorrent.com>.
You can set emitBatchSize to 1 and make sure emitTuples is called just 'x'
times within a window. You can do this manually by keeping a count and
resetting it in beginWindow().
~ Bhupesh
_______________________________________________________
Bhupesh Chawda
E: bhupesh@datatorrent.com | Twitter: @bhupeshsc
www.datatorrent.com | apex.apache.org
On Fri, Apr 7, 2017 at 1:38 PM, Ambarish Pande <ambarish.pande2011@gmail.com
> wrote:
> Yes i tried. That just gives me control on how many times emitTuples is
> called. I want control on number of tuples emitted.
>
> Thank you.
>
> Sent from my iPhone
>
> On 07-Apr-2017, at 8:08 AM, Yogi Devendra <de...@gmail.com>
> wrote:
>
> Have you tried *emitBatchSize *as mentioned https://apex.apache.
> org/docs/malhar/operators/fsInputOperator/
>
> ~ Yogi
>
> On 3 April 2017 at 00:05, Ambarish Pande <am...@gmail.com>
> wrote:
>
>> Hi,
>>
>> How can i make the AbstractFileInputOperator emit only 'x' number of
>> lines per window. Is there a hook for that. Or i have to do it manually?
>>
>> Thank You.
>>
>
>
Re: Restricting emit speed of AbstractFileInputOperator.
Posted by Ambarish Pande <am...@gmail.com>.
Yes i tried. That just gives me control on how many times emitTuples is called. I want control on number of tuples emitted.
Thank you.
Sent from my iPhone
> On 07-Apr-2017, at 8:08 AM, Yogi Devendra <de...@gmail.com> wrote:
>
> Have you tried emitBatchSize as mentioned https://apex.apache.org/docs/malhar/operators/fsInputOperator/
>
> ~ Yogi
>
>> On 3 April 2017 at 00:05, Ambarish Pande <am...@gmail.com> wrote:
>> Hi,
>>
>> How can i make the AbstractFileInputOperator emit only 'x' number of lines per window. Is there a hook for that. Or i have to do it manually?
>>
>> Thank You.
>
Re: Restricting emit speed of AbstractFileInputOperator.
Posted by Yogi Devendra <de...@gmail.com>.
Have you tried *emitBatchSize *as mentioned
https://apex.apache.org/docs/malhar/operators/fsInputOperator/
~ Yogi
On 3 April 2017 at 00:05, Ambarish Pande <am...@gmail.com>
wrote:
> Hi,
>
> How can i make the AbstractFileInputOperator emit only 'x' number of lines
> per window. Is there a hook for that. Or i have to do it manually?
>
> Thank You.
>