You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Sandesh Hegde <sa...@datatorrent.com> on 2016/03/24 20:27:07 UTC
Adaptive SPIN_MILLIS attributes
Hello Team,
The attributes SPIN_MILLIS is the amount of time the input operator sleeps
while there are no inputs to process and it's default value is 10ms.
Recently while I was doing performance experiments, reducing the
SPIN_MILLIS to 1ms increased the number of tuples processed by the input
operator. Instead of manually adjusting the SPIN_MILLIS for best
performance, how about changing it based on the heuristic?
Here is the idea
After sleeping for SPIN_MILLIS, if tuples are found reduce the SPIN_MILLIS
otherwise increase the SPIN_MILLIS.
Let me know your thoughts.
Thanks
Sandesh
Re: Adaptive SPIN_MILLIS attributes
Posted by Sandesh Hegde <sa...@datatorrent.com>.
Just to close this thread with numbers, this change is a part of 3.4
release.
Tested this change while doing Yahoo Benchmark,
Throughput increased from ~650K tuples/sec to over 1,000K tuples/sec, over
50% improvement, without any config change.
On Thu, Mar 24, 2016 at 1:00 PM Amol Kekre <am...@datatorrent.com> wrote:
> This is a good way, and it fits well with platform self-adjusting. We may
> need to still look at a way to allow users to keep SPIN_MILLIS fixed in
> case they depend on it for downstream intake.
>
> Thks
> Amol
>
>
> On Thu, Mar 24, 2016 at 12:45 PM, Sandesh Hegde <sa...@datatorrent.com>
> wrote:
>
> > That is cool, we need to something similar for input operators.
> >
> > On Thu, Mar 24, 2016 at 12:32 PM Thomas Weise <th...@datatorrent.com>
> > wrote:
> >
> > > Please see:
> > >
> > > https://issues.apache.org/jira/browse/APEXCORE-380
> > > https://github.com/apache/incubator-apex-core/pull/271
> > >
> > >
> > > On Thu, Mar 24, 2016 at 12:27 PM, Sandesh Hegde <
> sandesh@datatorrent.com
> > >
> > > wrote:
> > >
> > > > Hello Team,
> > > >
> > > > The attributes SPIN_MILLIS is the amount of time the input operator
> > > sleeps
> > > > while there are no inputs to process and it's default value is 10ms.
> > > >
> > > > Recently while I was doing performance experiments, reducing the
> > > > SPIN_MILLIS to 1ms increased the number of tuples processed by the
> > input
> > > > operator. Instead of manually adjusting the SPIN_MILLIS for best
> > > > performance, how about changing it based on the heuristic?
> > > >
> > > > Here is the idea
> > > > After sleeping for SPIN_MILLIS, if tuples are found reduce the
> > > SPIN_MILLIS
> > > > otherwise increase the SPIN_MILLIS.
> > > >
> > > > Let me know your thoughts.
> > > >
> > > > Thanks
> > > > Sandesh
> > > >
> > >
> >
>
Re: Adaptive SPIN_MILLIS attributes
Posted by Amol Kekre <am...@datatorrent.com>.
This is a good way, and it fits well with platform self-adjusting. We may
need to still look at a way to allow users to keep SPIN_MILLIS fixed in
case they depend on it for downstream intake.
Thks
Amol
On Thu, Mar 24, 2016 at 12:45 PM, Sandesh Hegde <sa...@datatorrent.com>
wrote:
> That is cool, we need to something similar for input operators.
>
> On Thu, Mar 24, 2016 at 12:32 PM Thomas Weise <th...@datatorrent.com>
> wrote:
>
> > Please see:
> >
> > https://issues.apache.org/jira/browse/APEXCORE-380
> > https://github.com/apache/incubator-apex-core/pull/271
> >
> >
> > On Thu, Mar 24, 2016 at 12:27 PM, Sandesh Hegde <sandesh@datatorrent.com
> >
> > wrote:
> >
> > > Hello Team,
> > >
> > > The attributes SPIN_MILLIS is the amount of time the input operator
> > sleeps
> > > while there are no inputs to process and it's default value is 10ms.
> > >
> > > Recently while I was doing performance experiments, reducing the
> > > SPIN_MILLIS to 1ms increased the number of tuples processed by the
> input
> > > operator. Instead of manually adjusting the SPIN_MILLIS for best
> > > performance, how about changing it based on the heuristic?
> > >
> > > Here is the idea
> > > After sleeping for SPIN_MILLIS, if tuples are found reduce the
> > SPIN_MILLIS
> > > otherwise increase the SPIN_MILLIS.
> > >
> > > Let me know your thoughts.
> > >
> > > Thanks
> > > Sandesh
> > >
> >
>
Re: Adaptive SPIN_MILLIS attributes
Posted by Sandesh Hegde <sa...@datatorrent.com>.
That is cool, we need to something similar for input operators.
On Thu, Mar 24, 2016 at 12:32 PM Thomas Weise <th...@datatorrent.com>
wrote:
> Please see:
>
> https://issues.apache.org/jira/browse/APEXCORE-380
> https://github.com/apache/incubator-apex-core/pull/271
>
>
> On Thu, Mar 24, 2016 at 12:27 PM, Sandesh Hegde <sa...@datatorrent.com>
> wrote:
>
> > Hello Team,
> >
> > The attributes SPIN_MILLIS is the amount of time the input operator
> sleeps
> > while there are no inputs to process and it's default value is 10ms.
> >
> > Recently while I was doing performance experiments, reducing the
> > SPIN_MILLIS to 1ms increased the number of tuples processed by the input
> > operator. Instead of manually adjusting the SPIN_MILLIS for best
> > performance, how about changing it based on the heuristic?
> >
> > Here is the idea
> > After sleeping for SPIN_MILLIS, if tuples are found reduce the
> SPIN_MILLIS
> > otherwise increase the SPIN_MILLIS.
> >
> > Let me know your thoughts.
> >
> > Thanks
> > Sandesh
> >
>
Re: Adaptive SPIN_MILLIS attributes
Posted by Thomas Weise <th...@datatorrent.com>.
Please see:
https://issues.apache.org/jira/browse/APEXCORE-380
https://github.com/apache/incubator-apex-core/pull/271
On Thu, Mar 24, 2016 at 12:27 PM, Sandesh Hegde <sa...@datatorrent.com>
wrote:
> Hello Team,
>
> The attributes SPIN_MILLIS is the amount of time the input operator sleeps
> while there are no inputs to process and it's default value is 10ms.
>
> Recently while I was doing performance experiments, reducing the
> SPIN_MILLIS to 1ms increased the number of tuples processed by the input
> operator. Instead of manually adjusting the SPIN_MILLIS for best
> performance, how about changing it based on the heuristic?
>
> Here is the idea
> After sleeping for SPIN_MILLIS, if tuples are found reduce the SPIN_MILLIS
> otherwise increase the SPIN_MILLIS.
>
> Let me know your thoughts.
>
> Thanks
> Sandesh
>