You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Timothy Farkas <ti...@datatorrent.com> on 2015/12/12 01:13:28 UTC
Operator Utility To Convert Time Into Number Of Windows
I'd like to propose creating an operator utility in
com.datatorrent.lib.operatorutils to convert time into number of windows
and tuples / second into tuples / window. The way in which this currently
can be specified is through a property like operationWindowCount or
tuplesPerWindow. This works but I've observed that this can lead to
confusion. The confusion comes from the fact that the people who develop
apps are not necessarily the ones who test deployment configurations and
manage the apps, so they think in time not application windows. Also the
default application window count and streaming window size is not common
knowledge, so it's not easy to compute the conversion from windows to time.
This small utility method could be reused across operators.
Any feedback?
Thanks,
Tim
Re: Operator Utility To Convert Time Into Number Of Windows
Posted by Siyuan Hua <si...@datatorrent.com>.
Hey Tim,
+1
Please add me to watcher list to the ticket for this. The highlevel API may
need this work.
For example, stream.window(TimeUnit.SECOND, 10) etc.
Thanks!
On Sat, Dec 12, 2015 at 5:36 PM, Timothy Farkas <ti...@datatorrent.com> wrote:
> will do
>
> On Sat, Dec 12, 2015 at 4:36 PM, Chandni Singh <ch...@datatorrent.com>
> wrote:
>
> > Hey Tim,
> >
> > There is a util package under library. Can this be put under
> util/windows ?
> >
> > Chandni
> >
> > On Fri, Dec 11, 2015 at 5:50 PM, Amol Kekre <am...@datatorrent.com>
> wrote:
> >
> > > Good idea.
> > >
> > > Thks
> > > Amol
> > >
> > > On Fri, Dec 11, 2015 at 4:13 PM, Timothy Farkas <ti...@datatorrent.com>
> > > wrote:
> > >
> > > > I'd like to propose creating an operator utility in
> > > > com.datatorrent.lib.operatorutils to convert time into number of
> > windows
> > > > and tuples / second into tuples / window. The way in which this
> > currently
> > > > can be specified is through a property like operationWindowCount or
> > > > tuplesPerWindow. This works but I've observed that this can lead to
> > > > confusion. The confusion comes from the fact that the people who
> > develop
> > > > apps are not necessarily the ones who test deployment configurations
> > and
> > > > manage the apps, so they think in time not application windows. Also
> > the
> > > > default application window count and streaming window size is not
> > common
> > > > knowledge, so it's not easy to compute the conversion from windows to
> > > time.
> > > > This small utility method could be reused across operators.
> > > >
> > > > Any feedback?
> > > >
> > > > Thanks,
> > > > Tim
> > > >
> > >
> >
>
Re: Operator Utility To Convert Time Into Number Of Windows
Posted by Timothy Farkas <ti...@datatorrent.com>.
will do
On Sat, Dec 12, 2015 at 4:36 PM, Chandni Singh <ch...@datatorrent.com>
wrote:
> Hey Tim,
>
> There is a util package under library. Can this be put under util/windows ?
>
> Chandni
>
> On Fri, Dec 11, 2015 at 5:50 PM, Amol Kekre <am...@datatorrent.com> wrote:
>
> > Good idea.
> >
> > Thks
> > Amol
> >
> > On Fri, Dec 11, 2015 at 4:13 PM, Timothy Farkas <ti...@datatorrent.com>
> > wrote:
> >
> > > I'd like to propose creating an operator utility in
> > > com.datatorrent.lib.operatorutils to convert time into number of
> windows
> > > and tuples / second into tuples / window. The way in which this
> currently
> > > can be specified is through a property like operationWindowCount or
> > > tuplesPerWindow. This works but I've observed that this can lead to
> > > confusion. The confusion comes from the fact that the people who
> develop
> > > apps are not necessarily the ones who test deployment configurations
> and
> > > manage the apps, so they think in time not application windows. Also
> the
> > > default application window count and streaming window size is not
> common
> > > knowledge, so it's not easy to compute the conversion from windows to
> > time.
> > > This small utility method could be reused across operators.
> > >
> > > Any feedback?
> > >
> > > Thanks,
> > > Tim
> > >
> >
>
Re: Operator Utility To Convert Time Into Number Of Windows
Posted by Chandni Singh <ch...@datatorrent.com>.
Hey Tim,
There is a util package under library. Can this be put under util/windows ?
Chandni
On Fri, Dec 11, 2015 at 5:50 PM, Amol Kekre <am...@datatorrent.com> wrote:
> Good idea.
>
> Thks
> Amol
>
> On Fri, Dec 11, 2015 at 4:13 PM, Timothy Farkas <ti...@datatorrent.com>
> wrote:
>
> > I'd like to propose creating an operator utility in
> > com.datatorrent.lib.operatorutils to convert time into number of windows
> > and tuples / second into tuples / window. The way in which this currently
> > can be specified is through a property like operationWindowCount or
> > tuplesPerWindow. This works but I've observed that this can lead to
> > confusion. The confusion comes from the fact that the people who develop
> > apps are not necessarily the ones who test deployment configurations and
> > manage the apps, so they think in time not application windows. Also the
> > default application window count and streaming window size is not common
> > knowledge, so it's not easy to compute the conversion from windows to
> time.
> > This small utility method could be reused across operators.
> >
> > Any feedback?
> >
> > Thanks,
> > Tim
> >
>
Re: Operator Utility To Convert Time Into Number Of Windows
Posted by Amol Kekre <am...@datatorrent.com>.
Good idea.
Thks
Amol
On Fri, Dec 11, 2015 at 4:13 PM, Timothy Farkas <ti...@datatorrent.com> wrote:
> I'd like to propose creating an operator utility in
> com.datatorrent.lib.operatorutils to convert time into number of windows
> and tuples / second into tuples / window. The way in which this currently
> can be specified is through a property like operationWindowCount or
> tuplesPerWindow. This works but I've observed that this can lead to
> confusion. The confusion comes from the fact that the people who develop
> apps are not necessarily the ones who test deployment configurations and
> manage the apps, so they think in time not application windows. Also the
> default application window count and streaming window size is not common
> knowledge, so it's not easy to compute the conversion from windows to time.
> This small utility method could be reused across operators.
>
> Any feedback?
>
> Thanks,
> Tim
>
Re: Operator Utility To Convert Time Into Number Of Windows
Posted by Chandni Singh <ch...@datatorrent.com>.
I agree with Tim. WindowGenerator is as its name suggests is not a general
purpose utility class that will be used by operator developer.
On Mon, Dec 14, 2015 at 10:24 AM, Timothy Farkas <ti...@datatorrent.com>
wrote:
> Hi Siyuan
>
> The ticket is here
>
> https://malhar.atlassian.net/browse/MLHR-1940
>
> A small set of utility methods have already been added to Malhar library
>
>
> https://github.com/apache/incubator-apex-malhar/blob/devel-3/library/src/main/java/com/datatorrent/lib/util/time/WindowUtils.java
>
> If you have any requests for additional methods to be used from your
> streaming api please let me know. It can also be moved if you need it to
> be. Since it looks like there is some interest and these things may change
> I will also mark the WinodwUtils class as Unstable.
>
> David
>
> I don't think these things should be moved to WindowGenerator since the
> time conversion requires information specific to an operator, namely the
> application window count. I think the WindowGenerator is a DAG level
> construct which has a specialized function, it shouldn't be populated with
> operator level conversion methods.
>
> Thanks,
> Tim
>
>
> On Mon, Dec 14, 2015 at 9:46 AM, David Yan <da...@datatorrent.com> wrote:
>
> > Similar utilities are already in WindowGenerator in apex core. Maybe add
> > to that class instead of creating a separate class?
> >
> > David
> > On Dec 11, 2015 4:13 PM, "Timothy Farkas" <ti...@datatorrent.com> wrote:
> >
> > > I'd like to propose creating an operator utility in
> > > com.datatorrent.lib.operatorutils to convert time into number of
> windows
> > > and tuples / second into tuples / window. The way in which this
> currently
> > > can be specified is through a property like operationWindowCount or
> > > tuplesPerWindow. This works but I've observed that this can lead to
> > > confusion. The confusion comes from the fact that the people who
> develop
> > > apps are not necessarily the ones who test deployment configurations
> and
> > > manage the apps, so they think in time not application windows. Also
> the
> > > default application window count and streaming window size is not
> common
> > > knowledge, so it's not easy to compute the conversion from windows to
> > time.
> > > This small utility method could be reused across operators.
> > >
> > > Any feedback?
> > >
> > > Thanks,
> > > Tim
> > >
> >
>
Re: Operator Utility To Convert Time Into Number Of Windows
Posted by Timothy Farkas <ti...@datatorrent.com>.
Hi Siyuan
The ticket is here
https://malhar.atlassian.net/browse/MLHR-1940
A small set of utility methods have already been added to Malhar library
https://github.com/apache/incubator-apex-malhar/blob/devel-3/library/src/main/java/com/datatorrent/lib/util/time/WindowUtils.java
If you have any requests for additional methods to be used from your
streaming api please let me know. It can also be moved if you need it to
be. Since it looks like there is some interest and these things may change
I will also mark the WinodwUtils class as Unstable.
David
I don't think these things should be moved to WindowGenerator since the
time conversion requires information specific to an operator, namely the
application window count. I think the WindowGenerator is a DAG level
construct which has a specialized function, it shouldn't be populated with
operator level conversion methods.
Thanks,
Tim
On Mon, Dec 14, 2015 at 9:46 AM, David Yan <da...@datatorrent.com> wrote:
> Similar utilities are already in WindowGenerator in apex core. Maybe add
> to that class instead of creating a separate class?
>
> David
> On Dec 11, 2015 4:13 PM, "Timothy Farkas" <ti...@datatorrent.com> wrote:
>
> > I'd like to propose creating an operator utility in
> > com.datatorrent.lib.operatorutils to convert time into number of windows
> > and tuples / second into tuples / window. The way in which this currently
> > can be specified is through a property like operationWindowCount or
> > tuplesPerWindow. This works but I've observed that this can lead to
> > confusion. The confusion comes from the fact that the people who develop
> > apps are not necessarily the ones who test deployment configurations and
> > manage the apps, so they think in time not application windows. Also the
> > default application window count and streaming window size is not common
> > knowledge, so it's not easy to compute the conversion from windows to
> time.
> > This small utility method could be reused across operators.
> >
> > Any feedback?
> >
> > Thanks,
> > Tim
> >
>
Re: Operator Utility To Convert Time Into Number Of Windows
Posted by David Yan <da...@datatorrent.com>.
Similar utilities are already in WindowGenerator in apex core. Maybe add
to that class instead of creating a separate class?
David
On Dec 11, 2015 4:13 PM, "Timothy Farkas" <ti...@datatorrent.com> wrote:
> I'd like to propose creating an operator utility in
> com.datatorrent.lib.operatorutils to convert time into number of windows
> and tuples / second into tuples / window. The way in which this currently
> can be specified is through a property like operationWindowCount or
> tuplesPerWindow. This works but I've observed that this can lead to
> confusion. The confusion comes from the fact that the people who develop
> apps are not necessarily the ones who test deployment configurations and
> manage the apps, so they think in time not application windows. Also the
> default application window count and streaming window size is not common
> knowledge, so it's not easy to compute the conversion from windows to time.
> This small utility method could be reused across operators.
>
> Any feedback?
>
> Thanks,
> Tim
>