You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Lakshmi Velineni <la...@datatorrent.com> on 2017/02/23 17:12:54 UTC
Re: example applications in malhar
I am ready to bring the examples over into the demos folder. I was
wondering if anybody has any input on Thomas's suggestion to rename the
demos folder to examples. I would rather do that first and then bring the
examples over instead of doing it the other way around as that would lead
to refactoring the new examples again.
Thanks
On Wed, Jan 25, 2017 at 8:12 AM, Lakshmi Velineni <la...@datatorrent.com>
wrote:
> Hi,
>
> Since the examples have little history I was planning to have two
> commits for every example, one for the code as the primary author of
> the example and another containing pom.xml and other changes to make
> it work under malhar.
>
> Thanks
>
> On Wed, Nov 2, 2016 at 9:49 PM, Lakshmi Velineni
> <la...@datatorrent.com> wrote:
> > Thanks for the suggestions and I am working on the process to migrate the
> > examples with the guidelines you mentioned. I will send out a list of
> > examples and the destination modules very soon.
> >
> >
> > On Thu, Oct 27, 2016 at 1:43 PM, Thomas Weise <th...@gmail.com>
> > wrote:
> >>
> >> Maybe a good first step would be to identify which examples to bring
> over
> >> and where appropriate how to structure them in Malhar (for example, I
> see
> >> multiple hdfs related apps that could go into the same Maven module).
> >>
> >>
> >> On Tue, Oct 25, 2016 at 1:00 PM, Thomas Weise <th...@apache.org> wrote:
> >>
> >> > That would be great. There are a few things to consider when working
> on
> >> > it:
> >> >
> >> > * preserve attribtion
> >> > * ensure there is a test that runs the application in the CI
> >> > * check that dependencies are compatible license
> >> > * maybe extract common boilerplate code from pom.xml
> >> >
> >> > etc.
> >> >
> >> > Existing examples are under https://github.com/apache/
> >> > apex-malhar/tree/master/demos
> >> >
> >> > Perhaps we should rename it to "examples"
> >> >
> >> > I also propose that each app has a README and we add those for
> existing
> >> > apps as well.
> >> >
> >> > Thanks,
> >> > Thomas
> >> >
> >> >
> >> >
> >> >
> >> > On Tue, Oct 25, 2016 at 12:49 PM, Lakshmi Velineni <
> >> > lakshmi@datatorrent.com> wrote:
> >> >
> >> >> Can i work on this?
> >> >>
> >> >> Thanks
> >> >> Lakshmi Prasanna
> >> >>
> >> >> On Mon, Sep 12, 2016 at 9:41 PM, Ashwin Chandra Putta <
> >> >> ashwinchandrap@gmail.com> wrote:
> >> >>
> >> >> > Here is the JIRA:
> >> >> > https://issues.apache.org/jira/browse/APEXMALHAR-2233
> >> >> >
> >> >> > On Tue, Sep 6, 2016 at 10:20 PM, Amol Kekre <am...@datatorrent.com>
> >> >> wrote:
> >> >> >
> >> >> > > Good idea to consolidate them into Malhar. We should bring in as
> >> >> > > many
> >> >> > from
> >> >> > > this gitHub as possible.
> >> >> > >
> >> >> > > Thks
> >> >> > > Amol
> >> >> > >
> >> >> > >
> >> >> > > On Tue, Sep 6, 2016 at 6:02 PM, Thomas Weise
> >> >> > > <th...@datatorrent.com>
> >> >> > > wrote:
> >> >> > >
> >> >> > > > I'm also for consolidating these different example locations.
> We
> >> >> should
> >> >> > > > also look if all of it is still relevant.
> >> >> > > >
> >> >> > > > The stuff from the DT repository needs to be brought into shape
> >> >> > > > wrt
> >> >> > > > licensing, checkstyle, CI support etc.
> >> >> > > >
> >> >> > > >
> >> >> > > > On Tue, Sep 6, 2016 at 4:34 PM, Pramod Immaneni <
> >> >> > pramod@datatorrent.com>
> >> >> > > > wrote:
> >> >> > > >
> >> >> > > > > Sounds like a good idea. How about merging demos with apps as
> >> >> well?
> >> >> > > > >
> >> >> > > > > On Tue, Sep 6, 2016 at 4:30 PM, Ashwin Chandra Putta <
> >> >> > > > > ashwinchandrap@gmail.com> wrote:
> >> >> > > > >
> >> >> > > > > > Hi All,
> >> >> > > > > >
> >> >> > > > > > We have a lot of examples for apex malhar operators in the
> >> >> > following
> >> >> > > > > > repository which resides outside of malhar.
> >> >> > > > > > https://github.com/DataTorrent/examples/tree/
> master/tutorials
> >> >> > > > > >
> >> >> > > > > > Now that it has grown quite a bit, does it make sense to
> >> >> > > > > > bring
> >> >> some
> >> >> > > of
> >> >> > > > > the
> >> >> > > > > > most common examples to malhar repository? Probably under
> >> >> > > > > > apps
> >> >> > > > directory?
> >> >> > > > > >
> >> >> > > > > > That way folks looking at malhar repository will have some
> >> >> samples
> >> >> > to
> >> >> > > > > look
> >> >> > > > > > at without having to search elsewhere.
> >> >> > > > > >
> >> >> > > > > > --
> >> >> > > > > >
> >> >> > > > > > Regards,
> >> >> > > > > > Ashwin.
> >> >> > > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> >
> >> >> > Regards,
> >> >> > Ashwin.
> >> >> >
> >> >>
> >> >
> >> >
> >
> >
>
Re: example applications in malhar
Posted by Amol Kekre <am...@datatorrent.com>.
+1 on renaming it to examples. Additionally we should ensure that the word
"demo" is not used at all. The word "demos" is very sales oriented and
should not be used. "Examples" are what folks learn from.
Thks
Amol
E:amol@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*
www.datatorrent.com | apex.apache.org
*Join us at Apex Big Data World-San Jose
<http://www.apexbigdata.com/san-jose.html>, April 4, 2017!*
[image: http://www.apexbigdata.com/san-jose-register.html]
<http://www.apexbigdata.com/san-jose-register.html>
On Thu, Feb 23, 2017 at 9:47 AM, Munagala Ramanath <ra...@datatorrent.com>
wrote:
> +1 for renaming to "examples"
>
> Ram
>
> On Thu, Feb 23, 2017 at 9:12 AM, Lakshmi Velineni <lakshmi@datatorrent.com
> >
> wrote:
>
> > I am ready to bring the examples over into the demos folder. I was
> > wondering if anybody has any input on Thomas's suggestion to rename the
> > demos folder to examples. I would rather do that first and then bring the
> > examples over instead of doing it the other way around as that would lead
> > to refactoring the new examples again.
> >
> > Thanks
> >
> > On Wed, Jan 25, 2017 at 8:12 AM, Lakshmi Velineni <
> lakshmi@datatorrent.com
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > Since the examples have little history I was planning to have two
> > > commits for every example, one for the code as the primary author of
> > > the example and another containing pom.xml and other changes to make
> > > it work under malhar.
> > >
> > > Thanks
> > >
> > > On Wed, Nov 2, 2016 at 9:49 PM, Lakshmi Velineni
> > > <la...@datatorrent.com> wrote:
> > > > Thanks for the suggestions and I am working on the process to migrate
> > the
> > > > examples with the guidelines you mentioned. I will send out a list of
> > > > examples and the destination modules very soon.
> > > >
> > > >
> > > > On Thu, Oct 27, 2016 at 1:43 PM, Thomas Weise <
> thomas.weise@gmail.com>
> > > > wrote:
> > > >>
> > > >> Maybe a good first step would be to identify which examples to bring
> > > over
> > > >> and where appropriate how to structure them in Malhar (for example,
> I
> > > see
> > > >> multiple hdfs related apps that could go into the same Maven
> module).
> > > >>
> > > >>
> > > >> On Tue, Oct 25, 2016 at 1:00 PM, Thomas Weise <th...@apache.org>
> wrote:
> > > >>
> > > >> > That would be great. There are a few things to consider when
> working
> > > on
> > > >> > it:
> > > >> >
> > > >> > * preserve attribtion
> > > >> > * ensure there is a test that runs the application in the CI
> > > >> > * check that dependencies are compatible license
> > > >> > * maybe extract common boilerplate code from pom.xml
> > > >> >
> > > >> > etc.
> > > >> >
> > > >> > Existing examples are under https://github.com/apache/
> > > >> > apex-malhar/tree/master/demos
> > > >> >
> > > >> > Perhaps we should rename it to "examples"
> > > >> >
> > > >> > I also propose that each app has a README and we add those for
> > > existing
> > > >> > apps as well.
> > > >> >
> > > >> > Thanks,
> > > >> > Thomas
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Tue, Oct 25, 2016 at 12:49 PM, Lakshmi Velineni <
> > > >> > lakshmi@datatorrent.com> wrote:
> > > >> >
> > > >> >> Can i work on this?
> > > >> >>
> > > >> >> Thanks
> > > >> >> Lakshmi Prasanna
> > > >> >>
> > > >> >> On Mon, Sep 12, 2016 at 9:41 PM, Ashwin Chandra Putta <
> > > >> >> ashwinchandrap@gmail.com> wrote:
> > > >> >>
> > > >> >> > Here is the JIRA:
> > > >> >> > https://issues.apache.org/jira/browse/APEXMALHAR-2233
> > > >> >> >
> > > >> >> > On Tue, Sep 6, 2016 at 10:20 PM, Amol Kekre <
> > amol@datatorrent.com>
> > > >> >> wrote:
> > > >> >> >
> > > >> >> > > Good idea to consolidate them into Malhar. We should bring in
> > as
> > > >> >> > > many
> > > >> >> > from
> > > >> >> > > this gitHub as possible.
> > > >> >> > >
> > > >> >> > > Thks
> > > >> >> > > Amol
> > > >> >> > >
> > > >> >> > >
> > > >> >> > > On Tue, Sep 6, 2016 at 6:02 PM, Thomas Weise
> > > >> >> > > <th...@datatorrent.com>
> > > >> >> > > wrote:
> > > >> >> > >
> > > >> >> > > > I'm also for consolidating these different example
> locations.
> > > We
> > > >> >> should
> > > >> >> > > > also look if all of it is still relevant.
> > > >> >> > > >
> > > >> >> > > > The stuff from the DT repository needs to be brought into
> > shape
> > > >> >> > > > wrt
> > > >> >> > > > licensing, checkstyle, CI support etc.
> > > >> >> > > >
> > > >> >> > > >
> > > >> >> > > > On Tue, Sep 6, 2016 at 4:34 PM, Pramod Immaneni <
> > > >> >> > pramod@datatorrent.com>
> > > >> >> > > > wrote:
> > > >> >> > > >
> > > >> >> > > > > Sounds like a good idea. How about merging demos with
> apps
> > as
> > > >> >> well?
> > > >> >> > > > >
> > > >> >> > > > > On Tue, Sep 6, 2016 at 4:30 PM, Ashwin Chandra Putta <
> > > >> >> > > > > ashwinchandrap@gmail.com> wrote:
> > > >> >> > > > >
> > > >> >> > > > > > Hi All,
> > > >> >> > > > > >
> > > >> >> > > > > > We have a lot of examples for apex malhar operators in
> > the
> > > >> >> > following
> > > >> >> > > > > > repository which resides outside of malhar.
> > > >> >> > > > > > https://github.com/DataTorrent/examples/tree/
> > > master/tutorials
> > > >> >> > > > > >
> > > >> >> > > > > > Now that it has grown quite a bit, does it make sense
> to
> > > >> >> > > > > > bring
> > > >> >> some
> > > >> >> > > of
> > > >> >> > > > > the
> > > >> >> > > > > > most common examples to malhar repository? Probably
> under
> > > >> >> > > > > > apps
> > > >> >> > > > directory?
> > > >> >> > > > > >
> > > >> >> > > > > > That way folks looking at malhar repository will have
> > some
> > > >> >> samples
> > > >> >> > to
> > > >> >> > > > > look
> > > >> >> > > > > > at without having to search elsewhere.
> > > >> >> > > > > >
> > > >> >> > > > > > --
> > > >> >> > > > > >
> > > >> >> > > > > > Regards,
> > > >> >> > > > > > Ashwin.
> > > >> >> > > > > >
> > > >> >> > > > >
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >> > --
> > > >> >> >
> > > >> >> > Regards,
> > > >> >> > Ashwin.
> > > >> >> >
> > > >> >>
> > > >> >
> > > >> >
> > > >
> > > >
> > >
> >
>
>
>
> --
>
> _______________________________________________________
>
> Munagala V. Ramanath
>
> Software Engineer
>
> E: ram@datatorrent.com | M: (408) 331-5034 | Twitter: @UnknownRam
>
> www.datatorrent.com | apex.apache.org
>
Re: example applications in malhar
Posted by Lakshmi Velineni <la...@datatorrent.com>.
Hi Thomas,
On Sun, Mar 12, 2017 at 10:13 AM, Thomas Weise <th...@apache.org> wrote:
> I think a flat list with good naming would make it easier to find a
> matching example.
>
I was looking at how examples are laid out in other projects like spark and
saw that there was top level classification such as r, ml, sql, streaming
etc., so I thought a top level folder, where there are lot of common themed
examples, such as datat movement, would be a good way to reduce clutter.
>
> topnwords ? There are already two related examples in Malhar: wordcount and
> twitter
>
It need not be brought over.
Can some of the examples be consolidated into a single example package,
> fileioXXX/fileoutput look like candidates.
>
I will look into it.
Thanks
Lakshmi Prasanna
>
>
> On Sat, Mar 11, 2017 at 6:36 PM, Lakshmi Velineni <lakshmi@datatorrent.com
> >
> wrote:
>
> > Hi,
> >
> > I have come up with the following list of applications to move from
> > datatorrent/examples into apex-malhar/examples. There were many examples
> > that do a similar functionality of moving data around from different
> > sources to destinations, so I separated them all into a separate folder
> > called "Ingestion".
> >
> > csvformatter
> >
> > dedup
> >
> > dynamic-partition
> >
> > enricher
> >
> > exactly-once
> >
> > filter
> >
> > innerjoin
> >
> > maprapp
> >
> > parser
> >
> > partition
> >
> > recordreader
> >
> > throttle
> >
> > topnwords
> >
> > transform
> >
> > unifiers
> >
> > *Ingestion/*
> >
> > cassandrainput
> >
> > cassandraoutput
> >
> > fileio
> >
> > fileio-multidir
> >
> > fileio-simple
> >
> > fileoutput
> >
> > filetojdbc
> >
> > hdfs2kafka
> >
> > hdfs-sync
> >
> > jdbcingest
> >
> > jdbctijdbc
> >
> > jmsActiveMQ
> >
> > jmsSqs
> >
> > kafka
> >
> > s3-to-hdfs-sync
> >
> > s3output
> >
> >
> > Let me know your thoughts.
> >
> >
> > Thanks
> >
> > Lakshmi Prasanna
> >
> >
> >
> >
> >
> >
> >
> > On Thu, Feb 23, 2017 at 10:34 AM, Amol Kekre <am...@datatorrent.com>
> wrote:
> >
> > > Yes, should merge samples into examples. Ideally the names of the
> > examples
> > > should be more descriptive in terms of "how to" as opposed to a title.
> PI
> > > demo for example has lots of ways to do compute. So it could be named
> as
> > > "pi - distributed compute" in examples. Similarly if others can be
> named
> > to
> > > bring out features to look at from the examples, it would make examples
> > > more useful to readers.
> > >
> > > Thks
> > > Amol
> > >
> > >
> > >
> > > E:amol@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*
> > >
> > > www.datatorrent.com | apex.apache.org
> > >
> > > *Join us at Apex Big Data World-San Jose
> > > <http://www.apexbigdata.com/san-jose.html>, April 4, 2017!*
> > > [image: http://www.apexbigdata.com/san-jose-register.html]
> > > <http://www.apexbigdata.com/san-jose-register.html>
> > >
> > > On Thu, Feb 23, 2017 at 10:07 AM, Sanjay Pujare <
> sanjay@datatorrent.com>
> > > wrote:
> > >
> > > > + for renaming to examples. While we are at it, how about merging
> > > "samples"
> > > > also in the new "examples" ?
> > > >
> > > > On Thu, Feb 23, 2017 at 9:47 AM, Munagala Ramanath <
> > ram@datatorrent.com>
> > > > wrote:
> > > >
> > > > > +1 for renaming to "examples"
> > > > >
> > > > > Ram
> > > > >
> > > > > On Thu, Feb 23, 2017 at 9:12 AM, Lakshmi Velineni <
> > > > lakshmi@datatorrent.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > I am ready to bring the examples over into the demos folder. I
> was
> > > > > > wondering if anybody has any input on Thomas's suggestion to
> rename
> > > the
> > > > > > demos folder to examples. I would rather do that first and then
> > bring
> > > > the
> > > > > > examples over instead of doing it the other way around as that
> > would
> > > > lead
> > > > > > to refactoring the new examples again.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Wed, Jan 25, 2017 at 8:12 AM, Lakshmi Velineni <
> > > > > lakshmi@datatorrent.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > Since the examples have little history I was planning to have
> two
> > > > > > > commits for every example, one for the code as the primary
> author
> > > of
> > > > > > > the example and another containing pom.xml and other changes to
> > > make
> > > > > > > it work under malhar.
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > On Wed, Nov 2, 2016 at 9:49 PM, Lakshmi Velineni
> > > > > > > <la...@datatorrent.com> wrote:
> > > > > > > > Thanks for the suggestions and I am working on the process to
> > > > migrate
> > > > > > the
> > > > > > > > examples with the guidelines you mentioned. I will send out a
> > > list
> > > > of
> > > > > > > > examples and the destination modules very soon.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Oct 27, 2016 at 1:43 PM, Thomas Weise <
> > > > > thomas.weise@gmail.com>
> > > > > > > > wrote:
> > > > > > > >>
> > > > > > > >> Maybe a good first step would be to identify which examples
> to
> > > > bring
> > > > > > > over
> > > > > > > >> and where appropriate how to structure them in Malhar (for
> > > > example,
> > > > > I
> > > > > > > see
> > > > > > > >> multiple hdfs related apps that could go into the same Maven
> > > > > module).
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Tue, Oct 25, 2016 at 1:00 PM, Thomas Weise <
> thw@apache.org
> > >
> > > > > wrote:
> > > > > > > >>
> > > > > > > >> > That would be great. There are a few things to consider
> when
> > > > > working
> > > > > > > on
> > > > > > > >> > it:
> > > > > > > >> >
> > > > > > > >> > * preserve attribtion
> > > > > > > >> > * ensure there is a test that runs the application in the
> CI
> > > > > > > >> > * check that dependencies are compatible license
> > > > > > > >> > * maybe extract common boilerplate code from pom.xml
> > > > > > > >> >
> > > > > > > >> > etc.
> > > > > > > >> >
> > > > > > > >> > Existing examples are under https://github.com/apache/
> > > > > > > >> > apex-malhar/tree/master/demos
> > > > > > > >> >
> > > > > > > >> > Perhaps we should rename it to "examples"
> > > > > > > >> >
> > > > > > > >> > I also propose that each app has a README and we add those
> > for
> > > > > > > existing
> > > > > > > >> > apps as well.
> > > > > > > >> >
> > > > > > > >> > Thanks,
> > > > > > > >> > Thomas
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Tue, Oct 25, 2016 at 12:49 PM, Lakshmi Velineni <
> > > > > > > >> > lakshmi@datatorrent.com> wrote:
> > > > > > > >> >
> > > > > > > >> >> Can i work on this?
> > > > > > > >> >>
> > > > > > > >> >> Thanks
> > > > > > > >> >> Lakshmi Prasanna
> > > > > > > >> >>
> > > > > > > >> >> On Mon, Sep 12, 2016 at 9:41 PM, Ashwin Chandra Putta <
> > > > > > > >> >> ashwinchandrap@gmail.com> wrote:
> > > > > > > >> >>
> > > > > > > >> >> > Here is the JIRA:
> > > > > > > >> >> > https://issues.apache.org/jira/browse/APEXMALHAR-2233
> > > > > > > >> >> >
> > > > > > > >> >> > On Tue, Sep 6, 2016 at 10:20 PM, Amol Kekre <
> > > > > > amol@datatorrent.com>
> > > > > > > >> >> wrote:
> > > > > > > >> >> >
> > > > > > > >> >> > > Good idea to consolidate them into Malhar. We should
> > > bring
> > > > in
> > > > > > as
> > > > > > > >> >> > > many
> > > > > > > >> >> > from
> > > > > > > >> >> > > this gitHub as possible.
> > > > > > > >> >> > >
> > > > > > > >> >> > > Thks
> > > > > > > >> >> > > Amol
> > > > > > > >> >> > >
> > > > > > > >> >> > >
> > > > > > > >> >> > > On Tue, Sep 6, 2016 at 6:02 PM, Thomas Weise
> > > > > > > >> >> > > <th...@datatorrent.com>
> > > > > > > >> >> > > wrote:
> > > > > > > >> >> > >
> > > > > > > >> >> > > > I'm also for consolidating these different example
> > > > > locations.
> > > > > > > We
> > > > > > > >> >> should
> > > > > > > >> >> > > > also look if all of it is still relevant.
> > > > > > > >> >> > > >
> > > > > > > >> >> > > > The stuff from the DT repository needs to be
> brought
> > > into
> > > > > > shape
> > > > > > > >> >> > > > wrt
> > > > > > > >> >> > > > licensing, checkstyle, CI support etc.
> > > > > > > >> >> > > >
> > > > > > > >> >> > > >
> > > > > > > >> >> > > > On Tue, Sep 6, 2016 at 4:34 PM, Pramod Immaneni <
> > > > > > > >> >> > pramod@datatorrent.com>
> > > > > > > >> >> > > > wrote:
> > > > > > > >> >> > > >
> > > > > > > >> >> > > > > Sounds like a good idea. How about merging demos
> > with
> > > > > apps
> > > > > > as
> > > > > > > >> >> well?
> > > > > > > >> >> > > > >
> > > > > > > >> >> > > > > On Tue, Sep 6, 2016 at 4:30 PM, Ashwin Chandra
> > Putta
> > > <
> > > > > > > >> >> > > > > ashwinchandrap@gmail.com> wrote:
> > > > > > > >> >> > > > >
> > > > > > > >> >> > > > > > Hi All,
> > > > > > > >> >> > > > > >
> > > > > > > >> >> > > > > > We have a lot of examples for apex malhar
> > operators
> > > > in
> > > > > > the
> > > > > > > >> >> > following
> > > > > > > >> >> > > > > > repository which resides outside of malhar.
> > > > > > > >> >> > > > > > https://github.com/DataTorrent/examples/tree/
> > > > > > > master/tutorials
> > > > > > > >> >> > > > > >
> > > > > > > >> >> > > > > > Now that it has grown quite a bit, does it make
> > > sense
> > > > > to
> > > > > > > >> >> > > > > > bring
> > > > > > > >> >> some
> > > > > > > >> >> > > of
> > > > > > > >> >> > > > > the
> > > > > > > >> >> > > > > > most common examples to malhar repository?
> > Probably
> > > > > under
> > > > > > > >> >> > > > > > apps
> > > > > > > >> >> > > > directory?
> > > > > > > >> >> > > > > >
> > > > > > > >> >> > > > > > That way folks looking at malhar repository
> will
> > > have
> > > > > > some
> > > > > > > >> >> samples
> > > > > > > >> >> > to
> > > > > > > >> >> > > > > look
> > > > > > > >> >> > > > > > at without having to search elsewhere.
> > > > > > > >> >> > > > > >
> > > > > > > >> >> > > > > > --
> > > > > > > >> >> > > > > >
> > > > > > > >> >> > > > > > Regards,
> > > > > > > >> >> > > > > > Ashwin.
> > > > > > > >> >> > > > > >
> > > > > > > >> >> > > > >
> > > > > > > >> >> > > >
> > > > > > > >> >> > >
> > > > > > > >> >> >
> > > > > > > >> >> >
> > > > > > > >> >> >
> > > > > > > >> >> > --
> > > > > > > >> >> >
> > > > > > > >> >> > Regards,
> > > > > > > >> >> > Ashwin.
> > > > > > > >> >> >
> > > > > > > >> >>
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > _______________________________________________________
> > > > >
> > > > > Munagala V. Ramanath
> > > > >
> > > > > Software Engineer
> > > > >
> > > > > E: ram@datatorrent.com | M: (408) 331-5034 | Twitter: @UnknownRam
> > > > >
> > > > > www.datatorrent.com | apex.apache.org
> > > > >
> > > >
> > >
> >
>
Re: example applications in malhar
Posted by Thomas Weise <th...@apache.org>.
I think a flat list with good naming would make it easier to find a
matching example.
topnwords ? There are already two related examples in Malhar: wordcount and
twitter
Can some of the examples be consolidated into a single example package,
fileioXXX/fileoutput look like candidates.
On Sat, Mar 11, 2017 at 6:36 PM, Lakshmi Velineni <la...@datatorrent.com>
wrote:
> Hi,
>
> I have come up with the following list of applications to move from
> datatorrent/examples into apex-malhar/examples. There were many examples
> that do a similar functionality of moving data around from different
> sources to destinations, so I separated them all into a separate folder
> called "Ingestion".
>
> csvformatter
>
> dedup
>
> dynamic-partition
>
> enricher
>
> exactly-once
>
> filter
>
> innerjoin
>
> maprapp
>
> parser
>
> partition
>
> recordreader
>
> throttle
>
> topnwords
>
> transform
>
> unifiers
>
> *Ingestion/*
>
> cassandrainput
>
> cassandraoutput
>
> fileio
>
> fileio-multidir
>
> fileio-simple
>
> fileoutput
>
> filetojdbc
>
> hdfs2kafka
>
> hdfs-sync
>
> jdbcingest
>
> jdbctijdbc
>
> jmsActiveMQ
>
> jmsSqs
>
> kafka
>
> s3-to-hdfs-sync
>
> s3output
>
>
> Let me know your thoughts.
>
>
> Thanks
>
> Lakshmi Prasanna
>
>
>
>
>
>
>
> On Thu, Feb 23, 2017 at 10:34 AM, Amol Kekre <am...@datatorrent.com> wrote:
>
> > Yes, should merge samples into examples. Ideally the names of the
> examples
> > should be more descriptive in terms of "how to" as opposed to a title. PI
> > demo for example has lots of ways to do compute. So it could be named as
> > "pi - distributed compute" in examples. Similarly if others can be named
> to
> > bring out features to look at from the examples, it would make examples
> > more useful to readers.
> >
> > Thks
> > Amol
> >
> >
> >
> > E:amol@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*
> >
> > www.datatorrent.com | apex.apache.org
> >
> > *Join us at Apex Big Data World-San Jose
> > <http://www.apexbigdata.com/san-jose.html>, April 4, 2017!*
> > [image: http://www.apexbigdata.com/san-jose-register.html]
> > <http://www.apexbigdata.com/san-jose-register.html>
> >
> > On Thu, Feb 23, 2017 at 10:07 AM, Sanjay Pujare <sa...@datatorrent.com>
> > wrote:
> >
> > > + for renaming to examples. While we are at it, how about merging
> > "samples"
> > > also in the new "examples" ?
> > >
> > > On Thu, Feb 23, 2017 at 9:47 AM, Munagala Ramanath <
> ram@datatorrent.com>
> > > wrote:
> > >
> > > > +1 for renaming to "examples"
> > > >
> > > > Ram
> > > >
> > > > On Thu, Feb 23, 2017 at 9:12 AM, Lakshmi Velineni <
> > > lakshmi@datatorrent.com
> > > > >
> > > > wrote:
> > > >
> > > > > I am ready to bring the examples over into the demos folder. I was
> > > > > wondering if anybody has any input on Thomas's suggestion to rename
> > the
> > > > > demos folder to examples. I would rather do that first and then
> bring
> > > the
> > > > > examples over instead of doing it the other way around as that
> would
> > > lead
> > > > > to refactoring the new examples again.
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Wed, Jan 25, 2017 at 8:12 AM, Lakshmi Velineni <
> > > > lakshmi@datatorrent.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Since the examples have little history I was planning to have two
> > > > > > commits for every example, one for the code as the primary author
> > of
> > > > > > the example and another containing pom.xml and other changes to
> > make
> > > > > > it work under malhar.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Wed, Nov 2, 2016 at 9:49 PM, Lakshmi Velineni
> > > > > > <la...@datatorrent.com> wrote:
> > > > > > > Thanks for the suggestions and I am working on the process to
> > > migrate
> > > > > the
> > > > > > > examples with the guidelines you mentioned. I will send out a
> > list
> > > of
> > > > > > > examples and the destination modules very soon.
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Oct 27, 2016 at 1:43 PM, Thomas Weise <
> > > > thomas.weise@gmail.com>
> > > > > > > wrote:
> > > > > > >>
> > > > > > >> Maybe a good first step would be to identify which examples to
> > > bring
> > > > > > over
> > > > > > >> and where appropriate how to structure them in Malhar (for
> > > example,
> > > > I
> > > > > > see
> > > > > > >> multiple hdfs related apps that could go into the same Maven
> > > > module).
> > > > > > >>
> > > > > > >>
> > > > > > >> On Tue, Oct 25, 2016 at 1:00 PM, Thomas Weise <thw@apache.org
> >
> > > > wrote:
> > > > > > >>
> > > > > > >> > That would be great. There are a few things to consider when
> > > > working
> > > > > > on
> > > > > > >> > it:
> > > > > > >> >
> > > > > > >> > * preserve attribtion
> > > > > > >> > * ensure there is a test that runs the application in the CI
> > > > > > >> > * check that dependencies are compatible license
> > > > > > >> > * maybe extract common boilerplate code from pom.xml
> > > > > > >> >
> > > > > > >> > etc.
> > > > > > >> >
> > > > > > >> > Existing examples are under https://github.com/apache/
> > > > > > >> > apex-malhar/tree/master/demos
> > > > > > >> >
> > > > > > >> > Perhaps we should rename it to "examples"
> > > > > > >> >
> > > > > > >> > I also propose that each app has a README and we add those
> for
> > > > > > existing
> > > > > > >> > apps as well.
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> > Thomas
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Tue, Oct 25, 2016 at 12:49 PM, Lakshmi Velineni <
> > > > > > >> > lakshmi@datatorrent.com> wrote:
> > > > > > >> >
> > > > > > >> >> Can i work on this?
> > > > > > >> >>
> > > > > > >> >> Thanks
> > > > > > >> >> Lakshmi Prasanna
> > > > > > >> >>
> > > > > > >> >> On Mon, Sep 12, 2016 at 9:41 PM, Ashwin Chandra Putta <
> > > > > > >> >> ashwinchandrap@gmail.com> wrote:
> > > > > > >> >>
> > > > > > >> >> > Here is the JIRA:
> > > > > > >> >> > https://issues.apache.org/jira/browse/APEXMALHAR-2233
> > > > > > >> >> >
> > > > > > >> >> > On Tue, Sep 6, 2016 at 10:20 PM, Amol Kekre <
> > > > > amol@datatorrent.com>
> > > > > > >> >> wrote:
> > > > > > >> >> >
> > > > > > >> >> > > Good idea to consolidate them into Malhar. We should
> > bring
> > > in
> > > > > as
> > > > > > >> >> > > many
> > > > > > >> >> > from
> > > > > > >> >> > > this gitHub as possible.
> > > > > > >> >> > >
> > > > > > >> >> > > Thks
> > > > > > >> >> > > Amol
> > > > > > >> >> > >
> > > > > > >> >> > >
> > > > > > >> >> > > On Tue, Sep 6, 2016 at 6:02 PM, Thomas Weise
> > > > > > >> >> > > <th...@datatorrent.com>
> > > > > > >> >> > > wrote:
> > > > > > >> >> > >
> > > > > > >> >> > > > I'm also for consolidating these different example
> > > > locations.
> > > > > > We
> > > > > > >> >> should
> > > > > > >> >> > > > also look if all of it is still relevant.
> > > > > > >> >> > > >
> > > > > > >> >> > > > The stuff from the DT repository needs to be brought
> > into
> > > > > shape
> > > > > > >> >> > > > wrt
> > > > > > >> >> > > > licensing, checkstyle, CI support etc.
> > > > > > >> >> > > >
> > > > > > >> >> > > >
> > > > > > >> >> > > > On Tue, Sep 6, 2016 at 4:34 PM, Pramod Immaneni <
> > > > > > >> >> > pramod@datatorrent.com>
> > > > > > >> >> > > > wrote:
> > > > > > >> >> > > >
> > > > > > >> >> > > > > Sounds like a good idea. How about merging demos
> with
> > > > apps
> > > > > as
> > > > > > >> >> well?
> > > > > > >> >> > > > >
> > > > > > >> >> > > > > On Tue, Sep 6, 2016 at 4:30 PM, Ashwin Chandra
> Putta
> > <
> > > > > > >> >> > > > > ashwinchandrap@gmail.com> wrote:
> > > > > > >> >> > > > >
> > > > > > >> >> > > > > > Hi All,
> > > > > > >> >> > > > > >
> > > > > > >> >> > > > > > We have a lot of examples for apex malhar
> operators
> > > in
> > > > > the
> > > > > > >> >> > following
> > > > > > >> >> > > > > > repository which resides outside of malhar.
> > > > > > >> >> > > > > > https://github.com/DataTorrent/examples/tree/
> > > > > > master/tutorials
> > > > > > >> >> > > > > >
> > > > > > >> >> > > > > > Now that it has grown quite a bit, does it make
> > sense
> > > > to
> > > > > > >> >> > > > > > bring
> > > > > > >> >> some
> > > > > > >> >> > > of
> > > > > > >> >> > > > > the
> > > > > > >> >> > > > > > most common examples to malhar repository?
> Probably
> > > > under
> > > > > > >> >> > > > > > apps
> > > > > > >> >> > > > directory?
> > > > > > >> >> > > > > >
> > > > > > >> >> > > > > > That way folks looking at malhar repository will
> > have
> > > > > some
> > > > > > >> >> samples
> > > > > > >> >> > to
> > > > > > >> >> > > > > look
> > > > > > >> >> > > > > > at without having to search elsewhere.
> > > > > > >> >> > > > > >
> > > > > > >> >> > > > > > --
> > > > > > >> >> > > > > >
> > > > > > >> >> > > > > > Regards,
> > > > > > >> >> > > > > > Ashwin.
> > > > > > >> >> > > > > >
> > > > > > >> >> > > > >
> > > > > > >> >> > > >
> > > > > > >> >> > >
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >> > --
> > > > > > >> >> >
> > > > > > >> >> > Regards,
> > > > > > >> >> > Ashwin.
> > > > > > >> >> >
> > > > > > >> >>
> > > > > > >> >
> > > > > > >> >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > _______________________________________________________
> > > >
> > > > Munagala V. Ramanath
> > > >
> > > > Software Engineer
> > > >
> > > > E: ram@datatorrent.com | M: (408) 331-5034 | Twitter: @UnknownRam
> > > >
> > > > www.datatorrent.com | apex.apache.org
> > > >
> > >
> >
>
Re: example applications in malhar
Posted by Lakshmi Velineni <la...@datatorrent.com>.
Hi,
I have come up with the following list of applications to move from
datatorrent/examples into apex-malhar/examples. There were many examples
that do a similar functionality of moving data around from different
sources to destinations, so I separated them all into a separate folder
called "Ingestion".
csvformatter
dedup
dynamic-partition
enricher
exactly-once
filter
innerjoin
maprapp
parser
partition
recordreader
throttle
topnwords
transform
unifiers
*Ingestion/*
cassandrainput
cassandraoutput
fileio
fileio-multidir
fileio-simple
fileoutput
filetojdbc
hdfs2kafka
hdfs-sync
jdbcingest
jdbctijdbc
jmsActiveMQ
jmsSqs
kafka
s3-to-hdfs-sync
s3output
Let me know your thoughts.
Thanks
Lakshmi Prasanna
On Thu, Feb 23, 2017 at 10:34 AM, Amol Kekre <am...@datatorrent.com> wrote:
> Yes, should merge samples into examples. Ideally the names of the examples
> should be more descriptive in terms of "how to" as opposed to a title. PI
> demo for example has lots of ways to do compute. So it could be named as
> "pi - distributed compute" in examples. Similarly if others can be named to
> bring out features to look at from the examples, it would make examples
> more useful to readers.
>
> Thks
> Amol
>
>
>
> E:amol@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*
>
> www.datatorrent.com | apex.apache.org
>
> *Join us at Apex Big Data World-San Jose
> <http://www.apexbigdata.com/san-jose.html>, April 4, 2017!*
> [image: http://www.apexbigdata.com/san-jose-register.html]
> <http://www.apexbigdata.com/san-jose-register.html>
>
> On Thu, Feb 23, 2017 at 10:07 AM, Sanjay Pujare <sa...@datatorrent.com>
> wrote:
>
> > + for renaming to examples. While we are at it, how about merging
> "samples"
> > also in the new "examples" ?
> >
> > On Thu, Feb 23, 2017 at 9:47 AM, Munagala Ramanath <ra...@datatorrent.com>
> > wrote:
> >
> > > +1 for renaming to "examples"
> > >
> > > Ram
> > >
> > > On Thu, Feb 23, 2017 at 9:12 AM, Lakshmi Velineni <
> > lakshmi@datatorrent.com
> > > >
> > > wrote:
> > >
> > > > I am ready to bring the examples over into the demos folder. I was
> > > > wondering if anybody has any input on Thomas's suggestion to rename
> the
> > > > demos folder to examples. I would rather do that first and then bring
> > the
> > > > examples over instead of doing it the other way around as that would
> > lead
> > > > to refactoring the new examples again.
> > > >
> > > > Thanks
> > > >
> > > > On Wed, Jan 25, 2017 at 8:12 AM, Lakshmi Velineni <
> > > lakshmi@datatorrent.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Since the examples have little history I was planning to have two
> > > > > commits for every example, one for the code as the primary author
> of
> > > > > the example and another containing pom.xml and other changes to
> make
> > > > > it work under malhar.
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Wed, Nov 2, 2016 at 9:49 PM, Lakshmi Velineni
> > > > > <la...@datatorrent.com> wrote:
> > > > > > Thanks for the suggestions and I am working on the process to
> > migrate
> > > > the
> > > > > > examples with the guidelines you mentioned. I will send out a
> list
> > of
> > > > > > examples and the destination modules very soon.
> > > > > >
> > > > > >
> > > > > > On Thu, Oct 27, 2016 at 1:43 PM, Thomas Weise <
> > > thomas.weise@gmail.com>
> > > > > > wrote:
> > > > > >>
> > > > > >> Maybe a good first step would be to identify which examples to
> > bring
> > > > > over
> > > > > >> and where appropriate how to structure them in Malhar (for
> > example,
> > > I
> > > > > see
> > > > > >> multiple hdfs related apps that could go into the same Maven
> > > module).
> > > > > >>
> > > > > >>
> > > > > >> On Tue, Oct 25, 2016 at 1:00 PM, Thomas Weise <th...@apache.org>
> > > wrote:
> > > > > >>
> > > > > >> > That would be great. There are a few things to consider when
> > > working
> > > > > on
> > > > > >> > it:
> > > > > >> >
> > > > > >> > * preserve attribtion
> > > > > >> > * ensure there is a test that runs the application in the CI
> > > > > >> > * check that dependencies are compatible license
> > > > > >> > * maybe extract common boilerplate code from pom.xml
> > > > > >> >
> > > > > >> > etc.
> > > > > >> >
> > > > > >> > Existing examples are under https://github.com/apache/
> > > > > >> > apex-malhar/tree/master/demos
> > > > > >> >
> > > > > >> > Perhaps we should rename it to "examples"
> > > > > >> >
> > > > > >> > I also propose that each app has a README and we add those for
> > > > > existing
> > > > > >> > apps as well.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > Thomas
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > On Tue, Oct 25, 2016 at 12:49 PM, Lakshmi Velineni <
> > > > > >> > lakshmi@datatorrent.com> wrote:
> > > > > >> >
> > > > > >> >> Can i work on this?
> > > > > >> >>
> > > > > >> >> Thanks
> > > > > >> >> Lakshmi Prasanna
> > > > > >> >>
> > > > > >> >> On Mon, Sep 12, 2016 at 9:41 PM, Ashwin Chandra Putta <
> > > > > >> >> ashwinchandrap@gmail.com> wrote:
> > > > > >> >>
> > > > > >> >> > Here is the JIRA:
> > > > > >> >> > https://issues.apache.org/jira/browse/APEXMALHAR-2233
> > > > > >> >> >
> > > > > >> >> > On Tue, Sep 6, 2016 at 10:20 PM, Amol Kekre <
> > > > amol@datatorrent.com>
> > > > > >> >> wrote:
> > > > > >> >> >
> > > > > >> >> > > Good idea to consolidate them into Malhar. We should
> bring
> > in
> > > > as
> > > > > >> >> > > many
> > > > > >> >> > from
> > > > > >> >> > > this gitHub as possible.
> > > > > >> >> > >
> > > > > >> >> > > Thks
> > > > > >> >> > > Amol
> > > > > >> >> > >
> > > > > >> >> > >
> > > > > >> >> > > On Tue, Sep 6, 2016 at 6:02 PM, Thomas Weise
> > > > > >> >> > > <th...@datatorrent.com>
> > > > > >> >> > > wrote:
> > > > > >> >> > >
> > > > > >> >> > > > I'm also for consolidating these different example
> > > locations.
> > > > > We
> > > > > >> >> should
> > > > > >> >> > > > also look if all of it is still relevant.
> > > > > >> >> > > >
> > > > > >> >> > > > The stuff from the DT repository needs to be brought
> into
> > > > shape
> > > > > >> >> > > > wrt
> > > > > >> >> > > > licensing, checkstyle, CI support etc.
> > > > > >> >> > > >
> > > > > >> >> > > >
> > > > > >> >> > > > On Tue, Sep 6, 2016 at 4:34 PM, Pramod Immaneni <
> > > > > >> >> > pramod@datatorrent.com>
> > > > > >> >> > > > wrote:
> > > > > >> >> > > >
> > > > > >> >> > > > > Sounds like a good idea. How about merging demos with
> > > apps
> > > > as
> > > > > >> >> well?
> > > > > >> >> > > > >
> > > > > >> >> > > > > On Tue, Sep 6, 2016 at 4:30 PM, Ashwin Chandra Putta
> <
> > > > > >> >> > > > > ashwinchandrap@gmail.com> wrote:
> > > > > >> >> > > > >
> > > > > >> >> > > > > > Hi All,
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > We have a lot of examples for apex malhar operators
> > in
> > > > the
> > > > > >> >> > following
> > > > > >> >> > > > > > repository which resides outside of malhar.
> > > > > >> >> > > > > > https://github.com/DataTorrent/examples/tree/
> > > > > master/tutorials
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > Now that it has grown quite a bit, does it make
> sense
> > > to
> > > > > >> >> > > > > > bring
> > > > > >> >> some
> > > > > >> >> > > of
> > > > > >> >> > > > > the
> > > > > >> >> > > > > > most common examples to malhar repository? Probably
> > > under
> > > > > >> >> > > > > > apps
> > > > > >> >> > > > directory?
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > That way folks looking at malhar repository will
> have
> > > > some
> > > > > >> >> samples
> > > > > >> >> > to
> > > > > >> >> > > > > look
> > > > > >> >> > > > > > at without having to search elsewhere.
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > --
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > Regards,
> > > > > >> >> > > > > > Ashwin.
> > > > > >> >> > > > > >
> > > > > >> >> > > > >
> > > > > >> >> > > >
> > > > > >> >> > >
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> > --
> > > > > >> >> >
> > > > > >> >> > Regards,
> > > > > >> >> > Ashwin.
> > > > > >> >> >
> > > > > >> >>
> > > > > >> >
> > > > > >> >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > _______________________________________________________
> > >
> > > Munagala V. Ramanath
> > >
> > > Software Engineer
> > >
> > > E: ram@datatorrent.com | M: (408) 331-5034 | Twitter: @UnknownRam
> > >
> > > www.datatorrent.com | apex.apache.org
> > >
> >
>
Re: example applications in malhar
Posted by Amol Kekre <am...@datatorrent.com>.
Yes, should merge samples into examples. Ideally the names of the examples
should be more descriptive in terms of "how to" as opposed to a title. PI
demo for example has lots of ways to do compute. So it could be named as
"pi - distributed compute" in examples. Similarly if others can be named to
bring out features to look at from the examples, it would make examples
more useful to readers.
Thks
Amol
E:amol@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*
www.datatorrent.com | apex.apache.org
*Join us at Apex Big Data World-San Jose
<http://www.apexbigdata.com/san-jose.html>, April 4, 2017!*
[image: http://www.apexbigdata.com/san-jose-register.html]
<http://www.apexbigdata.com/san-jose-register.html>
On Thu, Feb 23, 2017 at 10:07 AM, Sanjay Pujare <sa...@datatorrent.com>
wrote:
> + for renaming to examples. While we are at it, how about merging "samples"
> also in the new "examples" ?
>
> On Thu, Feb 23, 2017 at 9:47 AM, Munagala Ramanath <ra...@datatorrent.com>
> wrote:
>
> > +1 for renaming to "examples"
> >
> > Ram
> >
> > On Thu, Feb 23, 2017 at 9:12 AM, Lakshmi Velineni <
> lakshmi@datatorrent.com
> > >
> > wrote:
> >
> > > I am ready to bring the examples over into the demos folder. I was
> > > wondering if anybody has any input on Thomas's suggestion to rename the
> > > demos folder to examples. I would rather do that first and then bring
> the
> > > examples over instead of doing it the other way around as that would
> lead
> > > to refactoring the new examples again.
> > >
> > > Thanks
> > >
> > > On Wed, Jan 25, 2017 at 8:12 AM, Lakshmi Velineni <
> > lakshmi@datatorrent.com
> > > >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Since the examples have little history I was planning to have two
> > > > commits for every example, one for the code as the primary author of
> > > > the example and another containing pom.xml and other changes to make
> > > > it work under malhar.
> > > >
> > > > Thanks
> > > >
> > > > On Wed, Nov 2, 2016 at 9:49 PM, Lakshmi Velineni
> > > > <la...@datatorrent.com> wrote:
> > > > > Thanks for the suggestions and I am working on the process to
> migrate
> > > the
> > > > > examples with the guidelines you mentioned. I will send out a list
> of
> > > > > examples and the destination modules very soon.
> > > > >
> > > > >
> > > > > On Thu, Oct 27, 2016 at 1:43 PM, Thomas Weise <
> > thomas.weise@gmail.com>
> > > > > wrote:
> > > > >>
> > > > >> Maybe a good first step would be to identify which examples to
> bring
> > > > over
> > > > >> and where appropriate how to structure them in Malhar (for
> example,
> > I
> > > > see
> > > > >> multiple hdfs related apps that could go into the same Maven
> > module).
> > > > >>
> > > > >>
> > > > >> On Tue, Oct 25, 2016 at 1:00 PM, Thomas Weise <th...@apache.org>
> > wrote:
> > > > >>
> > > > >> > That would be great. There are a few things to consider when
> > working
> > > > on
> > > > >> > it:
> > > > >> >
> > > > >> > * preserve attribtion
> > > > >> > * ensure there is a test that runs the application in the CI
> > > > >> > * check that dependencies are compatible license
> > > > >> > * maybe extract common boilerplate code from pom.xml
> > > > >> >
> > > > >> > etc.
> > > > >> >
> > > > >> > Existing examples are under https://github.com/apache/
> > > > >> > apex-malhar/tree/master/demos
> > > > >> >
> > > > >> > Perhaps we should rename it to "examples"
> > > > >> >
> > > > >> > I also propose that each app has a README and we add those for
> > > > existing
> > > > >> > apps as well.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Thomas
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Tue, Oct 25, 2016 at 12:49 PM, Lakshmi Velineni <
> > > > >> > lakshmi@datatorrent.com> wrote:
> > > > >> >
> > > > >> >> Can i work on this?
> > > > >> >>
> > > > >> >> Thanks
> > > > >> >> Lakshmi Prasanna
> > > > >> >>
> > > > >> >> On Mon, Sep 12, 2016 at 9:41 PM, Ashwin Chandra Putta <
> > > > >> >> ashwinchandrap@gmail.com> wrote:
> > > > >> >>
> > > > >> >> > Here is the JIRA:
> > > > >> >> > https://issues.apache.org/jira/browse/APEXMALHAR-2233
> > > > >> >> >
> > > > >> >> > On Tue, Sep 6, 2016 at 10:20 PM, Amol Kekre <
> > > amol@datatorrent.com>
> > > > >> >> wrote:
> > > > >> >> >
> > > > >> >> > > Good idea to consolidate them into Malhar. We should bring
> in
> > > as
> > > > >> >> > > many
> > > > >> >> > from
> > > > >> >> > > this gitHub as possible.
> > > > >> >> > >
> > > > >> >> > > Thks
> > > > >> >> > > Amol
> > > > >> >> > >
> > > > >> >> > >
> > > > >> >> > > On Tue, Sep 6, 2016 at 6:02 PM, Thomas Weise
> > > > >> >> > > <th...@datatorrent.com>
> > > > >> >> > > wrote:
> > > > >> >> > >
> > > > >> >> > > > I'm also for consolidating these different example
> > locations.
> > > > We
> > > > >> >> should
> > > > >> >> > > > also look if all of it is still relevant.
> > > > >> >> > > >
> > > > >> >> > > > The stuff from the DT repository needs to be brought into
> > > shape
> > > > >> >> > > > wrt
> > > > >> >> > > > licensing, checkstyle, CI support etc.
> > > > >> >> > > >
> > > > >> >> > > >
> > > > >> >> > > > On Tue, Sep 6, 2016 at 4:34 PM, Pramod Immaneni <
> > > > >> >> > pramod@datatorrent.com>
> > > > >> >> > > > wrote:
> > > > >> >> > > >
> > > > >> >> > > > > Sounds like a good idea. How about merging demos with
> > apps
> > > as
> > > > >> >> well?
> > > > >> >> > > > >
> > > > >> >> > > > > On Tue, Sep 6, 2016 at 4:30 PM, Ashwin Chandra Putta <
> > > > >> >> > > > > ashwinchandrap@gmail.com> wrote:
> > > > >> >> > > > >
> > > > >> >> > > > > > Hi All,
> > > > >> >> > > > > >
> > > > >> >> > > > > > We have a lot of examples for apex malhar operators
> in
> > > the
> > > > >> >> > following
> > > > >> >> > > > > > repository which resides outside of malhar.
> > > > >> >> > > > > > https://github.com/DataTorrent/examples/tree/
> > > > master/tutorials
> > > > >> >> > > > > >
> > > > >> >> > > > > > Now that it has grown quite a bit, does it make sense
> > to
> > > > >> >> > > > > > bring
> > > > >> >> some
> > > > >> >> > > of
> > > > >> >> > > > > the
> > > > >> >> > > > > > most common examples to malhar repository? Probably
> > under
> > > > >> >> > > > > > apps
> > > > >> >> > > > directory?
> > > > >> >> > > > > >
> > > > >> >> > > > > > That way folks looking at malhar repository will have
> > > some
> > > > >> >> samples
> > > > >> >> > to
> > > > >> >> > > > > look
> > > > >> >> > > > > > at without having to search elsewhere.
> > > > >> >> > > > > >
> > > > >> >> > > > > > --
> > > > >> >> > > > > >
> > > > >> >> > > > > > Regards,
> > > > >> >> > > > > > Ashwin.
> > > > >> >> > > > > >
> > > > >> >> > > > >
> > > > >> >> > > >
> > > > >> >> > >
> > > > >> >> >
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > --
> > > > >> >> >
> > > > >> >> > Regards,
> > > > >> >> > Ashwin.
> > > > >> >> >
> > > > >> >>
> > > > >> >
> > > > >> >
> > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> >
> > _______________________________________________________
> >
> > Munagala V. Ramanath
> >
> > Software Engineer
> >
> > E: ram@datatorrent.com | M: (408) 331-5034 | Twitter: @UnknownRam
> >
> > www.datatorrent.com | apex.apache.org
> >
>
Re: example applications in malhar
Posted by Sanjay Pujare <sa...@datatorrent.com>.
+ for renaming to examples. While we are at it, how about merging "samples"
also in the new "examples" ?
On Thu, Feb 23, 2017 at 9:47 AM, Munagala Ramanath <ra...@datatorrent.com>
wrote:
> +1 for renaming to "examples"
>
> Ram
>
> On Thu, Feb 23, 2017 at 9:12 AM, Lakshmi Velineni <lakshmi@datatorrent.com
> >
> wrote:
>
> > I am ready to bring the examples over into the demos folder. I was
> > wondering if anybody has any input on Thomas's suggestion to rename the
> > demos folder to examples. I would rather do that first and then bring the
> > examples over instead of doing it the other way around as that would lead
> > to refactoring the new examples again.
> >
> > Thanks
> >
> > On Wed, Jan 25, 2017 at 8:12 AM, Lakshmi Velineni <
> lakshmi@datatorrent.com
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > Since the examples have little history I was planning to have two
> > > commits for every example, one for the code as the primary author of
> > > the example and another containing pom.xml and other changes to make
> > > it work under malhar.
> > >
> > > Thanks
> > >
> > > On Wed, Nov 2, 2016 at 9:49 PM, Lakshmi Velineni
> > > <la...@datatorrent.com> wrote:
> > > > Thanks for the suggestions and I am working on the process to migrate
> > the
> > > > examples with the guidelines you mentioned. I will send out a list of
> > > > examples and the destination modules very soon.
> > > >
> > > >
> > > > On Thu, Oct 27, 2016 at 1:43 PM, Thomas Weise <
> thomas.weise@gmail.com>
> > > > wrote:
> > > >>
> > > >> Maybe a good first step would be to identify which examples to bring
> > > over
> > > >> and where appropriate how to structure them in Malhar (for example,
> I
> > > see
> > > >> multiple hdfs related apps that could go into the same Maven
> module).
> > > >>
> > > >>
> > > >> On Tue, Oct 25, 2016 at 1:00 PM, Thomas Weise <th...@apache.org>
> wrote:
> > > >>
> > > >> > That would be great. There are a few things to consider when
> working
> > > on
> > > >> > it:
> > > >> >
> > > >> > * preserve attribtion
> > > >> > * ensure there is a test that runs the application in the CI
> > > >> > * check that dependencies are compatible license
> > > >> > * maybe extract common boilerplate code from pom.xml
> > > >> >
> > > >> > etc.
> > > >> >
> > > >> > Existing examples are under https://github.com/apache/
> > > >> > apex-malhar/tree/master/demos
> > > >> >
> > > >> > Perhaps we should rename it to "examples"
> > > >> >
> > > >> > I also propose that each app has a README and we add those for
> > > existing
> > > >> > apps as well.
> > > >> >
> > > >> > Thanks,
> > > >> > Thomas
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Tue, Oct 25, 2016 at 12:49 PM, Lakshmi Velineni <
> > > >> > lakshmi@datatorrent.com> wrote:
> > > >> >
> > > >> >> Can i work on this?
> > > >> >>
> > > >> >> Thanks
> > > >> >> Lakshmi Prasanna
> > > >> >>
> > > >> >> On Mon, Sep 12, 2016 at 9:41 PM, Ashwin Chandra Putta <
> > > >> >> ashwinchandrap@gmail.com> wrote:
> > > >> >>
> > > >> >> > Here is the JIRA:
> > > >> >> > https://issues.apache.org/jira/browse/APEXMALHAR-2233
> > > >> >> >
> > > >> >> > On Tue, Sep 6, 2016 at 10:20 PM, Amol Kekre <
> > amol@datatorrent.com>
> > > >> >> wrote:
> > > >> >> >
> > > >> >> > > Good idea to consolidate them into Malhar. We should bring in
> > as
> > > >> >> > > many
> > > >> >> > from
> > > >> >> > > this gitHub as possible.
> > > >> >> > >
> > > >> >> > > Thks
> > > >> >> > > Amol
> > > >> >> > >
> > > >> >> > >
> > > >> >> > > On Tue, Sep 6, 2016 at 6:02 PM, Thomas Weise
> > > >> >> > > <th...@datatorrent.com>
> > > >> >> > > wrote:
> > > >> >> > >
> > > >> >> > > > I'm also for consolidating these different example
> locations.
> > > We
> > > >> >> should
> > > >> >> > > > also look if all of it is still relevant.
> > > >> >> > > >
> > > >> >> > > > The stuff from the DT repository needs to be brought into
> > shape
> > > >> >> > > > wrt
> > > >> >> > > > licensing, checkstyle, CI support etc.
> > > >> >> > > >
> > > >> >> > > >
> > > >> >> > > > On Tue, Sep 6, 2016 at 4:34 PM, Pramod Immaneni <
> > > >> >> > pramod@datatorrent.com>
> > > >> >> > > > wrote:
> > > >> >> > > >
> > > >> >> > > > > Sounds like a good idea. How about merging demos with
> apps
> > as
> > > >> >> well?
> > > >> >> > > > >
> > > >> >> > > > > On Tue, Sep 6, 2016 at 4:30 PM, Ashwin Chandra Putta <
> > > >> >> > > > > ashwinchandrap@gmail.com> wrote:
> > > >> >> > > > >
> > > >> >> > > > > > Hi All,
> > > >> >> > > > > >
> > > >> >> > > > > > We have a lot of examples for apex malhar operators in
> > the
> > > >> >> > following
> > > >> >> > > > > > repository which resides outside of malhar.
> > > >> >> > > > > > https://github.com/DataTorrent/examples/tree/
> > > master/tutorials
> > > >> >> > > > > >
> > > >> >> > > > > > Now that it has grown quite a bit, does it make sense
> to
> > > >> >> > > > > > bring
> > > >> >> some
> > > >> >> > > of
> > > >> >> > > > > the
> > > >> >> > > > > > most common examples to malhar repository? Probably
> under
> > > >> >> > > > > > apps
> > > >> >> > > > directory?
> > > >> >> > > > > >
> > > >> >> > > > > > That way folks looking at malhar repository will have
> > some
> > > >> >> samples
> > > >> >> > to
> > > >> >> > > > > look
> > > >> >> > > > > > at without having to search elsewhere.
> > > >> >> > > > > >
> > > >> >> > > > > > --
> > > >> >> > > > > >
> > > >> >> > > > > > Regards,
> > > >> >> > > > > > Ashwin.
> > > >> >> > > > > >
> > > >> >> > > > >
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >> > --
> > > >> >> >
> > > >> >> > Regards,
> > > >> >> > Ashwin.
> > > >> >> >
> > > >> >>
> > > >> >
> > > >> >
> > > >
> > > >
> > >
> >
>
>
>
> --
>
> _______________________________________________________
>
> Munagala V. Ramanath
>
> Software Engineer
>
> E: ram@datatorrent.com | M: (408) 331-5034 | Twitter: @UnknownRam
>
> www.datatorrent.com | apex.apache.org
>
Re: example applications in malhar
Posted by Munagala Ramanath <ra...@datatorrent.com>.
+1 for renaming to "examples"
Ram
On Thu, Feb 23, 2017 at 9:12 AM, Lakshmi Velineni <la...@datatorrent.com>
wrote:
> I am ready to bring the examples over into the demos folder. I was
> wondering if anybody has any input on Thomas's suggestion to rename the
> demos folder to examples. I would rather do that first and then bring the
> examples over instead of doing it the other way around as that would lead
> to refactoring the new examples again.
>
> Thanks
>
> On Wed, Jan 25, 2017 at 8:12 AM, Lakshmi Velineni <lakshmi@datatorrent.com
> >
> wrote:
>
> > Hi,
> >
> > Since the examples have little history I was planning to have two
> > commits for every example, one for the code as the primary author of
> > the example and another containing pom.xml and other changes to make
> > it work under malhar.
> >
> > Thanks
> >
> > On Wed, Nov 2, 2016 at 9:49 PM, Lakshmi Velineni
> > <la...@datatorrent.com> wrote:
> > > Thanks for the suggestions and I am working on the process to migrate
> the
> > > examples with the guidelines you mentioned. I will send out a list of
> > > examples and the destination modules very soon.
> > >
> > >
> > > On Thu, Oct 27, 2016 at 1:43 PM, Thomas Weise <th...@gmail.com>
> > > wrote:
> > >>
> > >> Maybe a good first step would be to identify which examples to bring
> > over
> > >> and where appropriate how to structure them in Malhar (for example, I
> > see
> > >> multiple hdfs related apps that could go into the same Maven module).
> > >>
> > >>
> > >> On Tue, Oct 25, 2016 at 1:00 PM, Thomas Weise <th...@apache.org> wrote:
> > >>
> > >> > That would be great. There are a few things to consider when working
> > on
> > >> > it:
> > >> >
> > >> > * preserve attribtion
> > >> > * ensure there is a test that runs the application in the CI
> > >> > * check that dependencies are compatible license
> > >> > * maybe extract common boilerplate code from pom.xml
> > >> >
> > >> > etc.
> > >> >
> > >> > Existing examples are under https://github.com/apache/
> > >> > apex-malhar/tree/master/demos
> > >> >
> > >> > Perhaps we should rename it to "examples"
> > >> >
> > >> > I also propose that each app has a README and we add those for
> > existing
> > >> > apps as well.
> > >> >
> > >> > Thanks,
> > >> > Thomas
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On Tue, Oct 25, 2016 at 12:49 PM, Lakshmi Velineni <
> > >> > lakshmi@datatorrent.com> wrote:
> > >> >
> > >> >> Can i work on this?
> > >> >>
> > >> >> Thanks
> > >> >> Lakshmi Prasanna
> > >> >>
> > >> >> On Mon, Sep 12, 2016 at 9:41 PM, Ashwin Chandra Putta <
> > >> >> ashwinchandrap@gmail.com> wrote:
> > >> >>
> > >> >> > Here is the JIRA:
> > >> >> > https://issues.apache.org/jira/browse/APEXMALHAR-2233
> > >> >> >
> > >> >> > On Tue, Sep 6, 2016 at 10:20 PM, Amol Kekre <
> amol@datatorrent.com>
> > >> >> wrote:
> > >> >> >
> > >> >> > > Good idea to consolidate them into Malhar. We should bring in
> as
> > >> >> > > many
> > >> >> > from
> > >> >> > > this gitHub as possible.
> > >> >> > >
> > >> >> > > Thks
> > >> >> > > Amol
> > >> >> > >
> > >> >> > >
> > >> >> > > On Tue, Sep 6, 2016 at 6:02 PM, Thomas Weise
> > >> >> > > <th...@datatorrent.com>
> > >> >> > > wrote:
> > >> >> > >
> > >> >> > > > I'm also for consolidating these different example locations.
> > We
> > >> >> should
> > >> >> > > > also look if all of it is still relevant.
> > >> >> > > >
> > >> >> > > > The stuff from the DT repository needs to be brought into
> shape
> > >> >> > > > wrt
> > >> >> > > > licensing, checkstyle, CI support etc.
> > >> >> > > >
> > >> >> > > >
> > >> >> > > > On Tue, Sep 6, 2016 at 4:34 PM, Pramod Immaneni <
> > >> >> > pramod@datatorrent.com>
> > >> >> > > > wrote:
> > >> >> > > >
> > >> >> > > > > Sounds like a good idea. How about merging demos with apps
> as
> > >> >> well?
> > >> >> > > > >
> > >> >> > > > > On Tue, Sep 6, 2016 at 4:30 PM, Ashwin Chandra Putta <
> > >> >> > > > > ashwinchandrap@gmail.com> wrote:
> > >> >> > > > >
> > >> >> > > > > > Hi All,
> > >> >> > > > > >
> > >> >> > > > > > We have a lot of examples for apex malhar operators in
> the
> > >> >> > following
> > >> >> > > > > > repository which resides outside of malhar.
> > >> >> > > > > > https://github.com/DataTorrent/examples/tree/
> > master/tutorials
> > >> >> > > > > >
> > >> >> > > > > > Now that it has grown quite a bit, does it make sense to
> > >> >> > > > > > bring
> > >> >> some
> > >> >> > > of
> > >> >> > > > > the
> > >> >> > > > > > most common examples to malhar repository? Probably under
> > >> >> > > > > > apps
> > >> >> > > > directory?
> > >> >> > > > > >
> > >> >> > > > > > That way folks looking at malhar repository will have
> some
> > >> >> samples
> > >> >> > to
> > >> >> > > > > look
> > >> >> > > > > > at without having to search elsewhere.
> > >> >> > > > > >
> > >> >> > > > > > --
> > >> >> > > > > >
> > >> >> > > > > > Regards,
> > >> >> > > > > > Ashwin.
> > >> >> > > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> > --
> > >> >> >
> > >> >> > Regards,
> > >> >> > Ashwin.
> > >> >> >
> > >> >>
> > >> >
> > >> >
> > >
> > >
> >
>
--
_______________________________________________________
Munagala V. Ramanath
Software Engineer
E: ram@datatorrent.com | M: (408) 331-5034 | Twitter: @UnknownRam
www.datatorrent.com | apex.apache.org