You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streams.apache.org by Steve Blackmon <sb...@apache.org> on 2015/02/11 16:53:13 UTC
[DISCUSS] Official library of examples
Reviving an old discussion:
I propose that we create an official streams-examples repository in the
apache git repository with the following characteristics:
- maven multi-level multi-module project
- reference streams-project as parent
- first level : runtime (local, dropwizard, pig, storm, etc...)
- below, individual examples modules built upon that runtime.
Each example would:
- Contain a primary class which uses a StreamsBuilder to launch an
example stream, or set of complementary streams.
- Alternatively, contain a script suitable for execution within a
specific runtime (pig/spark shell for example)
- Optionally, contain processors and utility classes suited to a
specific purpose within
- Optionally, override the behavior of providers, processors, utility
classes from the core project to achieve a specific purpose
- Contain a configuration class which declares all required and optional
configuration
- Contain a README.md conforming to a template TBD, describing what the
example does
- Contain at least one example configuration in src/main/resources and
in README.md
- Contain a machine readable graph definition (dot or suitable
alternative) and png version visible in README.md
- Contain integration test(s) demonstrating that a small input set
traverses the stream, resulting in expected outputs and (optionally)
properly handled exceptions.
- Describe requirements for a successful deployment and execution in
README.md.
- Build an executable uber-jar
- Build an executable docker image
https://github.com/steveblackmon/streams-examples is a collection of
modules that conform in varying degrees to this pattern. Less organized
and less tested than I'm proposing, but containing code & poms that are
minimal, readable, and useful "out-of-the-box". I'll commit to increasing
the quality of 5-10 of them and submitting them to
Anyone agree this is a good idea? Have concerns? Want to help?
Steve Blackmon
sblackmon@apache.org
Re: [DISCUSS] Official library of examples
Posted by Matt Franklin <m....@gmail.com>.
On Thu Feb 12 2015 at 11:09:35 AM Robert Douglas <
robert.baker.douglas@gmail.com> wrote:
> Hey Steve, I think this is a great idea. Streams has a ton of use cases and
> I think to a new user, they can be a bit daunting. I'd be happy to help
> shore up some of the examples.
>
I would recommend having infra setup a separate git repository and adding
modules to that from scratch, rather than importing them from another
repository.
>
> -- Robert
>
> On Wed, Feb 11, 2015 at 9:53 AM, Steve Blackmon <sb...@apache.org>
> wrote:
>
> > Reviving an old discussion:
> >
> > I propose that we create an official streams-examples repository in the
> > apache git repository with the following characteristics:
> >
> > - maven multi-level multi-module project
> > - reference streams-project as parent
> > - first level : runtime (local, dropwizard, pig, storm, etc...)
> > - below, individual examples modules built upon that runtime.
> >
> > Each example would:
> >
> > - Contain a primary class which uses a StreamsBuilder to launch an
> > example stream, or set of complementary streams.
> > - Alternatively, contain a script suitable for execution within a
> > specific runtime (pig/spark shell for example)
> > - Optionally, contain processors and utility classes suited to a
> > specific purpose within
> > - Optionally, override the behavior of providers, processors, utility
> > classes from the core project to achieve a specific purpose
> > - Contain a configuration class which declares all required and
> optional
> > configuration
> > - Contain a README.md conforming to a template TBD, describing what
> the
> > example does
> > - Contain at least one example configuration in src/main/resources and
> > in README.md
> > - Contain a machine readable graph definition (dot or suitable
> > alternative) and png version visible in README.md
> > - Contain integration test(s) demonstrating that a small input set
> > traverses the stream, resulting in expected outputs and (optionally)
> > properly handled exceptions.
> > - Describe requirements for a successful deployment and execution in
> > README.md.
> > - Build an executable uber-jar
> > - Build an executable docker image
> >
> > https://github.com/steveblackmon/streams-examples is a collection of
> > modules that conform in varying degrees to this pattern. Less organized
> > and less tested than I'm proposing, but containing code & poms that are
> > minimal, readable, and useful "out-of-the-box". I'll commit to
> increasing
> > the quality of 5-10 of them and submitting them to
> >
> > Anyone agree this is a good idea? Have concerns? Want to help?
> >
> > Steve Blackmon
> > sblackmon@apache.org
> >
>
Re: [DISCUSS] Official library of examples
Posted by Robert Douglas <ro...@gmail.com>.
Hey Steve, I think this is a great idea. Streams has a ton of use cases and
I think to a new user, they can be a bit daunting. I'd be happy to help
shore up some of the examples.
-- Robert
On Wed, Feb 11, 2015 at 9:53 AM, Steve Blackmon <sb...@apache.org>
wrote:
> Reviving an old discussion:
>
> I propose that we create an official streams-examples repository in the
> apache git repository with the following characteristics:
>
> - maven multi-level multi-module project
> - reference streams-project as parent
> - first level : runtime (local, dropwizard, pig, storm, etc...)
> - below, individual examples modules built upon that runtime.
>
> Each example would:
>
> - Contain a primary class which uses a StreamsBuilder to launch an
> example stream, or set of complementary streams.
> - Alternatively, contain a script suitable for execution within a
> specific runtime (pig/spark shell for example)
> - Optionally, contain processors and utility classes suited to a
> specific purpose within
> - Optionally, override the behavior of providers, processors, utility
> classes from the core project to achieve a specific purpose
> - Contain a configuration class which declares all required and optional
> configuration
> - Contain a README.md conforming to a template TBD, describing what the
> example does
> - Contain at least one example configuration in src/main/resources and
> in README.md
> - Contain a machine readable graph definition (dot or suitable
> alternative) and png version visible in README.md
> - Contain integration test(s) demonstrating that a small input set
> traverses the stream, resulting in expected outputs and (optionally)
> properly handled exceptions.
> - Describe requirements for a successful deployment and execution in
> README.md.
> - Build an executable uber-jar
> - Build an executable docker image
>
> https://github.com/steveblackmon/streams-examples is a collection of
> modules that conform in varying degrees to this pattern. Less organized
> and less tested than I'm proposing, but containing code & poms that are
> minimal, readable, and useful "out-of-the-box". I'll commit to increasing
> the quality of 5-10 of them and submitting them to
>
> Anyone agree this is a good idea? Have concerns? Want to help?
>
> Steve Blackmon
> sblackmon@apache.org
>