You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by RJ Nowling <rn...@gmail.com> on 2015/08/23 18:53:58 UTC

Proposal for "BigTop Data Generators"

Hi BigTop,

I had a discussion with Jay yesterday, we'd like to propose a new component
for BigTop: BigTop Data Generators.

BigTop Data Generators would consist of a common set of libraries for
building data generators and three example data generators:

    * BigPetStore transaction generator (moved from BigPetStore)
    * BigTop Bazaar -- attendee movement and interactions with booths on a
showroom floor, at a conference, or at a mall
    * BigTop Weatherman -- stochastic weather simulation (temperature, wind
speed, wind chill, rainfall, etc.) per zip code.  (From a model trained on
NOAA historical weather data)

We believe that creating a common set of libraries will have several
benefits including:

     * Easier for others to build their own data generators
     * Make data generators smaller and easier to maintain
     * Share improvements across the data generators

More details on the libraries are below.

BigPetStore will be continue to focus on building  and maintaining
blueprints, powered by the BigTop Data Generators.

Our vision is that we get all of Apache coming to BigTop for tools for
building better, more comprehensive blueprints.  We want to support these
efforts through data generators and the initial set of blueprint we've been
building.

If the community is generally in support of this, I can create a top-level
"bigtop-data-generators" directory and put the data generators and
libraries in there.

Thanks!

RJ


-------
Library details:

So far, I've extracted the following common libraries:

     * Samplers -- provides classes for PDFs and various samplers
     * Name generator -- data set and samplers for generating names
     * Location data set -- data set and classes for US zip codes, their
GPS coordinates, median house hold incomes, and population sizes
     * Product generator -- library for enumerating products from a
specification file.  Comes with default specifications for BigPetStore

I also expect that I'll add libraries for:

      * Particle simulation -- customer movement in a room
      * Latent factor model generation -- generate latent factors and
customer weights to create something like MovieLens data.  Used in Bazaar
for booth preferences and potentially in BigPetStore for customer item
preferences

Most of these libraries came out of the BigPetStore data generator but the
other generators have been refactored to be based off the standard set of
libraries.

Re: Proposal for "BigTop Data Generators"

Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
Like this idea very much!

On Sun, Aug 23, 2015 at 9:53 AM, RJ Nowling <rn...@gmail.com> wrote:
> Hi BigTop,
>
> I had a discussion with Jay yesterday, we'd like to propose a new component
> for BigTop: BigTop Data Generators.
>
> BigTop Data Generators would consist of a common set of libraries for
> building data generators and three example data generators:
>
>     * BigPetStore transaction generator (moved from BigPetStore)
>     * BigTop Bazaar -- attendee movement and interactions with booths on a
> showroom floor, at a conference, or at a mall
>     * BigTop Weatherman -- stochastic weather simulation (temperature, wind
> speed, wind chill, rainfall, etc.) per zip code.  (From a model trained on
> NOAA historical weather data)
>
> We believe that creating a common set of libraries will have several
> benefits including:
>
>      * Easier for others to build their own data generators
>      * Make data generators smaller and easier to maintain
>      * Share improvements across the data generators
>
> More details on the libraries are below.
>
> BigPetStore will be continue to focus on building  and maintaining
> blueprints, powered by the BigTop Data Generators.
>
> Our vision is that we get all of Apache coming to BigTop for tools for
> building better, more comprehensive blueprints.  We want to support these
> efforts through data generators and the initial set of blueprint we've been
> building.
>
> If the community is generally in support of this, I can create a top-level
> "bigtop-data-generators" directory and put the data generators and
> libraries in there.
>
> Thanks!
>
> RJ
>
>
> -------
> Library details:
>
> So far, I've extracted the following common libraries:
>
>      * Samplers -- provides classes for PDFs and various samplers
>      * Name generator -- data set and samplers for generating names
>      * Location data set -- data set and classes for US zip codes, their
> GPS coordinates, median house hold incomes, and population sizes
>      * Product generator -- library for enumerating products from a
> specification file.  Comes with default specifications for BigPetStore
>
> I also expect that I'll add libraries for:
>
>       * Particle simulation -- customer movement in a room
>       * Latent factor model generation -- generate latent factors and
> customer weights to create something like MovieLens data.  Used in Bazaar
> for booth preferences and potentially in BigPetStore for customer item
> preferences
>
> Most of these libraries came out of the BigPetStore data generator but the
> other generators have been refactored to be based off the standard set of
> libraries.

Re: Proposal for "BigTop Data Generators"

Posted by RJ Nowling <rn...@gmail.com>.
Thanks all!

Jay and I have been talking about maven artifacts for a while.  We would
appreciate help or some docs on how best to do that.  :)

I set a gradle multi-project build within the data generators to make it
easy to build everything and their dependencies in go.  But that would
horribly complicate things on a BigTop-wide scale.


On Tue, Aug 25, 2015 at 8:41 PM, Konstantin Boudnik <co...@apache.org> wrote:

> It is pretty cool indeed!
>
> I wonder how it needs to be structured to be:
>  - easy to access/use from other components wherever it is needed
>  - doesn't interfere with the rest of the stack
>
> I guess one possible way would be to implement the generator as a set of
> maven
> artifacts, that could be installed/consumed transparently by just
> declaring a
> dependency e.g as proposed via top-level component.
>
> Another way is to have a new package like we do for bigtop-utils and such.
>
> Perhaps this discussion should be moved to JIRA or shall we continue on the
> dev@ ??
>
> Cos
>
> On Sun, Aug 23, 2015 at 11:53AM, RJ Nowling wrote:
> > Hi BigTop,
> >
> > I had a discussion with Jay yesterday, we'd like to propose a new
> component
> > for BigTop: BigTop Data Generators.
> >
> > BigTop Data Generators would consist of a common set of libraries for
> > building data generators and three example data generators:
> >
> >     * BigPetStore transaction generator (moved from BigPetStore)
> >     * BigTop Bazaar -- attendee movement and interactions with booths on
> a
> > showroom floor, at a conference, or at a mall
> >     * BigTop Weatherman -- stochastic weather simulation (temperature,
> wind
> > speed, wind chill, rainfall, etc.) per zip code.  (From a model trained
> on
> > NOAA historical weather data)
> >
> > We believe that creating a common set of libraries will have several
> > benefits including:
> >
> >      * Easier for others to build their own data generators
> >      * Make data generators smaller and easier to maintain
> >      * Share improvements across the data generators
> >
> > More details on the libraries are below.
> >
> > BigPetStore will be continue to focus on building  and maintaining
> > blueprints, powered by the BigTop Data Generators.
> >
> > Our vision is that we get all of Apache coming to BigTop for tools for
> > building better, more comprehensive blueprints.  We want to support these
> > efforts through data generators and the initial set of blueprint we've
> been
> > building.
> >
> > If the community is generally in support of this, I can create a
> top-level
> > "bigtop-data-generators" directory and put the data generators and
> > libraries in there.
> >
> > Thanks!
> >
> > RJ
> >
> >
> > -------
> > Library details:
> >
> > So far, I've extracted the following common libraries:
> >
> >      * Samplers -- provides classes for PDFs and various samplers
> >      * Name generator -- data set and samplers for generating names
> >      * Location data set -- data set and classes for US zip codes, their
> > GPS coordinates, median house hold incomes, and population sizes
> >      * Product generator -- library for enumerating products from a
> > specification file.  Comes with default specifications for BigPetStore
> >
> > I also expect that I'll add libraries for:
> >
> >       * Particle simulation -- customer movement in a room
> >       * Latent factor model generation -- generate latent factors and
> > customer weights to create something like MovieLens data.  Used in Bazaar
> > for booth preferences and potentially in BigPetStore for customer item
> > preferences
> >
> > Most of these libraries came out of the BigPetStore data generator but
> the
> > other generators have been refactored to be based off the standard set of
> > libraries.
>

Re: Proposal for "BigTop Data Generators"

Posted by Konstantin Boudnik <co...@apache.org>.
I don't think it should either docker or linux package. Docker assumes certain
isolation (despite poorly defined and implemented). So, if the sample is
generated inside of the docker and is needed elsewhere - it's a hassle. But if
the package is available, it could be easily installed inside of a container
and used as Nate has pointed out.

Cos

On Mon, Aug 31, 2015 at 06:24AM, Jay Vyas wrote:
> Nate: Good idea to abstract the interface one level higher....
> 
> How about a docker run command ? That is probably the easiest way for Linux
> folks to run one off Java apps nowadays.  
> 
> docker run bigtop/bigtop-data-gen --scheme weather --size 5GB --output
> data-dir --etc  foo --etc bar
> 
> I'm happy to curate such a docker image, I already am doing something like
> this in kube for bigtop-transaction-queue, which continuously pumps data
> generator outputs into a REST endpoint or file Queue... So it could be
> extended to support other generators.
> 
> 
> > om> <na...@reactor8.com> wrote:
> > 
> > Could picture at some point supporting something like this for non-jvm folk just looking for test/demo data:
> > 
> > apt-get install bigtop-data-gen
> > ~/ $ bigtop-data-gen --scheme weather --size 5GB --output data-dir --etc  foo --etc bar
> > 
> > 
> > 
> > -----Original Message-----
> > From: jay vyas [mailto:jayunit100.apache@gmail.com] 
> > Sent: Sunday, August 30, 2015 5:11 PM
> > To: dev@bigtop.apache.org
> > Subject: Re: Proposal for "BigTop Data Generators"
> > 
> > Hola nate.  Well, here are the Use cases I know of that I have used the data generators for.
> > 
> > Dockerfile:
> > 
> > (1) for testing kubernetes.  For this, I just use transaction-queue docker file.
> > (2) for testing GlusterFS small file workloads, maybe with other analytics tools...
> > 
> > Maven repo
> > 
> > (3) Java maprduce/ignite/spark applications, which can just add a mvn repo when compiling.  Java developers never add jars through RPM repos.
> > 
> > RPM/DEB packages:
> > 
> > I could see people using an RPM/DEB data generator, and I'm not against it.  But I simply don't know of any real world projects which *currently* need RPM/Deb packages, which is why I haven't bothered to propose it as a requirement.  Nevertheless linux packages are always a welcome addition if  someone wants to create em !
> > 
> > 
> > 
> > 
> >> On Sun, Aug 30, 2015 at 4:34 PM, <na...@reactor8.com> wrote:
> >> 
> >> Would container be in addition to deb/rpm, or instead of?  If latter 
> >> can we do deb/rpm as base then have container either created from them 
> >> or directly from artifacts?
> >> 
> >> On test usage side, seems could probably break up tests into 
> >> base/required and then optional/add-on tests/test-suites.  Think 
> >> remember seeing mention of certain tests that are failing at times on 
> >> certain component(s) anyways in the core builds but don’t mean that 
> >> the build is broken, so would make sense to have some clean up around those anyways.
> >> 
> >> -----Original Message-----
> >> From: RJ Nowling [mailto:rnowling@gmail.com]
> >> Sent: Sunday, August 30, 2015 1:11 PM
> >> To: dev@bigtop.apache.org
> >> Subject: Re: Proposal for "BigTop Data Generators"
> >> 
> >> I agree with the above. :)
> >> 
> >> On Sun, Aug 30, 2015 at 11:19 AM, Jay Vyas 
> >> <ja...@gmail.com>
> >> wrote:
> >> 
> >>> Hi RJ.
> >>> 
> >>> Maven repositories and docker containers for the transaction queue 
> >>> are good enough IMO.  That will give people a way to compose them in 
> >>> different idioms (one for Java folks, another for broader Linux 
> >>> audience
> >> ).
> >>> 
> >>> I think the lib designs are fairly intuitive.  I would say that we 
> >>> should constrain them all to being written in Java or Groovy to keep 
> >>> the bigtop theme of "JVM for everything" :).
> >>> 
> >>> Any particular questions you have around technical design can be 
> >>> followed in a JIRA or else maybe a Readme spec that goes in a  top 
> >>> level of the data-generators dir...
> >>> 
> >>>> On Aug 30, 2015, at 1:51 AM, RJ Nowling <rn...@gmail.com> wrote:
> >>>> 
> >>>> I'd like to keep this conversation going.
> >>>> 
> >>>> So here are a few discussion points:
> >>>> 
> >>>> 1. How do we want to make the data generators available?  Maven?
> >>>> RPMs
> >>> and
> >>>> Debs?
> >>>> 
> >>>> For now, I'm using a gradle multi-project build to easily build 
> >>>> and
> >>> install
> >>>> the BPS data generators and its libraries into a local maven repo.
> >>>> This makes development easy.  Eventually, I would like to post 
> >>>> binaries
> >>> through
> >>>> Maven for easy integration by users.  RPMs / Debs could be 
> >>>> interesting since I use a pattern where the data generators are 
> >>>> libraries (to support application integration / parallelization by 
> >>>> the host framework) but also provide CLI drivers for local testing.
> >>>> 
> >>>> 2.  The idea of using the data generators as part of the smoke 
> >>>> tests came up.  Since there is concern about making the data 
> >>>> generators required, we could offer the blueprints (BigPetStore) 
> >>>> as optional smoke tests.  Would that be a good compromise?
> >>>> 
> >>>> 3.  How will they be maintained?
> >>>> 
> >>>> I'll certainly add myself to the maintainers list and will be 
> >>>> taking responsibility.  I'm happy to have others help as well if 
> >>>> anyone wants to
> >>>> -- if not, that's cool, too.
> >>>> 
> >>>> 4. Is anyone interested at all in discussing library APIs and designs?
> >>>> What about internal interfaces and such?
> >>>> 
> >>>> 
> >>>> My plan was to add at least one more data generator (weather
> >>>> simulator)
> >>> to
> >>>> bigtop-data-generators in the short term.  However, given the 
> >>>> concerns raised by Cos (more discussion needed) and Olaf (don't 
> >>>> want to force data generators on unsuspecting users ;) ), I would 
> >>>> like to reach some
> >>> consensus
> >>>> on what people are concerned about and solutions.
> >>>> 
> >>>> On Thu, Aug 27, 2015 at 12:38 PM, Konstantin Boudnik 
> >>>> <co...@apache.org>
> >>> wrote:
> >>>> 
> >>>>> Fine by me. I have linked this thread to the JIRA ticket that RJ
> >>> created,
> >>>>> so
> >>>>> we have a way to connect one to another ;)
> >>>>> 
> >>>>>> On Thu, Aug 27, 2015 at 01:02PM, Olaf Flebbe wrote:
> >>>>>> Hi,
> >>>>>> 
> >>>>>> I am not confident that moving important design discussions with 
> >>>>>> impact
> >>>>> to
> >>>>>> the whole project to jira is a good idea.
> >>>>>> 
> >>>>>> In the current JIRA Traffic storm it is not easy to identify and 
> >>>>>> follow
> >>>>> important tickets.
> >>>>>> 
> >>>>>> Please keep discussions on the list or at least, please state on 
> >>>>>> this
> >>>>> list which Ticket to follow ...
> >>>>>> 
> >>>>>> Olaf
> >>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>>> Am 26.08.2015 um 22:56 schrieb Konstantin Boudnik <co...@apache.org>:
> >>>>>>> 
> >>>>>>> On Wed, Aug 26, 2015 at 10:38PM, Olaf Flebbe wrote:
> >>>>>>>> Hi,
> >>>>>>>> 
> >>>>>>>> Nive to have data generators in Bigtop.
> >>>>>>>> 
> >>>>>>>> But please do not include it in bigtop_utils, since this 
> >>>>>>>> package is mandatory. Not everyone needs a data generator .
> >>>>>>> 
> >>>>>>> Yup. And let's move further design discussion to the JIRA!
> >>>>>>> 
> >>>>>>>> Olaf
> >>>>>>>> 
> >>>>>>>> 
> >>>>>>>>> Am 26.08.2015 um 11:25 schrieb Jay Vyas <
> >>> jayunit100.apache@gmail.com
> >>>>>> :
> >>>>>>>>> 
> >>>>>>>>> Publishing the jar to bigtops maven is probably a good first 
> >>>>>>>>> step
> >>>>> ,Then apps can just include it as needed...?.
> >>>>>>>>> 
> >>>>>>>>> I'm not against packaging if someone wants packages for this.
> >>>>>>>>> Maybe
> >>>>> even include it in bigtop util ?
> >>>>>>>>> 
> >>>>>>>>> Let's move to jira,
> >>>>>>>>> 
> >>>>>>>>>> On Aug 25, 2015, at 9:41 PM, Konstantin Boudnik 
> >>>>>>>>>> <co...@apache.org>
> >>>>> wrote:
> >>>>>>>>>> 
> >>>>>>>>>> It is pretty cool indeed!
> >>>>>>>>>> 
> >>>>>>>>>> I wonder how it needs to be structured to be:
> >>>>>>>>>> - easy to access/use from other components wherever it is 
> >>>>>>>>>> needed
> >>>>>>>>>> - doesn't interfere with the rest of the stack
> >>>>>>>>>> 
> >>>>>>>>>> I guess one possible way would be to implement the generator 
> >>>>>>>>>> as a
> >>>>> set of maven
> >>>>>>>>>> artifacts, that could be installed/consumed transparently by 
> >>>>>>>>>> just
> >>>>> declaring a
> >>>>>>>>>> dependency e.g as proposed via top-level component.
> >>>>>>>>>> 
> >>>>>>>>>> Another way is to have a new package like we do for 
> >>>>>>>>>> bigtop-utils
> >>>>> and such.
> >>>>>>>>>> 
> >>>>>>>>>> Perhaps this discussion should be moved to JIRA or shall we
> >>>>> continue on the
> >>>>>>>>>> dev@ ??
> >>>>>>>>>> 
> >>>>>>>>>> Cos
> >>>>>>>>>> 
> >>>>>>>>>>> On Sun, Aug 23, 2015 at 11:53AM, RJ Nowling wrote:
> >>>>>>>>>>> Hi BigTop,
> >>>>>>>>>>> 
> >>>>>>>>>>> I had a discussion with Jay yesterday, we'd like to propose 
> >>>>>>>>>>> a new
> >>>>> component
> >>>>>>>>>>> for BigTop: BigTop Data Generators.
> >>>>>>>>>>> 
> >>>>>>>>>>> BigTop Data Generators would consist of a common set of 
> >>>>>>>>>>> libraries
> >>>>> for
> >>>>>>>>>>> building data generators and three example data generators:
> >>>>>>>>>>> 
> >>>>>>>>>>> * BigPetStore transaction generator (moved from 
> >>>>>>>>>>> BigPetStore)
> >>>>>>>>>>> * BigTop Bazaar -- attendee movement and interactions with 
> >>>>>>>>>>> booths
> >>>>> on a
> >>>>>>>>>>> showroom floor, at a conference, or at a mall
> >>>>>>>>>>> * BigTop Weatherman -- stochastic weather simulation
> >>>>> (temperature, wind
> >>>>>>>>>>> speed, wind chill, rainfall, etc.) per zip code.  (From a 
> >>>>>>>>>>> model
> >>>>> trained on
> >>>>>>>>>>> NOAA historical weather data)
> >>>>>>>>>>> 
> >>>>>>>>>>> We believe that creating a common set of libraries will 
> >>>>>>>>>>> have
> >>>>> several
> >>>>>>>>>>> benefits including:
> >>>>>>>>>>> 
> >>>>>>>>>>> * Easier for others to build their own data generators
> >>>>>>>>>>> * Make data generators smaller and easier to maintain
> >>>>>>>>>>> * Share improvements across the data generators
> >>>>>>>>>>> 
> >>>>>>>>>>> More details on the libraries are below.
> >>>>>>>>>>> 
> >>>>>>>>>>> BigPetStore will be continue to focus on building  and 
> >>>>>>>>>>> maintaining blueprints, powered by the BigTop Data Generators.
> >>>>>>>>>>> 
> >>>>>>>>>>> Our vision is that we get all of Apache coming to BigTop 
> >>>>>>>>>>> for tools
> >>>>> for
> >>>>>>>>>>> building better, more comprehensive blueprints.  We want to
> >>>>> support these
> >>>>>>>>>>> efforts through data generators and the initial set of 
> >>>>>>>>>>> blueprint
> >>>>> we've been
> >>>>>>>>>>> building.
> >>>>>>>>>>> 
> >>>>>>>>>>> If the community is generally in support of this, I can 
> >>>>>>>>>>> create a
> >>>>> top-level
> >>>>>>>>>>> "bigtop-data-generators" directory and put the data 
> >>>>>>>>>>> generators and libraries in there.
> >>>>>>>>>>> 
> >>>>>>>>>>> Thanks!
> >>>>>>>>>>> 
> >>>>>>>>>>> RJ
> >>>>>>>>>>> 
> >>>>>>>>>>> 
> >>>>>>>>>>> -------
> >>>>>>>>>>> Library details:
> >>>>>>>>>>> 
> >>>>>>>>>>> So far, I've extracted the following common libraries:
> >>>>>>>>>>> 
> >>>>>>>>>>> * Samplers -- provides classes for PDFs and various 
> >>>>>>>>>>> samplers
> >>>>>>>>>>> * Name generator -- data set and samplers for generating 
> >>>>>>>>>>> names
> >>>>>>>>>>> * Location data set -- data set and classes for US zip 
> >>>>>>>>>>> codes,
> >>>>> their
> >>>>>>>>>>> GPS coordinates, median house hold incomes, and population 
> >>>>>>>>>>> sizes
> >>>>>>>>>>> * Product generator -- library for enumerating products 
> >>>>>>>>>>> from a specification file.  Comes with default 
> >>>>>>>>>>> specifications for
> >>>>> BigPetStore
> >>>>>>>>>>> 
> >>>>>>>>>>> I also expect that I'll add libraries for:
> >>>>>>>>>>> 
> >>>>>>>>>>>  * Particle simulation -- customer movement in a room
> >>>>>>>>>>>  * Latent factor model generation -- generate latent 
> >>>>>>>>>>> factors and customer weights to create something like MovieLens data.
> >>>>>>>>>>> Used in
> >>>>> Bazaar
> >>>>>>>>>>> for booth preferences and potentially in BigPetStore for 
> >>>>>>>>>>> customer
> >>>>> item
> >>>>>>>>>>> preferences
> >>>>>>>>>>> 
> >>>>>>>>>>> Most of these libraries came out of the BigPetStore data 
> >>>>>>>>>>> generator
> >>>>> but the
> >>>>>>>>>>> other generators have been refactored to be based off the 
> >>>>>>>>>>> standard
> >>>>> set of
> >>>>>>>>>>> libraries.
> > 
> > 
> > --
> > jay vyas
> > 

Re: Proposal for "BigTop Data Generators"

Posted by RJ Nowling <rn...@gmail.com>.
Thanks for everyone's input!  I've haven't add new code to the data
generators (just refactorings, bug fixes) while this discussion is
ongoing.  I'd like to pick up the work again.

I think we can agree that we won't:
* Add the data generators to required packages so users aren't forced to
install them
* Make Docker the only delivery mechanism

and agree that the data generators will have CLIs.

The BigPetStore data generator already has CLI support built into the jar:
$ java -jar bigpetstore-data-generator-1.1.0-SNAPSHOT.jar outputDir nStores
nCustomers nPurchasingModels simulationLength seed

I think packaging, maven artifacts, Docker images, and integration with
smoke tests can all be handled in subsequent JIRAs.  If any of these
interest you, please file the appropriate JIRA.  :)

Does this sound good to everyone?

On Mon, Aug 31, 2015 at 7:08 PM, Konstantin Boudnik <co...@apache.org> wrote:

> On Mon, Aug 31, 2015 at 08:01PM, jay vyas wrote:
> > - I agree Gradle is better than "yet another script".
> >
> > - And docker container, even if suboptimal, as thin wrapper to gradle
> > container is all you need to deliver the data-generator on the masses so
> > they can try it w/ zero startup cost.
>
> That's true. But it didn't sound this way originally, hence I've asked.
>
> > On Mon, Aug 31, 2015 at 7:36 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> >
> > > Why do would we need yet another script (and potentially an extra
> readme to
> > > explain its command options) when we have the gradle?
> > >
> > > Cos
> > >
> > > On Mon, Aug 31, 2015 at 10:51PM, Olaf Flebbe wrote:
> > > > +1 to the CLI /shell script interface.
> > > >
> > > > If I can choose I like to have a apt-get install
> bigtop-datagenerator ,
> > > running for instance
> > > >
> > > > bigtop-data-generatoroutputDir nStores nCustomers nPurchasingModels
> > > simulationLength seed
> > > >
> > > > I can help out with packaging if needed.
> > > >
> > > > Why should we use the docker indirection for a plain CLI file ? Of
> > > course, We can provide a trivial Dockerfile to create a container
> supplying
> > > a JVM and running the CLI ... But I do not like to depend our services
> on
> > > docker registry more than we do now.
> > > >
> > > > Olaf
> > > >
> > > >
> > > >
> > > > > Am 31.08.2015 um 16:40 schrieb Evans Ye <ev...@apache.org>:
> > > > >
> > > > > I am very much like the shell script wrapper and docker image idea
> > > since
> > > > > that way we can integrate it directly with bigtop provisioner which
> > > yield a
> > > > > perfect ux for the whole things. I think its not too hard to do it
> > > both, we
> > > > > just need to add a parameter to turn the script into daemon mode.
> I see
> > > > > lots of image doing this way.
> > > > >
> > > > > docker run bigtop/bigtop-data-gen --scheme weather --size 5GB
> --output
> > > > > data-dir --etc  foo --etc bar --daemon
> > > > > 2015年8月31日 下午9:06於 "RJ Nowling" <rn...@gmail.com>寫道:
> > > > >
> > > > >> The BigPetStore, Bazaar, and weather data generators have
> > > single-threaded
> > > > >> command-line interfaces.  We could do the same with the smaller
> > > generators
> > > > >> (names, locations, etc.) if there is interest.
> > > > >>
> > > > >> On Mon, Aug 31, 2015 at 5:24 AM, Jay Vyas <
> > > jayunit100.apache@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >>> Nate: Good idea to abstract the interface one level higher....
> > > > >>>
> > > > >>> How about a docker run command ? That is probably the easiest
> way for
> > > > >>> Linux folks to run one off Java apps nowadays.
> > > > >>>
> > > > >>> docker run bigtop/bigtop-data-gen --scheme weather --size 5GB
> > > --output
> > > > >>> data-dir --etc  foo --etc bar
> > > > >>>
> > > > >>> I'm happy to curate such a docker image, I already am doing
> something
> > > > >> like
> > > > >>> this in kube for bigtop-transaction-queue, which continuously
> pumps
> > > data
> > > > >>> generator outputs into a REST endpoint or file
> > > > >>> Queue... So it could be extended to support other generators.
> > > > >>>
> > > > >>>
> > > > >>>> om> <na...@reactor8.com> wrote:
> > > > >>>>
> > > > >>>> Could picture at some point supporting something like this for
> > > non-jvm
> > > > >>> folk just looking for test/demo data:
> > > > >>>>
> > > > >>>> apt-get install bigtop-data-gen
> > > > >>>> ~/ $ bigtop-data-gen --scheme weather --size 5GB --output
> data-dir
> > > > >>> --etc  foo --etc bar
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> -----Original Message-----
> > > > >>>> From: jay vyas [mailto:jayunit100.apache@gmail.com]
> > > > >>>> Sent: Sunday, August 30, 2015 5:11 PM
> > > > >>>> To: dev@bigtop.apache.org
> > > > >>>> Subject: Re: Proposal for "BigTop Data Generators"
> > > > >>>>
> > > > >>>> Hola nate.  Well, here are the Use cases I know of that I have
> used
> > > the
> > > > >>> data generators for.
> > > > >>>>
> > > > >>>> Dockerfile:
> > > > >>>>
> > > > >>>> (1) for testing kubernetes.  For this, I just use
> transaction-queue
> > > > >>> docker file.
> > > > >>>> (2) for testing GlusterFS small file workloads, maybe with other
> > > > >>> analytics tools...
> > > > >>>>
> > > > >>>> Maven repo
> > > > >>>>
> > > > >>>> (3) Java maprduce/ignite/spark applications, which can just add
> a
> > > mvn
> > > > >>> repo when compiling.  Java developers never add jars through RPM
> > > repos.
> > > > >>>>
> > > > >>>> RPM/DEB packages:
> > > > >>>>
> > > > >>>> I could see people using an RPM/DEB data generator, and I'm not
> > > against
> > > > >>> it.  But I simply don't know of any real world projects which
> > > *currently*
> > > > >>> need RPM/Deb packages, which is why I haven't bothered to
> propose it
> > > as a
> > > > >>> requirement.  Nevertheless linux packages are always a welcome
> > > addition
> > > > >> if
> > > > >>> someone wants to create em !
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>> On Sun, Aug 30, 2015 at 4:34 PM, <na...@reactor8.com> wrote:
> > > > >>>>>
> > > > >>>>> Would container be in addition to deb/rpm, or instead of?  If
> > > latter
> > > > >>>>> can we do deb/rpm as base then have container either created
> from
> > > them
> > > > >>>>> or directly from artifacts?
> > > > >>>>>
> > > > >>>>> On test usage side, seems could probably break up tests into
> > > > >>>>> base/required and then optional/add-on tests/test-suites.
> Think
> > > > >>>>> remember seeing mention of certain tests that are failing at
> times
> > > on
> > > > >>>>> certain component(s) anyways in the core builds but don’t mean
> that
> > > > >>>>> the build is broken, so would make sense to have some clean up
> > > around
> > > > >>> those anyways.
> > > > >>>>>
> > > > >>>>> -----Original Message-----
> > > > >>>>> From: RJ Nowling [mailto:rnowling@gmail.com]
> > > > >>>>> Sent: Sunday, August 30, 2015 1:11 PM
> > > > >>>>> To: dev@bigtop.apache.org
> > > > >>>>> Subject: Re: Proposal for "BigTop Data Generators"
> > > > >>>>>
> > > > >>>>> I agree with the above. :)
> > > > >>>>>
> > > > >>>>> On Sun, Aug 30, 2015 at 11:19 AM, Jay Vyas
> > > > >>>>> <ja...@gmail.com>
> > > > >>>>> wrote:
> > > > >>>>>
> > > > >>>>>> Hi RJ.
> > > > >>>>>>
> > > > >>>>>> Maven repositories and docker containers for the transaction
> queue
> > > > >>>>>> are good enough IMO.  That will give people a way to compose
> them
> > > in
> > > > >>>>>> different idioms (one for Java folks, another for broader
> Linux
> > > > >>>>>> audience
> > > > >>>>> ).
> > > > >>>>>>
> > > > >>>>>> I think the lib designs are fairly intuitive.  I would say
> that we
> > > > >>>>>> should constrain them all to being written in Java or Groovy
> to
> > > keep
> > > > >>>>>> the bigtop theme of "JVM for everything" :).
> > > > >>>>>>
> > > > >>>>>> Any particular questions you have around technical design can
> be
> > > > >>>>>> followed in a JIRA or else maybe a Readme spec that goes in
> a  top
> > > > >>>>>> level of the data-generators dir...
> > > > >>>>>>
> > > > >>>>>>> On Aug 30, 2015, at 1:51 AM, RJ Nowling <rn...@gmail.com>
> > > wrote:
> > > > >>>>>>>
> > > > >>>>>>> I'd like to keep this conversation going.
> > > > >>>>>>>
> > > > >>>>>>> So here are a few discussion points:
> > > > >>>>>>>
> > > > >>>>>>> 1. How do we want to make the data generators available?
> Maven?
> > > > >>>>>>> RPMs
> > > > >>>>>> and
> > > > >>>>>>> Debs?
> > > > >>>>>>>
> > > > >>>>>>> For now, I'm using a gradle multi-project build to easily
> build
> > > > >>>>>>> and
> > > > >>>>>> install
> > > > >>>>>>> the BPS data generators and its libraries into a local maven
> > > repo.
> > > > >>>>>>> This makes development easy.  Eventually, I would like to
> post
> > > > >>>>>>> binaries
> > > > >>>>>> through
> > > > >>>>>>> Maven for easy integration by users.  RPMs / Debs could be
> > > > >>>>>>> interesting since I use a pattern where the data generators
> are
> > > > >>>>>>> libraries (to support application integration /
> parallelization
> > > by
> > > > >>>>>>> the host framework) but also provide CLI drivers for local
> > > testing.
> > > > >>>>>>>
> > > > >>>>>>> 2.  The idea of using the data generators as part of the
> smoke
> > > > >>>>>>> tests came up.  Since there is concern about making the data
> > > > >>>>>>> generators required, we could offer the blueprints
> (BigPetStore)
> > > > >>>>>>> as optional smoke tests.  Would that be a good compromise?
> > > > >>>>>>>
> > > > >>>>>>> 3.  How will they be maintained?
> > > > >>>>>>>
> > > > >>>>>>> I'll certainly add myself to the maintainers list and will be
> > > > >>>>>>> taking responsibility.  I'm happy to have others help as
> well if
> > > > >>>>>>> anyone wants to
> > > > >>>>>>> -- if not, that's cool, too.
> > > > >>>>>>>
> > > > >>>>>>> 4. Is anyone interested at all in discussing library APIs and
> > > > >> designs?
> > > > >>>>>>> What about internal interfaces and such?
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> My plan was to add at least one more data generator (weather
> > > > >>>>>>> simulator)
> > > > >>>>>> to
> > > > >>>>>>> bigtop-data-generators in the short term.  However, given the
> > > > >>>>>>> concerns raised by Cos (more discussion needed) and Olaf
> (don't
> > > > >>>>>>> want to force data generators on unsuspecting users ;) ), I
> would
> > > > >>>>>>> like to reach some
> > > > >>>>>> consensus
> > > > >>>>>>> on what people are concerned about and solutions.
> > > > >>>>>>>
> > > > >>>>>>> On Thu, Aug 27, 2015 at 12:38 PM, Konstantin Boudnik
> > > > >>>>>>> <co...@apache.org>
> > > > >>>>>> wrote:
> > > > >>>>>>>
> > > > >>>>>>>> Fine by me. I have linked this thread to the JIRA ticket
> that RJ
> > > > >>>>>> created,
> > > > >>>>>>>> so
> > > > >>>>>>>> we have a way to connect one to another ;)
> > > > >>>>>>>>
> > > > >>>>>>>>> On Thu, Aug 27, 2015 at 01:02PM, Olaf Flebbe wrote:
> > > > >>>>>>>>> Hi,
> > > > >>>>>>>>>
> > > > >>>>>>>>> I am not confident that moving important design discussions
> > > with
> > > > >>>>>>>>> impact
> > > > >>>>>>>> to
> > > > >>>>>>>>> the whole project to jira is a good idea.
> > > > >>>>>>>>>
> > > > >>>>>>>>> In the current JIRA Traffic storm it is not easy to
> identify
> > > and
> > > > >>>>>>>>> follow
> > > > >>>>>>>> important tickets.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Please keep discussions on the list or at least, please
> state
> > > on
> > > > >>>>>>>>> this
> > > > >>>>>>>> list which Ticket to follow ...
> > > > >>>>>>>>>
> > > > >>>>>>>>> Olaf
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>>> Am 26.08.2015 um 22:56 schrieb Konstantin Boudnik <
> > > > >> cos@apache.org
> > > > >>>> :
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Wed, Aug 26, 2015 at 10:38PM, Olaf Flebbe wrote:
> > > > >>>>>>>>>>> Hi,
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Nive to have data generators in Bigtop.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> But please do not include it in bigtop_utils, since this
> > > > >>>>>>>>>>> package is mandatory. Not everyone needs a data
> generator .
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Yup. And let's move further design discussion to the JIRA!
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> Olaf
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> Am 26.08.2015 um 11:25 schrieb Jay Vyas <
> > > > >>>>>> jayunit100.apache@gmail.com
> > > > >>>>>>>>> :
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Publishing the jar to bigtops maven is probably a good
> first
> > > > >>>>>>>>>>>> step
> > > > >>>>>>>> ,Then apps can just include it as needed...?.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> I'm not against packaging if someone wants packages for
> > > this.
> > > > >>>>>>>>>>>> Maybe
> > > > >>>>>>>> even include it in bigtop util ?
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Let's move to jira,
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> On Aug 25, 2015, at 9:41 PM, Konstantin Boudnik
> > > > >>>>>>>>>>>>> <co...@apache.org>
> > > > >>>>>>>> wrote:
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> It is pretty cool indeed!
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> I wonder how it needs to be structured to be:
> > > > >>>>>>>>>>>>> - easy to access/use from other components wherever it
> is
> > > > >>>>>>>>>>>>> needed
> > > > >>>>>>>>>>>>> - doesn't interfere with the rest of the stack
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> I guess one possible way would be to implement the
> > > generator
> > > > >>>>>>>>>>>>> as a
> > > > >>>>>>>> set of maven
> > > > >>>>>>>>>>>>> artifacts, that could be installed/consumed
> transparently
> > > by
> > > > >>>>>>>>>>>>> just
> > > > >>>>>>>> declaring a
> > > > >>>>>>>>>>>>> dependency e.g as proposed via top-level component.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Another way is to have a new package like we do for
> > > > >>>>>>>>>>>>> bigtop-utils
> > > > >>>>>>>> and such.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Perhaps this discussion should be moved to JIRA or
> shall we
> > > > >>>>>>>> continue on the
> > > > >>>>>>>>>>>>> dev@ ??
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Cos
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> On Sun, Aug 23, 2015 at 11:53AM, RJ Nowling wrote:
> > > > >>>>>>>>>>>>>> Hi BigTop,
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> I had a discussion with Jay yesterday, we'd like to
> > > propose
> > > > >>>>>>>>>>>>>> a new
> > > > >>>>>>>> component
> > > > >>>>>>>>>>>>>> for BigTop: BigTop Data Generators.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> BigTop Data Generators would consist of a common set
> of
> > > > >>>>>>>>>>>>>> libraries
> > > > >>>>>>>> for
> > > > >>>>>>>>>>>>>> building data generators and three example data
> > > generators:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> * BigPetStore transaction generator (moved from
> > > > >>>>>>>>>>>>>> BigPetStore)
> > > > >>>>>>>>>>>>>> * BigTop Bazaar -- attendee movement and interactions
> with
> > > > >>>>>>>>>>>>>> booths
> > > > >>>>>>>> on a
> > > > >>>>>>>>>>>>>> showroom floor, at a conference, or at a mall
> > > > >>>>>>>>>>>>>> * BigTop Weatherman -- stochastic weather simulation
> > > > >>>>>>>> (temperature, wind
> > > > >>>>>>>>>>>>>> speed, wind chill, rainfall, etc.) per zip code.
> (From a
> > > > >>>>>>>>>>>>>> model
> > > > >>>>>>>> trained on
> > > > >>>>>>>>>>>>>> NOAA historical weather data)
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> We believe that creating a common set of libraries
> will
> > > > >>>>>>>>>>>>>> have
> > > > >>>>>>>> several
> > > > >>>>>>>>>>>>>> benefits including:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> * Easier for others to build their own data generators
> > > > >>>>>>>>>>>>>> * Make data generators smaller and easier to maintain
> > > > >>>>>>>>>>>>>> * Share improvements across the data generators
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> More details on the libraries are below.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> BigPetStore will be continue to focus on building  and
> > > > >>>>>>>>>>>>>> maintaining blueprints, powered by the BigTop Data
> > > > >> Generators.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Our vision is that we get all of Apache coming to
> BigTop
> > > > >>>>>>>>>>>>>> for tools
> > > > >>>>>>>> for
> > > > >>>>>>>>>>>>>> building better, more comprehensive blueprints.  We
> want
> > > to
> > > > >>>>>>>> support these
> > > > >>>>>>>>>>>>>> efforts through data generators and the initial set of
> > > > >>>>>>>>>>>>>> blueprint
> > > > >>>>>>>> we've been
> > > > >>>>>>>>>>>>>> building.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> If the community is generally in support of this, I
> can
> > > > >>>>>>>>>>>>>> create a
> > > > >>>>>>>> top-level
> > > > >>>>>>>>>>>>>> "bigtop-data-generators" directory and put the data
> > > > >>>>>>>>>>>>>> generators and libraries in there.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Thanks!
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> RJ
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> -------
> > > > >>>>>>>>>>>>>> Library details:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> So far, I've extracted the following common libraries:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> * Samplers -- provides classes for PDFs and various
> > > > >>>>>>>>>>>>>> samplers
> > > > >>>>>>>>>>>>>> * Name generator -- data set and samplers for
> generating
> > > > >>>>>>>>>>>>>> names
> > > > >>>>>>>>>>>>>> * Location data set -- data set and classes for US zip
> > > > >>>>>>>>>>>>>> codes,
> > > > >>>>>>>> their
> > > > >>>>>>>>>>>>>> GPS coordinates, median house hold incomes, and
> population
> > > > >>>>>>>>>>>>>> sizes
> > > > >>>>>>>>>>>>>> * Product generator -- library for enumerating
> products
> > > > >>>>>>>>>>>>>> from a specification file.  Comes with default
> > > > >>>>>>>>>>>>>> specifications for
> > > > >>>>>>>> BigPetStore
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> I also expect that I'll add libraries for:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> * Particle simulation -- customer movement in a room
> > > > >>>>>>>>>>>>>> * Latent factor model generation -- generate latent
> > > > >>>>>>>>>>>>>> factors and customer weights to create something like
> > > > >>> MovieLens data.
> > > > >>>>>>>>>>>>>> Used in
> > > > >>>>>>>> Bazaar
> > > > >>>>>>>>>>>>>> for booth preferences and potentially in BigPetStore
> for
> > > > >>>>>>>>>>>>>> customer
> > > > >>>>>>>> item
> > > > >>>>>>>>>>>>>> preferences
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Most of these libraries came out of the BigPetStore
> data
> > > > >>>>>>>>>>>>>> generator
> > > > >>>>>>>> but the
> > > > >>>>>>>>>>>>>> other generators have been refactored to be based off
> the
> > > > >>>>>>>>>>>>>> standard
> > > > >>>>>>>> set of
> > > > >>>>>>>>>>>>>> libraries.
> > > > >>>>
> > > > >>>>
> > > > >>>> --
> > > > >>>> jay vyas
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > >
> > >
> > >
> >
> >
> > --
> > jay vyas
>

Re: Proposal for "BigTop Data Generators"

Posted by Konstantin Boudnik <co...@apache.org>.
On Mon, Aug 31, 2015 at 08:01PM, jay vyas wrote:
> - I agree Gradle is better than "yet another script".
> 
> - And docker container, even if suboptimal, as thin wrapper to gradle
> container is all you need to deliver the data-generator on the masses so
> they can try it w/ zero startup cost.

That's true. But it didn't sound this way originally, hence I've asked.

> On Mon, Aug 31, 2015 at 7:36 PM, Konstantin Boudnik <co...@apache.org> wrote:
> 
> > Why do would we need yet another script (and potentially an extra readme to
> > explain its command options) when we have the gradle?
> >
> > Cos
> >
> > On Mon, Aug 31, 2015 at 10:51PM, Olaf Flebbe wrote:
> > > +1 to the CLI /shell script interface.
> > >
> > > If I can choose I like to have a apt-get install bigtop-datagenerator ,
> > running for instance
> > >
> > > bigtop-data-generatoroutputDir nStores nCustomers nPurchasingModels
> > simulationLength seed
> > >
> > > I can help out with packaging if needed.
> > >
> > > Why should we use the docker indirection for a plain CLI file ? Of
> > course, We can provide a trivial Dockerfile to create a container supplying
> > a JVM and running the CLI ... But I do not like to depend our services on
> > docker registry more than we do now.
> > >
> > > Olaf
> > >
> > >
> > >
> > > > Am 31.08.2015 um 16:40 schrieb Evans Ye <ev...@apache.org>:
> > > >
> > > > I am very much like the shell script wrapper and docker image idea
> > since
> > > > that way we can integrate it directly with bigtop provisioner which
> > yield a
> > > > perfect ux for the whole things. I think its not too hard to do it
> > both, we
> > > > just need to add a parameter to turn the script into daemon mode. I see
> > > > lots of image doing this way.
> > > >
> > > > docker run bigtop/bigtop-data-gen --scheme weather --size 5GB --output
> > > > data-dir --etc  foo --etc bar --daemon
> > > > 2015年8月31日 下午9:06於 "RJ Nowling" <rn...@gmail.com>寫道:
> > > >
> > > >> The BigPetStore, Bazaar, and weather data generators have
> > single-threaded
> > > >> command-line interfaces.  We could do the same with the smaller
> > generators
> > > >> (names, locations, etc.) if there is interest.
> > > >>
> > > >> On Mon, Aug 31, 2015 at 5:24 AM, Jay Vyas <
> > jayunit100.apache@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> Nate: Good idea to abstract the interface one level higher....
> > > >>>
> > > >>> How about a docker run command ? That is probably the easiest way for
> > > >>> Linux folks to run one off Java apps nowadays.
> > > >>>
> > > >>> docker run bigtop/bigtop-data-gen --scheme weather --size 5GB
> > --output
> > > >>> data-dir --etc  foo --etc bar
> > > >>>
> > > >>> I'm happy to curate such a docker image, I already am doing something
> > > >> like
> > > >>> this in kube for bigtop-transaction-queue, which continuously pumps
> > data
> > > >>> generator outputs into a REST endpoint or file
> > > >>> Queue... So it could be extended to support other generators.
> > > >>>
> > > >>>
> > > >>>> om> <na...@reactor8.com> wrote:
> > > >>>>
> > > >>>> Could picture at some point supporting something like this for
> > non-jvm
> > > >>> folk just looking for test/demo data:
> > > >>>>
> > > >>>> apt-get install bigtop-data-gen
> > > >>>> ~/ $ bigtop-data-gen --scheme weather --size 5GB --output data-dir
> > > >>> --etc  foo --etc bar
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> -----Original Message-----
> > > >>>> From: jay vyas [mailto:jayunit100.apache@gmail.com]
> > > >>>> Sent: Sunday, August 30, 2015 5:11 PM
> > > >>>> To: dev@bigtop.apache.org
> > > >>>> Subject: Re: Proposal for "BigTop Data Generators"
> > > >>>>
> > > >>>> Hola nate.  Well, here are the Use cases I know of that I have used
> > the
> > > >>> data generators for.
> > > >>>>
> > > >>>> Dockerfile:
> > > >>>>
> > > >>>> (1) for testing kubernetes.  For this, I just use transaction-queue
> > > >>> docker file.
> > > >>>> (2) for testing GlusterFS small file workloads, maybe with other
> > > >>> analytics tools...
> > > >>>>
> > > >>>> Maven repo
> > > >>>>
> > > >>>> (3) Java maprduce/ignite/spark applications, which can just add a
> > mvn
> > > >>> repo when compiling.  Java developers never add jars through RPM
> > repos.
> > > >>>>
> > > >>>> RPM/DEB packages:
> > > >>>>
> > > >>>> I could see people using an RPM/DEB data generator, and I'm not
> > against
> > > >>> it.  But I simply don't know of any real world projects which
> > *currently*
> > > >>> need RPM/Deb packages, which is why I haven't bothered to propose it
> > as a
> > > >>> requirement.  Nevertheless linux packages are always a welcome
> > addition
> > > >> if
> > > >>> someone wants to create em !
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>> On Sun, Aug 30, 2015 at 4:34 PM, <na...@reactor8.com> wrote:
> > > >>>>>
> > > >>>>> Would container be in addition to deb/rpm, or instead of?  If
> > latter
> > > >>>>> can we do deb/rpm as base then have container either created from
> > them
> > > >>>>> or directly from artifacts?
> > > >>>>>
> > > >>>>> On test usage side, seems could probably break up tests into
> > > >>>>> base/required and then optional/add-on tests/test-suites.  Think
> > > >>>>> remember seeing mention of certain tests that are failing at times
> > on
> > > >>>>> certain component(s) anyways in the core builds but don’t mean that
> > > >>>>> the build is broken, so would make sense to have some clean up
> > around
> > > >>> those anyways.
> > > >>>>>
> > > >>>>> -----Original Message-----
> > > >>>>> From: RJ Nowling [mailto:rnowling@gmail.com]
> > > >>>>> Sent: Sunday, August 30, 2015 1:11 PM
> > > >>>>> To: dev@bigtop.apache.org
> > > >>>>> Subject: Re: Proposal for "BigTop Data Generators"
> > > >>>>>
> > > >>>>> I agree with the above. :)
> > > >>>>>
> > > >>>>> On Sun, Aug 30, 2015 at 11:19 AM, Jay Vyas
> > > >>>>> <ja...@gmail.com>
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> Hi RJ.
> > > >>>>>>
> > > >>>>>> Maven repositories and docker containers for the transaction queue
> > > >>>>>> are good enough IMO.  That will give people a way to compose them
> > in
> > > >>>>>> different idioms (one for Java folks, another for broader Linux
> > > >>>>>> audience
> > > >>>>> ).
> > > >>>>>>
> > > >>>>>> I think the lib designs are fairly intuitive.  I would say that we
> > > >>>>>> should constrain them all to being written in Java or Groovy to
> > keep
> > > >>>>>> the bigtop theme of "JVM for everything" :).
> > > >>>>>>
> > > >>>>>> Any particular questions you have around technical design can be
> > > >>>>>> followed in a JIRA or else maybe a Readme spec that goes in a  top
> > > >>>>>> level of the data-generators dir...
> > > >>>>>>
> > > >>>>>>> On Aug 30, 2015, at 1:51 AM, RJ Nowling <rn...@gmail.com>
> > wrote:
> > > >>>>>>>
> > > >>>>>>> I'd like to keep this conversation going.
> > > >>>>>>>
> > > >>>>>>> So here are a few discussion points:
> > > >>>>>>>
> > > >>>>>>> 1. How do we want to make the data generators available?  Maven?
> > > >>>>>>> RPMs
> > > >>>>>> and
> > > >>>>>>> Debs?
> > > >>>>>>>
> > > >>>>>>> For now, I'm using a gradle multi-project build to easily build
> > > >>>>>>> and
> > > >>>>>> install
> > > >>>>>>> the BPS data generators and its libraries into a local maven
> > repo.
> > > >>>>>>> This makes development easy.  Eventually, I would like to post
> > > >>>>>>> binaries
> > > >>>>>> through
> > > >>>>>>> Maven for easy integration by users.  RPMs / Debs could be
> > > >>>>>>> interesting since I use a pattern where the data generators are
> > > >>>>>>> libraries (to support application integration / parallelization
> > by
> > > >>>>>>> the host framework) but also provide CLI drivers for local
> > testing.
> > > >>>>>>>
> > > >>>>>>> 2.  The idea of using the data generators as part of the smoke
> > > >>>>>>> tests came up.  Since there is concern about making the data
> > > >>>>>>> generators required, we could offer the blueprints (BigPetStore)
> > > >>>>>>> as optional smoke tests.  Would that be a good compromise?
> > > >>>>>>>
> > > >>>>>>> 3.  How will they be maintained?
> > > >>>>>>>
> > > >>>>>>> I'll certainly add myself to the maintainers list and will be
> > > >>>>>>> taking responsibility.  I'm happy to have others help as well if
> > > >>>>>>> anyone wants to
> > > >>>>>>> -- if not, that's cool, too.
> > > >>>>>>>
> > > >>>>>>> 4. Is anyone interested at all in discussing library APIs and
> > > >> designs?
> > > >>>>>>> What about internal interfaces and such?
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> My plan was to add at least one more data generator (weather
> > > >>>>>>> simulator)
> > > >>>>>> to
> > > >>>>>>> bigtop-data-generators in the short term.  However, given the
> > > >>>>>>> concerns raised by Cos (more discussion needed) and Olaf (don't
> > > >>>>>>> want to force data generators on unsuspecting users ;) ), I would
> > > >>>>>>> like to reach some
> > > >>>>>> consensus
> > > >>>>>>> on what people are concerned about and solutions.
> > > >>>>>>>
> > > >>>>>>> On Thu, Aug 27, 2015 at 12:38 PM, Konstantin Boudnik
> > > >>>>>>> <co...@apache.org>
> > > >>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> Fine by me. I have linked this thread to the JIRA ticket that RJ
> > > >>>>>> created,
> > > >>>>>>>> so
> > > >>>>>>>> we have a way to connect one to another ;)
> > > >>>>>>>>
> > > >>>>>>>>> On Thu, Aug 27, 2015 at 01:02PM, Olaf Flebbe wrote:
> > > >>>>>>>>> Hi,
> > > >>>>>>>>>
> > > >>>>>>>>> I am not confident that moving important design discussions
> > with
> > > >>>>>>>>> impact
> > > >>>>>>>> to
> > > >>>>>>>>> the whole project to jira is a good idea.
> > > >>>>>>>>>
> > > >>>>>>>>> In the current JIRA Traffic storm it is not easy to identify
> > and
> > > >>>>>>>>> follow
> > > >>>>>>>> important tickets.
> > > >>>>>>>>>
> > > >>>>>>>>> Please keep discussions on the list or at least, please state
> > on
> > > >>>>>>>>> this
> > > >>>>>>>> list which Ticket to follow ...
> > > >>>>>>>>>
> > > >>>>>>>>> Olaf
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>> Am 26.08.2015 um 22:56 schrieb Konstantin Boudnik <
> > > >> cos@apache.org
> > > >>>> :
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Wed, Aug 26, 2015 at 10:38PM, Olaf Flebbe wrote:
> > > >>>>>>>>>>> Hi,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Nive to have data generators in Bigtop.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> But please do not include it in bigtop_utils, since this
> > > >>>>>>>>>>> package is mandatory. Not everyone needs a data generator .
> > > >>>>>>>>>>
> > > >>>>>>>>>> Yup. And let's move further design discussion to the JIRA!
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Olaf
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Am 26.08.2015 um 11:25 schrieb Jay Vyas <
> > > >>>>>> jayunit100.apache@gmail.com
> > > >>>>>>>>> :
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Publishing the jar to bigtops maven is probably a good first
> > > >>>>>>>>>>>> step
> > > >>>>>>>> ,Then apps can just include it as needed...?.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I'm not against packaging if someone wants packages for
> > this.
> > > >>>>>>>>>>>> Maybe
> > > >>>>>>>> even include it in bigtop util ?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Let's move to jira,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Aug 25, 2015, at 9:41 PM, Konstantin Boudnik
> > > >>>>>>>>>>>>> <co...@apache.org>
> > > >>>>>>>> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> It is pretty cool indeed!
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> I wonder how it needs to be structured to be:
> > > >>>>>>>>>>>>> - easy to access/use from other components wherever it is
> > > >>>>>>>>>>>>> needed
> > > >>>>>>>>>>>>> - doesn't interfere with the rest of the stack
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> I guess one possible way would be to implement the
> > generator
> > > >>>>>>>>>>>>> as a
> > > >>>>>>>> set of maven
> > > >>>>>>>>>>>>> artifacts, that could be installed/consumed transparently
> > by
> > > >>>>>>>>>>>>> just
> > > >>>>>>>> declaring a
> > > >>>>>>>>>>>>> dependency e.g as proposed via top-level component.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Another way is to have a new package like we do for
> > > >>>>>>>>>>>>> bigtop-utils
> > > >>>>>>>> and such.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Perhaps this discussion should be moved to JIRA or shall we
> > > >>>>>>>> continue on the
> > > >>>>>>>>>>>>> dev@ ??
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Cos
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Sun, Aug 23, 2015 at 11:53AM, RJ Nowling wrote:
> > > >>>>>>>>>>>>>> Hi BigTop,
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> I had a discussion with Jay yesterday, we'd like to
> > propose
> > > >>>>>>>>>>>>>> a new
> > > >>>>>>>> component
> > > >>>>>>>>>>>>>> for BigTop: BigTop Data Generators.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> BigTop Data Generators would consist of a common set of
> > > >>>>>>>>>>>>>> libraries
> > > >>>>>>>> for
> > > >>>>>>>>>>>>>> building data generators and three example data
> > generators:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> * BigPetStore transaction generator (moved from
> > > >>>>>>>>>>>>>> BigPetStore)
> > > >>>>>>>>>>>>>> * BigTop Bazaar -- attendee movement and interactions with
> > > >>>>>>>>>>>>>> booths
> > > >>>>>>>> on a
> > > >>>>>>>>>>>>>> showroom floor, at a conference, or at a mall
> > > >>>>>>>>>>>>>> * BigTop Weatherman -- stochastic weather simulation
> > > >>>>>>>> (temperature, wind
> > > >>>>>>>>>>>>>> speed, wind chill, rainfall, etc.) per zip code.  (From a
> > > >>>>>>>>>>>>>> model
> > > >>>>>>>> trained on
> > > >>>>>>>>>>>>>> NOAA historical weather data)
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> We believe that creating a common set of libraries will
> > > >>>>>>>>>>>>>> have
> > > >>>>>>>> several
> > > >>>>>>>>>>>>>> benefits including:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> * Easier for others to build their own data generators
> > > >>>>>>>>>>>>>> * Make data generators smaller and easier to maintain
> > > >>>>>>>>>>>>>> * Share improvements across the data generators
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> More details on the libraries are below.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> BigPetStore will be continue to focus on building  and
> > > >>>>>>>>>>>>>> maintaining blueprints, powered by the BigTop Data
> > > >> Generators.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Our vision is that we get all of Apache coming to BigTop
> > > >>>>>>>>>>>>>> for tools
> > > >>>>>>>> for
> > > >>>>>>>>>>>>>> building better, more comprehensive blueprints.  We want
> > to
> > > >>>>>>>> support these
> > > >>>>>>>>>>>>>> efforts through data generators and the initial set of
> > > >>>>>>>>>>>>>> blueprint
> > > >>>>>>>> we've been
> > > >>>>>>>>>>>>>> building.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> If the community is generally in support of this, I can
> > > >>>>>>>>>>>>>> create a
> > > >>>>>>>> top-level
> > > >>>>>>>>>>>>>> "bigtop-data-generators" directory and put the data
> > > >>>>>>>>>>>>>> generators and libraries in there.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Thanks!
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> RJ
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> -------
> > > >>>>>>>>>>>>>> Library details:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> So far, I've extracted the following common libraries:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> * Samplers -- provides classes for PDFs and various
> > > >>>>>>>>>>>>>> samplers
> > > >>>>>>>>>>>>>> * Name generator -- data set and samplers for generating
> > > >>>>>>>>>>>>>> names
> > > >>>>>>>>>>>>>> * Location data set -- data set and classes for US zip
> > > >>>>>>>>>>>>>> codes,
> > > >>>>>>>> their
> > > >>>>>>>>>>>>>> GPS coordinates, median house hold incomes, and population
> > > >>>>>>>>>>>>>> sizes
> > > >>>>>>>>>>>>>> * Product generator -- library for enumerating products
> > > >>>>>>>>>>>>>> from a specification file.  Comes with default
> > > >>>>>>>>>>>>>> specifications for
> > > >>>>>>>> BigPetStore
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> I also expect that I'll add libraries for:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> * Particle simulation -- customer movement in a room
> > > >>>>>>>>>>>>>> * Latent factor model generation -- generate latent
> > > >>>>>>>>>>>>>> factors and customer weights to create something like
> > > >>> MovieLens data.
> > > >>>>>>>>>>>>>> Used in
> > > >>>>>>>> Bazaar
> > > >>>>>>>>>>>>>> for booth preferences and potentially in BigPetStore for
> > > >>>>>>>>>>>>>> customer
> > > >>>>>>>> item
> > > >>>>>>>>>>>>>> preferences
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Most of these libraries came out of the BigPetStore data
> > > >>>>>>>>>>>>>> generator
> > > >>>>>>>> but the
> > > >>>>>>>>>>>>>> other generators have been refactored to be based off the
> > > >>>>>>>>>>>>>> standard
> > > >>>>>>>> set of
> > > >>>>>>>>>>>>>> libraries.
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> jay vyas
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> >
> >
> 
> 
> -- 
> jay vyas

Re: Proposal for "BigTop Data Generators"

Posted by jay vyas <ja...@gmail.com>.
- I agree Gradle is better than "yet another script".

- And docker container, even if suboptimal, as thin wrapper to gradle
container is all you need to deliver the data-generator on the masses so
they can try it w/ zero startup cost.



On Mon, Aug 31, 2015 at 7:36 PM, Konstantin Boudnik <co...@apache.org> wrote:

> Why do would we need yet another script (and potentially an extra readme to
> explain its command options) when we have the gradle?
>
> Cos
>
> On Mon, Aug 31, 2015 at 10:51PM, Olaf Flebbe wrote:
> > +1 to the CLI /shell script interface.
> >
> > If I can choose I like to have a apt-get install bigtop-datagenerator ,
> running for instance
> >
> > bigtop-data-generatoroutputDir nStores nCustomers nPurchasingModels
> simulationLength seed
> >
> > I can help out with packaging if needed.
> >
> > Why should we use the docker indirection for a plain CLI file ? Of
> course, We can provide a trivial Dockerfile to create a container supplying
> a JVM and running the CLI ... But I do not like to depend our services on
> docker registry more than we do now.
> >
> > Olaf
> >
> >
> >
> > > Am 31.08.2015 um 16:40 schrieb Evans Ye <ev...@apache.org>:
> > >
> > > I am very much like the shell script wrapper and docker image idea
> since
> > > that way we can integrate it directly with bigtop provisioner which
> yield a
> > > perfect ux for the whole things. I think its not too hard to do it
> both, we
> > > just need to add a parameter to turn the script into daemon mode. I see
> > > lots of image doing this way.
> > >
> > > docker run bigtop/bigtop-data-gen --scheme weather --size 5GB --output
> > > data-dir --etc  foo --etc bar --daemon
> > > 2015年8月31日 下午9:06於 "RJ Nowling" <rn...@gmail.com>寫道:
> > >
> > >> The BigPetStore, Bazaar, and weather data generators have
> single-threaded
> > >> command-line interfaces.  We could do the same with the smaller
> generators
> > >> (names, locations, etc.) if there is interest.
> > >>
> > >> On Mon, Aug 31, 2015 at 5:24 AM, Jay Vyas <
> jayunit100.apache@gmail.com>
> > >> wrote:
> > >>
> > >>> Nate: Good idea to abstract the interface one level higher....
> > >>>
> > >>> How about a docker run command ? That is probably the easiest way for
> > >>> Linux folks to run one off Java apps nowadays.
> > >>>
> > >>> docker run bigtop/bigtop-data-gen --scheme weather --size 5GB
> --output
> > >>> data-dir --etc  foo --etc bar
> > >>>
> > >>> I'm happy to curate such a docker image, I already am doing something
> > >> like
> > >>> this in kube for bigtop-transaction-queue, which continuously pumps
> data
> > >>> generator outputs into a REST endpoint or file
> > >>> Queue... So it could be extended to support other generators.
> > >>>
> > >>>
> > >>>> om> <na...@reactor8.com> wrote:
> > >>>>
> > >>>> Could picture at some point supporting something like this for
> non-jvm
> > >>> folk just looking for test/demo data:
> > >>>>
> > >>>> apt-get install bigtop-data-gen
> > >>>> ~/ $ bigtop-data-gen --scheme weather --size 5GB --output data-dir
> > >>> --etc  foo --etc bar
> > >>>>
> > >>>>
> > >>>>
> > >>>> -----Original Message-----
> > >>>> From: jay vyas [mailto:jayunit100.apache@gmail.com]
> > >>>> Sent: Sunday, August 30, 2015 5:11 PM
> > >>>> To: dev@bigtop.apache.org
> > >>>> Subject: Re: Proposal for "BigTop Data Generators"
> > >>>>
> > >>>> Hola nate.  Well, here are the Use cases I know of that I have used
> the
> > >>> data generators for.
> > >>>>
> > >>>> Dockerfile:
> > >>>>
> > >>>> (1) for testing kubernetes.  For this, I just use transaction-queue
> > >>> docker file.
> > >>>> (2) for testing GlusterFS small file workloads, maybe with other
> > >>> analytics tools...
> > >>>>
> > >>>> Maven repo
> > >>>>
> > >>>> (3) Java maprduce/ignite/spark applications, which can just add a
> mvn
> > >>> repo when compiling.  Java developers never add jars through RPM
> repos.
> > >>>>
> > >>>> RPM/DEB packages:
> > >>>>
> > >>>> I could see people using an RPM/DEB data generator, and I'm not
> against
> > >>> it.  But I simply don't know of any real world projects which
> *currently*
> > >>> need RPM/Deb packages, which is why I haven't bothered to propose it
> as a
> > >>> requirement.  Nevertheless linux packages are always a welcome
> addition
> > >> if
> > >>> someone wants to create em !
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>> On Sun, Aug 30, 2015 at 4:34 PM, <na...@reactor8.com> wrote:
> > >>>>>
> > >>>>> Would container be in addition to deb/rpm, or instead of?  If
> latter
> > >>>>> can we do deb/rpm as base then have container either created from
> them
> > >>>>> or directly from artifacts?
> > >>>>>
> > >>>>> On test usage side, seems could probably break up tests into
> > >>>>> base/required and then optional/add-on tests/test-suites.  Think
> > >>>>> remember seeing mention of certain tests that are failing at times
> on
> > >>>>> certain component(s) anyways in the core builds but don’t mean that
> > >>>>> the build is broken, so would make sense to have some clean up
> around
> > >>> those anyways.
> > >>>>>
> > >>>>> -----Original Message-----
> > >>>>> From: RJ Nowling [mailto:rnowling@gmail.com]
> > >>>>> Sent: Sunday, August 30, 2015 1:11 PM
> > >>>>> To: dev@bigtop.apache.org
> > >>>>> Subject: Re: Proposal for "BigTop Data Generators"
> > >>>>>
> > >>>>> I agree with the above. :)
> > >>>>>
> > >>>>> On Sun, Aug 30, 2015 at 11:19 AM, Jay Vyas
> > >>>>> <ja...@gmail.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Hi RJ.
> > >>>>>>
> > >>>>>> Maven repositories and docker containers for the transaction queue
> > >>>>>> are good enough IMO.  That will give people a way to compose them
> in
> > >>>>>> different idioms (one for Java folks, another for broader Linux
> > >>>>>> audience
> > >>>>> ).
> > >>>>>>
> > >>>>>> I think the lib designs are fairly intuitive.  I would say that we
> > >>>>>> should constrain them all to being written in Java or Groovy to
> keep
> > >>>>>> the bigtop theme of "JVM for everything" :).
> > >>>>>>
> > >>>>>> Any particular questions you have around technical design can be
> > >>>>>> followed in a JIRA or else maybe a Readme spec that goes in a  top
> > >>>>>> level of the data-generators dir...
> > >>>>>>
> > >>>>>>> On Aug 30, 2015, at 1:51 AM, RJ Nowling <rn...@gmail.com>
> wrote:
> > >>>>>>>
> > >>>>>>> I'd like to keep this conversation going.
> > >>>>>>>
> > >>>>>>> So here are a few discussion points:
> > >>>>>>>
> > >>>>>>> 1. How do we want to make the data generators available?  Maven?
> > >>>>>>> RPMs
> > >>>>>> and
> > >>>>>>> Debs?
> > >>>>>>>
> > >>>>>>> For now, I'm using a gradle multi-project build to easily build
> > >>>>>>> and
> > >>>>>> install
> > >>>>>>> the BPS data generators and its libraries into a local maven
> repo.
> > >>>>>>> This makes development easy.  Eventually, I would like to post
> > >>>>>>> binaries
> > >>>>>> through
> > >>>>>>> Maven for easy integration by users.  RPMs / Debs could be
> > >>>>>>> interesting since I use a pattern where the data generators are
> > >>>>>>> libraries (to support application integration / parallelization
> by
> > >>>>>>> the host framework) but also provide CLI drivers for local
> testing.
> > >>>>>>>
> > >>>>>>> 2.  The idea of using the data generators as part of the smoke
> > >>>>>>> tests came up.  Since there is concern about making the data
> > >>>>>>> generators required, we could offer the blueprints (BigPetStore)
> > >>>>>>> as optional smoke tests.  Would that be a good compromise?
> > >>>>>>>
> > >>>>>>> 3.  How will they be maintained?
> > >>>>>>>
> > >>>>>>> I'll certainly add myself to the maintainers list and will be
> > >>>>>>> taking responsibility.  I'm happy to have others help as well if
> > >>>>>>> anyone wants to
> > >>>>>>> -- if not, that's cool, too.
> > >>>>>>>
> > >>>>>>> 4. Is anyone interested at all in discussing library APIs and
> > >> designs?
> > >>>>>>> What about internal interfaces and such?
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> My plan was to add at least one more data generator (weather
> > >>>>>>> simulator)
> > >>>>>> to
> > >>>>>>> bigtop-data-generators in the short term.  However, given the
> > >>>>>>> concerns raised by Cos (more discussion needed) and Olaf (don't
> > >>>>>>> want to force data generators on unsuspecting users ;) ), I would
> > >>>>>>> like to reach some
> > >>>>>> consensus
> > >>>>>>> on what people are concerned about and solutions.
> > >>>>>>>
> > >>>>>>> On Thu, Aug 27, 2015 at 12:38 PM, Konstantin Boudnik
> > >>>>>>> <co...@apache.org>
> > >>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Fine by me. I have linked this thread to the JIRA ticket that RJ
> > >>>>>> created,
> > >>>>>>>> so
> > >>>>>>>> we have a way to connect one to another ;)
> > >>>>>>>>
> > >>>>>>>>> On Thu, Aug 27, 2015 at 01:02PM, Olaf Flebbe wrote:
> > >>>>>>>>> Hi,
> > >>>>>>>>>
> > >>>>>>>>> I am not confident that moving important design discussions
> with
> > >>>>>>>>> impact
> > >>>>>>>> to
> > >>>>>>>>> the whole project to jira is a good idea.
> > >>>>>>>>>
> > >>>>>>>>> In the current JIRA Traffic storm it is not easy to identify
> and
> > >>>>>>>>> follow
> > >>>>>>>> important tickets.
> > >>>>>>>>>
> > >>>>>>>>> Please keep discussions on the list or at least, please state
> on
> > >>>>>>>>> this
> > >>>>>>>> list which Ticket to follow ...
> > >>>>>>>>>
> > >>>>>>>>> Olaf
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>> Am 26.08.2015 um 22:56 schrieb Konstantin Boudnik <
> > >> cos@apache.org
> > >>>> :
> > >>>>>>>>>>
> > >>>>>>>>>> On Wed, Aug 26, 2015 at 10:38PM, Olaf Flebbe wrote:
> > >>>>>>>>>>> Hi,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Nive to have data generators in Bigtop.
> > >>>>>>>>>>>
> > >>>>>>>>>>> But please do not include it in bigtop_utils, since this
> > >>>>>>>>>>> package is mandatory. Not everyone needs a data generator .
> > >>>>>>>>>>
> > >>>>>>>>>> Yup. And let's move further design discussion to the JIRA!
> > >>>>>>>>>>
> > >>>>>>>>>>> Olaf
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Am 26.08.2015 um 11:25 schrieb Jay Vyas <
> > >>>>>> jayunit100.apache@gmail.com
> > >>>>>>>>> :
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Publishing the jar to bigtops maven is probably a good first
> > >>>>>>>>>>>> step
> > >>>>>>>> ,Then apps can just include it as needed...?.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I'm not against packaging if someone wants packages for
> this.
> > >>>>>>>>>>>> Maybe
> > >>>>>>>> even include it in bigtop util ?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Let's move to jira,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> On Aug 25, 2015, at 9:41 PM, Konstantin Boudnik
> > >>>>>>>>>>>>> <co...@apache.org>
> > >>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> It is pretty cool indeed!
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I wonder how it needs to be structured to be:
> > >>>>>>>>>>>>> - easy to access/use from other components wherever it is
> > >>>>>>>>>>>>> needed
> > >>>>>>>>>>>>> - doesn't interfere with the rest of the stack
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I guess one possible way would be to implement the
> generator
> > >>>>>>>>>>>>> as a
> > >>>>>>>> set of maven
> > >>>>>>>>>>>>> artifacts, that could be installed/consumed transparently
> by
> > >>>>>>>>>>>>> just
> > >>>>>>>> declaring a
> > >>>>>>>>>>>>> dependency e.g as proposed via top-level component.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Another way is to have a new package like we do for
> > >>>>>>>>>>>>> bigtop-utils
> > >>>>>>>> and such.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Perhaps this discussion should be moved to JIRA or shall we
> > >>>>>>>> continue on the
> > >>>>>>>>>>>>> dev@ ??
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Cos
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Sun, Aug 23, 2015 at 11:53AM, RJ Nowling wrote:
> > >>>>>>>>>>>>>> Hi BigTop,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I had a discussion with Jay yesterday, we'd like to
> propose
> > >>>>>>>>>>>>>> a new
> > >>>>>>>> component
> > >>>>>>>>>>>>>> for BigTop: BigTop Data Generators.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> BigTop Data Generators would consist of a common set of
> > >>>>>>>>>>>>>> libraries
> > >>>>>>>> for
> > >>>>>>>>>>>>>> building data generators and three example data
> generators:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> * BigPetStore transaction generator (moved from
> > >>>>>>>>>>>>>> BigPetStore)
> > >>>>>>>>>>>>>> * BigTop Bazaar -- attendee movement and interactions with
> > >>>>>>>>>>>>>> booths
> > >>>>>>>> on a
> > >>>>>>>>>>>>>> showroom floor, at a conference, or at a mall
> > >>>>>>>>>>>>>> * BigTop Weatherman -- stochastic weather simulation
> > >>>>>>>> (temperature, wind
> > >>>>>>>>>>>>>> speed, wind chill, rainfall, etc.) per zip code.  (From a
> > >>>>>>>>>>>>>> model
> > >>>>>>>> trained on
> > >>>>>>>>>>>>>> NOAA historical weather data)
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> We believe that creating a common set of libraries will
> > >>>>>>>>>>>>>> have
> > >>>>>>>> several
> > >>>>>>>>>>>>>> benefits including:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> * Easier for others to build their own data generators
> > >>>>>>>>>>>>>> * Make data generators smaller and easier to maintain
> > >>>>>>>>>>>>>> * Share improvements across the data generators
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> More details on the libraries are below.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> BigPetStore will be continue to focus on building  and
> > >>>>>>>>>>>>>> maintaining blueprints, powered by the BigTop Data
> > >> Generators.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Our vision is that we get all of Apache coming to BigTop
> > >>>>>>>>>>>>>> for tools
> > >>>>>>>> for
> > >>>>>>>>>>>>>> building better, more comprehensive blueprints.  We want
> to
> > >>>>>>>> support these
> > >>>>>>>>>>>>>> efforts through data generators and the initial set of
> > >>>>>>>>>>>>>> blueprint
> > >>>>>>>> we've been
> > >>>>>>>>>>>>>> building.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> If the community is generally in support of this, I can
> > >>>>>>>>>>>>>> create a
> > >>>>>>>> top-level
> > >>>>>>>>>>>>>> "bigtop-data-generators" directory and put the data
> > >>>>>>>>>>>>>> generators and libraries in there.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks!
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> RJ
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> -------
> > >>>>>>>>>>>>>> Library details:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> So far, I've extracted the following common libraries:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> * Samplers -- provides classes for PDFs and various
> > >>>>>>>>>>>>>> samplers
> > >>>>>>>>>>>>>> * Name generator -- data set and samplers for generating
> > >>>>>>>>>>>>>> names
> > >>>>>>>>>>>>>> * Location data set -- data set and classes for US zip
> > >>>>>>>>>>>>>> codes,
> > >>>>>>>> their
> > >>>>>>>>>>>>>> GPS coordinates, median house hold incomes, and population
> > >>>>>>>>>>>>>> sizes
> > >>>>>>>>>>>>>> * Product generator -- library for enumerating products
> > >>>>>>>>>>>>>> from a specification file.  Comes with default
> > >>>>>>>>>>>>>> specifications for
> > >>>>>>>> BigPetStore
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I also expect that I'll add libraries for:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> * Particle simulation -- customer movement in a room
> > >>>>>>>>>>>>>> * Latent factor model generation -- generate latent
> > >>>>>>>>>>>>>> factors and customer weights to create something like
> > >>> MovieLens data.
> > >>>>>>>>>>>>>> Used in
> > >>>>>>>> Bazaar
> > >>>>>>>>>>>>>> for booth preferences and potentially in BigPetStore for
> > >>>>>>>>>>>>>> customer
> > >>>>>>>> item
> > >>>>>>>>>>>>>> preferences
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Most of these libraries came out of the BigPetStore data
> > >>>>>>>>>>>>>> generator
> > >>>>>>>> but the
> > >>>>>>>>>>>>>> other generators have been refactored to be based off the
> > >>>>>>>>>>>>>> standard
> > >>>>>>>> set of
> > >>>>>>>>>>>>>> libraries.
> > >>>>
> > >>>>
> > >>>> --
> > >>>> jay vyas
> > >>>>
> > >>>
> > >>
> >
>
>
>


-- 
jay vyas

Re: Proposal for "BigTop Data Generators"

Posted by Konstantin Boudnik <co...@apache.org>.
Why do would we need yet another script (and potentially an extra readme to
explain its command options) when we have the gradle?

Cos

On Mon, Aug 31, 2015 at 10:51PM, Olaf Flebbe wrote:
> +1 to the CLI /shell script interface.
> 
> If I can choose I like to have a apt-get install bigtop-datagenerator , running for instance
> 
> bigtop-data-generatoroutputDir nStores nCustomers nPurchasingModels simulationLength seed
> 
> I can help out with packaging if needed.
> 
> Why should we use the docker indirection for a plain CLI file ? Of course, We can provide a trivial Dockerfile to create a container supplying a JVM and running the CLI ... But I do not like to depend our services on docker registry more than we do now.
> 
> Olaf
> 
> 
> 
> > Am 31.08.2015 um 16:40 schrieb Evans Ye <ev...@apache.org>:
> > 
> > I am very much like the shell script wrapper and docker image idea since
> > that way we can integrate it directly with bigtop provisioner which yield a
> > perfect ux for the whole things. I think its not too hard to do it both, we
> > just need to add a parameter to turn the script into daemon mode. I see
> > lots of image doing this way.
> > 
> > docker run bigtop/bigtop-data-gen --scheme weather --size 5GB --output
> > data-dir --etc  foo --etc bar --daemon
> > 2015年8月31日 下午9:06於 "RJ Nowling" <rn...@gmail.com>寫道:
> > 
> >> The BigPetStore, Bazaar, and weather data generators have single-threaded
> >> command-line interfaces.  We could do the same with the smaller generators
> >> (names, locations, etc.) if there is interest.
> >> 
> >> On Mon, Aug 31, 2015 at 5:24 AM, Jay Vyas <ja...@gmail.com>
> >> wrote:
> >> 
> >>> Nate: Good idea to abstract the interface one level higher....
> >>> 
> >>> How about a docker run command ? That is probably the easiest way for
> >>> Linux folks to run one off Java apps nowadays.
> >>> 
> >>> docker run bigtop/bigtop-data-gen --scheme weather --size 5GB --output
> >>> data-dir --etc  foo --etc bar
> >>> 
> >>> I'm happy to curate such a docker image, I already am doing something
> >> like
> >>> this in kube for bigtop-transaction-queue, which continuously pumps data
> >>> generator outputs into a REST endpoint or file
> >>> Queue... So it could be extended to support other generators.
> >>> 
> >>> 
> >>>> om> <na...@reactor8.com> wrote:
> >>>> 
> >>>> Could picture at some point supporting something like this for non-jvm
> >>> folk just looking for test/demo data:
> >>>> 
> >>>> apt-get install bigtop-data-gen
> >>>> ~/ $ bigtop-data-gen --scheme weather --size 5GB --output data-dir
> >>> --etc  foo --etc bar
> >>>> 
> >>>> 
> >>>> 
> >>>> -----Original Message-----
> >>>> From: jay vyas [mailto:jayunit100.apache@gmail.com]
> >>>> Sent: Sunday, August 30, 2015 5:11 PM
> >>>> To: dev@bigtop.apache.org
> >>>> Subject: Re: Proposal for "BigTop Data Generators"
> >>>> 
> >>>> Hola nate.  Well, here are the Use cases I know of that I have used the
> >>> data generators for.
> >>>> 
> >>>> Dockerfile:
> >>>> 
> >>>> (1) for testing kubernetes.  For this, I just use transaction-queue
> >>> docker file.
> >>>> (2) for testing GlusterFS small file workloads, maybe with other
> >>> analytics tools...
> >>>> 
> >>>> Maven repo
> >>>> 
> >>>> (3) Java maprduce/ignite/spark applications, which can just add a mvn
> >>> repo when compiling.  Java developers never add jars through RPM repos.
> >>>> 
> >>>> RPM/DEB packages:
> >>>> 
> >>>> I could see people using an RPM/DEB data generator, and I'm not against
> >>> it.  But I simply don't know of any real world projects which *currently*
> >>> need RPM/Deb packages, which is why I haven't bothered to propose it as a
> >>> requirement.  Nevertheless linux packages are always a welcome addition
> >> if
> >>> someone wants to create em !
> >>>> 
> >>>> 
> >>>> 
> >>>> 
> >>>>> On Sun, Aug 30, 2015 at 4:34 PM, <na...@reactor8.com> wrote:
> >>>>> 
> >>>>> Would container be in addition to deb/rpm, or instead of?  If latter
> >>>>> can we do deb/rpm as base then have container either created from them
> >>>>> or directly from artifacts?
> >>>>> 
> >>>>> On test usage side, seems could probably break up tests into
> >>>>> base/required and then optional/add-on tests/test-suites.  Think
> >>>>> remember seeing mention of certain tests that are failing at times on
> >>>>> certain component(s) anyways in the core builds but don’t mean that
> >>>>> the build is broken, so would make sense to have some clean up around
> >>> those anyways.
> >>>>> 
> >>>>> -----Original Message-----
> >>>>> From: RJ Nowling [mailto:rnowling@gmail.com]
> >>>>> Sent: Sunday, August 30, 2015 1:11 PM
> >>>>> To: dev@bigtop.apache.org
> >>>>> Subject: Re: Proposal for "BigTop Data Generators"
> >>>>> 
> >>>>> I agree with the above. :)
> >>>>> 
> >>>>> On Sun, Aug 30, 2015 at 11:19 AM, Jay Vyas
> >>>>> <ja...@gmail.com>
> >>>>> wrote:
> >>>>> 
> >>>>>> Hi RJ.
> >>>>>> 
> >>>>>> Maven repositories and docker containers for the transaction queue
> >>>>>> are good enough IMO.  That will give people a way to compose them in
> >>>>>> different idioms (one for Java folks, another for broader Linux
> >>>>>> audience
> >>>>> ).
> >>>>>> 
> >>>>>> I think the lib designs are fairly intuitive.  I would say that we
> >>>>>> should constrain them all to being written in Java or Groovy to keep
> >>>>>> the bigtop theme of "JVM for everything" :).
> >>>>>> 
> >>>>>> Any particular questions you have around technical design can be
> >>>>>> followed in a JIRA or else maybe a Readme spec that goes in a  top
> >>>>>> level of the data-generators dir...
> >>>>>> 
> >>>>>>> On Aug 30, 2015, at 1:51 AM, RJ Nowling <rn...@gmail.com> wrote:
> >>>>>>> 
> >>>>>>> I'd like to keep this conversation going.
> >>>>>>> 
> >>>>>>> So here are a few discussion points:
> >>>>>>> 
> >>>>>>> 1. How do we want to make the data generators available?  Maven?
> >>>>>>> RPMs
> >>>>>> and
> >>>>>>> Debs?
> >>>>>>> 
> >>>>>>> For now, I'm using a gradle multi-project build to easily build
> >>>>>>> and
> >>>>>> install
> >>>>>>> the BPS data generators and its libraries into a local maven repo.
> >>>>>>> This makes development easy.  Eventually, I would like to post
> >>>>>>> binaries
> >>>>>> through
> >>>>>>> Maven for easy integration by users.  RPMs / Debs could be
> >>>>>>> interesting since I use a pattern where the data generators are
> >>>>>>> libraries (to support application integration / parallelization by
> >>>>>>> the host framework) but also provide CLI drivers for local testing.
> >>>>>>> 
> >>>>>>> 2.  The idea of using the data generators as part of the smoke
> >>>>>>> tests came up.  Since there is concern about making the data
> >>>>>>> generators required, we could offer the blueprints (BigPetStore)
> >>>>>>> as optional smoke tests.  Would that be a good compromise?
> >>>>>>> 
> >>>>>>> 3.  How will they be maintained?
> >>>>>>> 
> >>>>>>> I'll certainly add myself to the maintainers list and will be
> >>>>>>> taking responsibility.  I'm happy to have others help as well if
> >>>>>>> anyone wants to
> >>>>>>> -- if not, that's cool, too.
> >>>>>>> 
> >>>>>>> 4. Is anyone interested at all in discussing library APIs and
> >> designs?
> >>>>>>> What about internal interfaces and such?
> >>>>>>> 
> >>>>>>> 
> >>>>>>> My plan was to add at least one more data generator (weather
> >>>>>>> simulator)
> >>>>>> to
> >>>>>>> bigtop-data-generators in the short term.  However, given the
> >>>>>>> concerns raised by Cos (more discussion needed) and Olaf (don't
> >>>>>>> want to force data generators on unsuspecting users ;) ), I would
> >>>>>>> like to reach some
> >>>>>> consensus
> >>>>>>> on what people are concerned about and solutions.
> >>>>>>> 
> >>>>>>> On Thu, Aug 27, 2015 at 12:38 PM, Konstantin Boudnik
> >>>>>>> <co...@apache.org>
> >>>>>> wrote:
> >>>>>>> 
> >>>>>>>> Fine by me. I have linked this thread to the JIRA ticket that RJ
> >>>>>> created,
> >>>>>>>> so
> >>>>>>>> we have a way to connect one to another ;)
> >>>>>>>> 
> >>>>>>>>> On Thu, Aug 27, 2015 at 01:02PM, Olaf Flebbe wrote:
> >>>>>>>>> Hi,
> >>>>>>>>> 
> >>>>>>>>> I am not confident that moving important design discussions with
> >>>>>>>>> impact
> >>>>>>>> to
> >>>>>>>>> the whole project to jira is a good idea.
> >>>>>>>>> 
> >>>>>>>>> In the current JIRA Traffic storm it is not easy to identify and
> >>>>>>>>> follow
> >>>>>>>> important tickets.
> >>>>>>>>> 
> >>>>>>>>> Please keep discussions on the list or at least, please state on
> >>>>>>>>> this
> >>>>>>>> list which Ticket to follow ...
> >>>>>>>>> 
> >>>>>>>>> Olaf
> >>>>>>>>> 
> >>>>>>>>> 
> >>>>>>>>> 
> >>>>>>>>>> Am 26.08.2015 um 22:56 schrieb Konstantin Boudnik <
> >> cos@apache.org
> >>>> :
> >>>>>>>>>> 
> >>>>>>>>>> On Wed, Aug 26, 2015 at 10:38PM, Olaf Flebbe wrote:
> >>>>>>>>>>> Hi,
> >>>>>>>>>>> 
> >>>>>>>>>>> Nive to have data generators in Bigtop.
> >>>>>>>>>>> 
> >>>>>>>>>>> But please do not include it in bigtop_utils, since this
> >>>>>>>>>>> package is mandatory. Not everyone needs a data generator .
> >>>>>>>>>> 
> >>>>>>>>>> Yup. And let's move further design discussion to the JIRA!
> >>>>>>>>>> 
> >>>>>>>>>>> Olaf
> >>>>>>>>>>> 
> >>>>>>>>>>> 
> >>>>>>>>>>>> Am 26.08.2015 um 11:25 schrieb Jay Vyas <
> >>>>>> jayunit100.apache@gmail.com
> >>>>>>>>> :
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Publishing the jar to bigtops maven is probably a good first
> >>>>>>>>>>>> step
> >>>>>>>> ,Then apps can just include it as needed...?.
> >>>>>>>>>>>> 
> >>>>>>>>>>>> I'm not against packaging if someone wants packages for this.
> >>>>>>>>>>>> Maybe
> >>>>>>>> even include it in bigtop util ?
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Let's move to jira,
> >>>>>>>>>>>> 
> >>>>>>>>>>>>> On Aug 25, 2015, at 9:41 PM, Konstantin Boudnik
> >>>>>>>>>>>>> <co...@apache.org>
> >>>>>>>> wrote:
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> It is pretty cool indeed!
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> I wonder how it needs to be structured to be:
> >>>>>>>>>>>>> - easy to access/use from other components wherever it is
> >>>>>>>>>>>>> needed
> >>>>>>>>>>>>> - doesn't interfere with the rest of the stack
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> I guess one possible way would be to implement the generator
> >>>>>>>>>>>>> as a
> >>>>>>>> set of maven
> >>>>>>>>>>>>> artifacts, that could be installed/consumed transparently by
> >>>>>>>>>>>>> just
> >>>>>>>> declaring a
> >>>>>>>>>>>>> dependency e.g as proposed via top-level component.
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> Another way is to have a new package like we do for
> >>>>>>>>>>>>> bigtop-utils
> >>>>>>>> and such.
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> Perhaps this discussion should be moved to JIRA or shall we
> >>>>>>>> continue on the
> >>>>>>>>>>>>> dev@ ??
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> Cos
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>>> On Sun, Aug 23, 2015 at 11:53AM, RJ Nowling wrote:
> >>>>>>>>>>>>>> Hi BigTop,
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> I had a discussion with Jay yesterday, we'd like to propose
> >>>>>>>>>>>>>> a new
> >>>>>>>> component
> >>>>>>>>>>>>>> for BigTop: BigTop Data Generators.
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> BigTop Data Generators would consist of a common set of
> >>>>>>>>>>>>>> libraries
> >>>>>>>> for
> >>>>>>>>>>>>>> building data generators and three example data generators:
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> * BigPetStore transaction generator (moved from
> >>>>>>>>>>>>>> BigPetStore)
> >>>>>>>>>>>>>> * BigTop Bazaar -- attendee movement and interactions with
> >>>>>>>>>>>>>> booths
> >>>>>>>> on a
> >>>>>>>>>>>>>> showroom floor, at a conference, or at a mall
> >>>>>>>>>>>>>> * BigTop Weatherman -- stochastic weather simulation
> >>>>>>>> (temperature, wind
> >>>>>>>>>>>>>> speed, wind chill, rainfall, etc.) per zip code.  (From a
> >>>>>>>>>>>>>> model
> >>>>>>>> trained on
> >>>>>>>>>>>>>> NOAA historical weather data)
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> We believe that creating a common set of libraries will
> >>>>>>>>>>>>>> have
> >>>>>>>> several
> >>>>>>>>>>>>>> benefits including:
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> * Easier for others to build their own data generators
> >>>>>>>>>>>>>> * Make data generators smaller and easier to maintain
> >>>>>>>>>>>>>> * Share improvements across the data generators
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> More details on the libraries are below.
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> BigPetStore will be continue to focus on building  and
> >>>>>>>>>>>>>> maintaining blueprints, powered by the BigTop Data
> >> Generators.
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> Our vision is that we get all of Apache coming to BigTop
> >>>>>>>>>>>>>> for tools
> >>>>>>>> for
> >>>>>>>>>>>>>> building better, more comprehensive blueprints.  We want to
> >>>>>>>> support these
> >>>>>>>>>>>>>> efforts through data generators and the initial set of
> >>>>>>>>>>>>>> blueprint
> >>>>>>>> we've been
> >>>>>>>>>>>>>> building.
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> If the community is generally in support of this, I can
> >>>>>>>>>>>>>> create a
> >>>>>>>> top-level
> >>>>>>>>>>>>>> "bigtop-data-generators" directory and put the data
> >>>>>>>>>>>>>> generators and libraries in there.
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> RJ
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> -------
> >>>>>>>>>>>>>> Library details:
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> So far, I've extracted the following common libraries:
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> * Samplers -- provides classes for PDFs and various
> >>>>>>>>>>>>>> samplers
> >>>>>>>>>>>>>> * Name generator -- data set and samplers for generating
> >>>>>>>>>>>>>> names
> >>>>>>>>>>>>>> * Location data set -- data set and classes for US zip
> >>>>>>>>>>>>>> codes,
> >>>>>>>> their
> >>>>>>>>>>>>>> GPS coordinates, median house hold incomes, and population
> >>>>>>>>>>>>>> sizes
> >>>>>>>>>>>>>> * Product generator -- library for enumerating products
> >>>>>>>>>>>>>> from a specification file.  Comes with default
> >>>>>>>>>>>>>> specifications for
> >>>>>>>> BigPetStore
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> I also expect that I'll add libraries for:
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> * Particle simulation -- customer movement in a room
> >>>>>>>>>>>>>> * Latent factor model generation -- generate latent
> >>>>>>>>>>>>>> factors and customer weights to create something like
> >>> MovieLens data.
> >>>>>>>>>>>>>> Used in
> >>>>>>>> Bazaar
> >>>>>>>>>>>>>> for booth preferences and potentially in BigPetStore for
> >>>>>>>>>>>>>> customer
> >>>>>>>> item
> >>>>>>>>>>>>>> preferences
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> Most of these libraries came out of the BigPetStore data
> >>>>>>>>>>>>>> generator
> >>>>>>>> but the
> >>>>>>>>>>>>>> other generators have been refactored to be based off the
> >>>>>>>>>>>>>> standard
> >>>>>>>> set of
> >>>>>>>>>>>>>> libraries.
> >>>> 
> >>>> 
> >>>> --
> >>>> jay vyas
> >>>> 
> >>> 
> >> 
> 



Re: Proposal for "BigTop Data Generators"

Posted by RJ Nowling <rn...@gmail.com>.
We could create a single docker image with a wrapper that delegates to each
data generator or create a separate docker image for each generator.  Each
data generator has its own separate of parameters that need to be specified
though (e.g., numbers of customers and stores).

However, we could use Jay's transaction queue idea for all of the data
generators.

Evans, can you elaborate on the use case?  If you want to write to HDFS, we
would need a driver that pulls in some Hadoop libraries.  Generally, we've
been solving this by writing MapReduce or Spark jobs that call the
libraries to parallelize and generate the data.  E.g., data  generators are
just libraries that do computation while the driver handles parallelization
and output.

On Mon, Aug 31, 2015 at 9:44 AM, Jay Vyas <ja...@gmail.com>
wrote:

> Rj can we abstract the command line so that we have "one cli to rule them
> all" into an interface?
>
>
> > On Aug 31, 2015, at 10:40 AM, Evans Ye <ev...@apache.org> wrote:
> >
> > I am very much like the shell script wrapper and docker image idea since
> > that way we can integrate it directly with bigtop provisioner which
> yield a
> > perfect ux for the whole things. I think its not too hard to do it both,
> we
> > just need to add a parameter to turn the script into daemon mode. I see
> > lots of image doing this way.
> >
> > docker run bigtop/bigtop-data-gen --scheme weather --size 5GB --output
> > data-dir --etc  foo --etc bar --daemon
> > 2015年8月31日 下午9:06於 "RJ Nowling" <rn...@gmail.com>寫道:
> >
> >> The BigPetStore, Bazaar, and weather data generators have
> single-threaded
> >> command-line interfaces.  We could do the same with the smaller
> generators
> >> (names, locations, etc.) if there is interest.
> >>
> >> On Mon, Aug 31, 2015 at 5:24 AM, Jay Vyas <ja...@gmail.com>
> >> wrote:
> >>
> >>> Nate: Good idea to abstract the interface one level higher....
> >>>
> >>> How about a docker run command ? That is probably the easiest way for
> >>> Linux folks to run one off Java apps nowadays.
> >>>
> >>> docker run bigtop/bigtop-data-gen --scheme weather --size 5GB --output
> >>> data-dir --etc  foo --etc bar
> >>>
> >>> I'm happy to curate such a docker image, I already am doing something
> >> like
> >>> this in kube for bigtop-transaction-queue, which continuously pumps
> data
> >>> generator outputs into a REST endpoint or file
> >>> Queue... So it could be extended to support other generators.
> >>>
> >>>
> >>>> om> <na...@reactor8.com> wrote:
> >>>>
> >>>> Could picture at some point supporting something like this for non-jvm
> >>> folk just looking for test/demo data:
> >>>>
> >>>> apt-get install bigtop-data-gen
> >>>> ~/ $ bigtop-data-gen --scheme weather --size 5GB --output data-dir
> >>> --etc  foo --etc bar
> >>>>
> >>>>
> >>>>
> >>>> -----Original Message-----
> >>>> From: jay vyas [mailto:jayunit100.apache@gmail.com]
> >>>> Sent: Sunday, August 30, 2015 5:11 PM
> >>>> To: dev@bigtop.apache.org
> >>>> Subject: Re: Proposal for "BigTop Data Generators"
> >>>>
> >>>> Hola nate.  Well, here are the Use cases I know of that I have used
> the
> >>> data generators for.
> >>>>
> >>>> Dockerfile:
> >>>>
> >>>> (1) for testing kubernetes.  For this, I just use transaction-queue
> >>> docker file.
> >>>> (2) for testing GlusterFS small file workloads, maybe with other
> >>> analytics tools...
> >>>>
> >>>> Maven repo
> >>>>
> >>>> (3) Java maprduce/ignite/spark applications, which can just add a mvn
> >>> repo when compiling.  Java developers never add jars through RPM repos.
> >>>>
> >>>> RPM/DEB packages:
> >>>>
> >>>> I could see people using an RPM/DEB data generator, and I'm not
> against
> >>> it.  But I simply don't know of any real world projects which
> *currently*
> >>> need RPM/Deb packages, which is why I haven't bothered to propose it
> as a
> >>> requirement.  Nevertheless linux packages are always a welcome addition
> >> if
> >>> someone wants to create em !
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>> On Sun, Aug 30, 2015 at 4:34 PM, <na...@reactor8.com> wrote:
> >>>>>
> >>>>> Would container be in addition to deb/rpm, or instead of?  If latter
> >>>>> can we do deb/rpm as base then have container either created from
> them
> >>>>> or directly from artifacts?
> >>>>>
> >>>>> On test usage side, seems could probably break up tests into
> >>>>> base/required and then optional/add-on tests/test-suites.  Think
> >>>>> remember seeing mention of certain tests that are failing at times on
> >>>>> certain component(s) anyways in the core builds but don’t mean that
> >>>>> the build is broken, so would make sense to have some clean up around
> >>> those anyways.
> >>>>>
> >>>>> -----Original Message-----
> >>>>> From: RJ Nowling [mailto:rnowling@gmail.com]
> >>>>> Sent: Sunday, August 30, 2015 1:11 PM
> >>>>> To: dev@bigtop.apache.org
> >>>>> Subject: Re: Proposal for "BigTop Data Generators"
> >>>>>
> >>>>> I agree with the above. :)
> >>>>>
> >>>>> On Sun, Aug 30, 2015 at 11:19 AM, Jay Vyas
> >>>>> <ja...@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi RJ.
> >>>>>>
> >>>>>> Maven repositories and docker containers for the transaction queue
> >>>>>> are good enough IMO.  That will give people a way to compose them in
> >>>>>> different idioms (one for Java folks, another for broader Linux
> >>>>>> audience
> >>>>> ).
> >>>>>>
> >>>>>> I think the lib designs are fairly intuitive.  I would say that we
> >>>>>> should constrain them all to being written in Java or Groovy to keep
> >>>>>> the bigtop theme of "JVM for everything" :).
> >>>>>>
> >>>>>> Any particular questions you have around technical design can be
> >>>>>> followed in a JIRA or else maybe a Readme spec that goes in a  top
> >>>>>> level of the data-generators dir...
> >>>>>>
> >>>>>>> On Aug 30, 2015, at 1:51 AM, RJ Nowling <rn...@gmail.com>
> wrote:
> >>>>>>>
> >>>>>>> I'd like to keep this conversation going.
> >>>>>>>
> >>>>>>> So here are a few discussion points:
> >>>>>>>
> >>>>>>> 1. How do we want to make the data generators available?  Maven?
> >>>>>>> RPMs
> >>>>>> and
> >>>>>>> Debs?
> >>>>>>>
> >>>>>>> For now, I'm using a gradle multi-project build to easily build
> >>>>>>> and
> >>>>>> install
> >>>>>>> the BPS data generators and its libraries into a local maven repo.
> >>>>>>> This makes development easy.  Eventually, I would like to post
> >>>>>>> binaries
> >>>>>> through
> >>>>>>> Maven for easy integration by users.  RPMs / Debs could be
> >>>>>>> interesting since I use a pattern where the data generators are
> >>>>>>> libraries (to support application integration / parallelization by
> >>>>>>> the host framework) but also provide CLI drivers for local testing.
> >>>>>>>
> >>>>>>> 2.  The idea of using the data generators as part of the smoke
> >>>>>>> tests came up.  Since there is concern about making the data
> >>>>>>> generators required, we could offer the blueprints (BigPetStore)
> >>>>>>> as optional smoke tests.  Would that be a good compromise?
> >>>>>>>
> >>>>>>> 3.  How will they be maintained?
> >>>>>>>
> >>>>>>> I'll certainly add myself to the maintainers list and will be
> >>>>>>> taking responsibility.  I'm happy to have others help as well if
> >>>>>>> anyone wants to
> >>>>>>> -- if not, that's cool, too.
> >>>>>>>
> >>>>>>> 4. Is anyone interested at all in discussing library APIs and
> >> designs?
> >>>>>>> What about internal interfaces and such?
> >>>>>>>
> >>>>>>>
> >>>>>>> My plan was to add at least one more data generator (weather
> >>>>>>> simulator)
> >>>>>> to
> >>>>>>> bigtop-data-generators in the short term.  However, given the
> >>>>>>> concerns raised by Cos (more discussion needed) and Olaf (don't
> >>>>>>> want to force data generators on unsuspecting users ;) ), I would
> >>>>>>> like to reach some
> >>>>>> consensus
> >>>>>>> on what people are concerned about and solutions.
> >>>>>>>
> >>>>>>> On Thu, Aug 27, 2015 at 12:38 PM, Konstantin Boudnik
> >>>>>>> <co...@apache.org>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>> Fine by me. I have linked this thread to the JIRA ticket that RJ
> >>>>>> created,
> >>>>>>>> so
> >>>>>>>> we have a way to connect one to another ;)
> >>>>>>>>
> >>>>>>>>> On Thu, Aug 27, 2015 at 01:02PM, Olaf Flebbe wrote:
> >>>>>>>>> Hi,
> >>>>>>>>>
> >>>>>>>>> I am not confident that moving important design discussions with
> >>>>>>>>> impact
> >>>>>>>> to
> >>>>>>>>> the whole project to jira is a good idea.
> >>>>>>>>>
> >>>>>>>>> In the current JIRA Traffic storm it is not easy to identify and
> >>>>>>>>> follow
> >>>>>>>> important tickets.
> >>>>>>>>>
> >>>>>>>>> Please keep discussions on the list or at least, please state on
> >>>>>>>>> this
> >>>>>>>> list which Ticket to follow ...
> >>>>>>>>>
> >>>>>>>>> Olaf
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> Am 26.08.2015 um 22:56 schrieb Konstantin Boudnik <
> >> cos@apache.org
> >>>> :
> >>>>>>>>>>
> >>>>>>>>>>> On Wed, Aug 26, 2015 at 10:38PM, Olaf Flebbe wrote:
> >>>>>>>>>>> Hi,
> >>>>>>>>>>>
> >>>>>>>>>>> Nive to have data generators in Bigtop.
> >>>>>>>>>>>
> >>>>>>>>>>> But please do not include it in bigtop_utils, since this
> >>>>>>>>>>> package is mandatory. Not everyone needs a data generator .
> >>>>>>>>>>
> >>>>>>>>>> Yup. And let's move further design discussion to the JIRA!
> >>>>>>>>>>
> >>>>>>>>>>> Olaf
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> Am 26.08.2015 um 11:25 schrieb Jay Vyas <
> >>>>>> jayunit100.apache@gmail.com
> >>>>>>>>> :
> >>>>>>>>>>>>
> >>>>>>>>>>>> Publishing the jar to bigtops maven is probably a good first
> >>>>>>>>>>>> step
> >>>>>>>> ,Then apps can just include it as needed...?.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I'm not against packaging if someone wants packages for this.
> >>>>>>>>>>>> Maybe
> >>>>>>>> even include it in bigtop util ?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Let's move to jira,
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On Aug 25, 2015, at 9:41 PM, Konstantin Boudnik
> >>>>>>>>>>>>> <co...@apache.org>
> >>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> It is pretty cool indeed!
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I wonder how it needs to be structured to be:
> >>>>>>>>>>>>> - easy to access/use from other components wherever it is
> >>>>>>>>>>>>> needed
> >>>>>>>>>>>>> - doesn't interfere with the rest of the stack
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I guess one possible way would be to implement the generator
> >>>>>>>>>>>>> as a
> >>>>>>>> set of maven
> >>>>>>>>>>>>> artifacts, that could be installed/consumed transparently by
> >>>>>>>>>>>>> just
> >>>>>>>> declaring a
> >>>>>>>>>>>>> dependency e.g as proposed via top-level component.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Another way is to have a new package like we do for
> >>>>>>>>>>>>> bigtop-utils
> >>>>>>>> and such.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Perhaps this discussion should be moved to JIRA or shall we
> >>>>>>>> continue on the
> >>>>>>>>>>>>> dev@ ??
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cos
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Sun, Aug 23, 2015 at 11:53AM, RJ Nowling wrote:
> >>>>>>>>>>>>>> Hi BigTop,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I had a discussion with Jay yesterday, we'd like to propose
> >>>>>>>>>>>>>> a new
> >>>>>>>> component
> >>>>>>>>>>>>>> for BigTop: BigTop Data Generators.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> BigTop Data Generators would consist of a common set of
> >>>>>>>>>>>>>> libraries
> >>>>>>>> for
> >>>>>>>>>>>>>> building data generators and three example data generators:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> * BigPetStore transaction generator (moved from
> >>>>>>>>>>>>>> BigPetStore)
> >>>>>>>>>>>>>> * BigTop Bazaar -- attendee movement and interactions with
> >>>>>>>>>>>>>> booths
> >>>>>>>> on a
> >>>>>>>>>>>>>> showroom floor, at a conference, or at a mall
> >>>>>>>>>>>>>> * BigTop Weatherman -- stochastic weather simulation
> >>>>>>>> (temperature, wind
> >>>>>>>>>>>>>> speed, wind chill, rainfall, etc.) per zip code.  (From a
> >>>>>>>>>>>>>> model
> >>>>>>>> trained on
> >>>>>>>>>>>>>> NOAA historical weather data)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> We believe that creating a common set of libraries will
> >>>>>>>>>>>>>> have
> >>>>>>>> several
> >>>>>>>>>>>>>> benefits including:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> * Easier for others to build their own data generators
> >>>>>>>>>>>>>> * Make data generators smaller and easier to maintain
> >>>>>>>>>>>>>> * Share improvements across the data generators
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> More details on the libraries are below.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> BigPetStore will be continue to focus on building  and
> >>>>>>>>>>>>>> maintaining blueprints, powered by the BigTop Data
> >> Generators.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Our vision is that we get all of Apache coming to BigTop
> >>>>>>>>>>>>>> for tools
> >>>>>>>> for
> >>>>>>>>>>>>>> building better, more comprehensive blueprints.  We want to
> >>>>>>>> support these
> >>>>>>>>>>>>>> efforts through data generators and the initial set of
> >>>>>>>>>>>>>> blueprint
> >>>>>>>> we've been
> >>>>>>>>>>>>>> building.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> If the community is generally in support of this, I can
> >>>>>>>>>>>>>> create a
> >>>>>>>> top-level
> >>>>>>>>>>>>>> "bigtop-data-generators" directory and put the data
> >>>>>>>>>>>>>> generators and libraries in there.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> RJ
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -------
> >>>>>>>>>>>>>> Library details:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> So far, I've extracted the following common libraries:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> * Samplers -- provides classes for PDFs and various
> >>>>>>>>>>>>>> samplers
> >>>>>>>>>>>>>> * Name generator -- data set and samplers for generating
> >>>>>>>>>>>>>> names
> >>>>>>>>>>>>>> * Location data set -- data set and classes for US zip
> >>>>>>>>>>>>>> codes,
> >>>>>>>> their
> >>>>>>>>>>>>>> GPS coordinates, median house hold incomes, and population
> >>>>>>>>>>>>>> sizes
> >>>>>>>>>>>>>> * Product generator -- library for enumerating products
> >>>>>>>>>>>>>> from a specification file.  Comes with default
> >>>>>>>>>>>>>> specifications for
> >>>>>>>> BigPetStore
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I also expect that I'll add libraries for:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> * Particle simulation -- customer movement in a room
> >>>>>>>>>>>>>> * Latent factor model generation -- generate latent
> >>>>>>>>>>>>>> factors and customer weights to create something like
> >>> MovieLens data.
> >>>>>>>>>>>>>> Used in
> >>>>>>>> Bazaar
> >>>>>>>>>>>>>> for booth preferences and potentially in BigPetStore for
> >>>>>>>>>>>>>> customer
> >>>>>>>> item
> >>>>>>>>>>>>>> preferences
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Most of these libraries came out of the BigPetStore data
> >>>>>>>>>>>>>> generator
> >>>>>>>> but the
> >>>>>>>>>>>>>> other generators have been refactored to be based off the
> >>>>>>>>>>>>>> standard
> >>>>>>>> set of
> >>>>>>>>>>>>>> libraries.
> >>>>
> >>>>
> >>>> --
> >>>> jay vyas
> >>
>

Re: Proposal for "BigTop Data Generators"

Posted by Evans Ye <ev...@apache.org>.
Or probably it's way simpler to juat have one script since the data
generator runs once and it can be gone.
2015年8月31日 下午10:45於 "Jay Vyas" <ja...@gmail.com>寫道:

> Rj can we abstract the command line so that we have "one cli to rule them
> all" into an interface?
>
>
> > On Aug 31, 2015, at 10:40 AM, Evans Ye <ev...@apache.org> wrote:
> >
> > I am very much like the shell script wrapper and docker image idea since
> > that way we can integrate it directly with bigtop provisioner which
> yield a
> > perfect ux for the whole things. I think its not too hard to do it both,
> we
> > just need to add a parameter to turn the script into daemon mode. I see
> > lots of image doing this way.
> >
> > docker run bigtop/bigtop-data-gen --scheme weather --size 5GB --output
> > data-dir --etc  foo --etc bar --daemon
> > 2015年8月31日 下午9:06於 "RJ Nowling" <rn...@gmail.com>寫道:
> >
> >> The BigPetStore, Bazaar, and weather data generators have
> single-threaded
> >> command-line interfaces.  We could do the same with the smaller
> generators
> >> (names, locations, etc.) if there is interest.
> >>
> >> On Mon, Aug 31, 2015 at 5:24 AM, Jay Vyas <ja...@gmail.com>
> >> wrote:
> >>
> >>> Nate: Good idea to abstract the interface one level higher....
> >>>
> >>> How about a docker run command ? That is probably the easiest way for
> >>> Linux folks to run one off Java apps nowadays.
> >>>
> >>> docker run bigtop/bigtop-data-gen --scheme weather --size 5GB --output
> >>> data-dir --etc  foo --etc bar
> >>>
> >>> I'm happy to curate such a docker image, I already am doing something
> >> like
> >>> this in kube for bigtop-transaction-queue, which continuously pumps
> data
> >>> generator outputs into a REST endpoint or file
> >>> Queue... So it could be extended to support other generators.
> >>>
> >>>
> >>>> om> <na...@reactor8.com> wrote:
> >>>>
> >>>> Could picture at some point supporting something like this for non-jvm
> >>> folk just looking for test/demo data:
> >>>>
> >>>> apt-get install bigtop-data-gen
> >>>> ~/ $ bigtop-data-gen --scheme weather --size 5GB --output data-dir
> >>> --etc  foo --etc bar
> >>>>
> >>>>
> >>>>
> >>>> -----Original Message-----
> >>>> From: jay vyas [mailto:jayunit100.apache@gmail.com]
> >>>> Sent: Sunday, August 30, 2015 5:11 PM
> >>>> To: dev@bigtop.apache.org
> >>>> Subject: Re: Proposal for "BigTop Data Generators"
> >>>>
> >>>> Hola nate.  Well, here are the Use cases I know of that I have used
> the
> >>> data generators for.
> >>>>
> >>>> Dockerfile:
> >>>>
> >>>> (1) for testing kubernetes.  For this, I just use transaction-queue
> >>> docker file.
> >>>> (2) for testing GlusterFS small file workloads, maybe with other
> >>> analytics tools...
> >>>>
> >>>> Maven repo
> >>>>
> >>>> (3) Java maprduce/ignite/spark applications, which can just add a mvn
> >>> repo when compiling.  Java developers never add jars through RPM repos.
> >>>>
> >>>> RPM/DEB packages:
> >>>>
> >>>> I could see people using an RPM/DEB data generator, and I'm not
> against
> >>> it.  But I simply don't know of any real world projects which
> *currently*
> >>> need RPM/Deb packages, which is why I haven't bothered to propose it
> as a
> >>> requirement.  Nevertheless linux packages are always a welcome addition
> >> if
> >>> someone wants to create em !
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>> On Sun, Aug 30, 2015 at 4:34 PM, <na...@reactor8.com> wrote:
> >>>>>
> >>>>> Would container be in addition to deb/rpm, or instead of?  If latter
> >>>>> can we do deb/rpm as base then have container either created from
> them
> >>>>> or directly from artifacts?
> >>>>>
> >>>>> On test usage side, seems could probably break up tests into
> >>>>> base/required and then optional/add-on tests/test-suites.  Think
> >>>>> remember seeing mention of certain tests that are failing at times on
> >>>>> certain component(s) anyways in the core builds but don’t mean that
> >>>>> the build is broken, so would make sense to have some clean up around
> >>> those anyways.
> >>>>>
> >>>>> -----Original Message-----
> >>>>> From: RJ Nowling [mailto:rnowling@gmail.com]
> >>>>> Sent: Sunday, August 30, 2015 1:11 PM
> >>>>> To: dev@bigtop.apache.org
> >>>>> Subject: Re: Proposal for "BigTop Data Generators"
> >>>>>
> >>>>> I agree with the above. :)
> >>>>>
> >>>>> On Sun, Aug 30, 2015 at 11:19 AM, Jay Vyas
> >>>>> <ja...@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi RJ.
> >>>>>>
> >>>>>> Maven repositories and docker containers for the transaction queue
> >>>>>> are good enough IMO.  That will give people a way to compose them in
> >>>>>> different idioms (one for Java folks, another for broader Linux
> >>>>>> audience
> >>>>> ).
> >>>>>>
> >>>>>> I think the lib designs are fairly intuitive.  I would say that we
> >>>>>> should constrain them all to being written in Java or Groovy to keep
> >>>>>> the bigtop theme of "JVM for everything" :).
> >>>>>>
> >>>>>> Any particular questions you have around technical design can be
> >>>>>> followed in a JIRA or else maybe a Readme spec that goes in a  top
> >>>>>> level of the data-generators dir...
> >>>>>>
> >>>>>>> On Aug 30, 2015, at 1:51 AM, RJ Nowling <rn...@gmail.com>
> wrote:
> >>>>>>>
> >>>>>>> I'd like to keep this conversation going.
> >>>>>>>
> >>>>>>> So here are a few discussion points:
> >>>>>>>
> >>>>>>> 1. How do we want to make the data generators available?  Maven?
> >>>>>>> RPMs
> >>>>>> and
> >>>>>>> Debs?
> >>>>>>>
> >>>>>>> For now, I'm using a gradle multi-project build to easily build
> >>>>>>> and
> >>>>>> install
> >>>>>>> the BPS data generators and its libraries into a local maven repo.
> >>>>>>> This makes development easy.  Eventually, I would like to post
> >>>>>>> binaries
> >>>>>> through
> >>>>>>> Maven for easy integration by users.  RPMs / Debs could be
> >>>>>>> interesting since I use a pattern where the data generators are
> >>>>>>> libraries (to support application integration / parallelization by
> >>>>>>> the host framework) but also provide CLI drivers for local testing.
> >>>>>>>
> >>>>>>> 2.  The idea of using the data generators as part of the smoke
> >>>>>>> tests came up.  Since there is concern about making the data
> >>>>>>> generators required, we could offer the blueprints (BigPetStore)
> >>>>>>> as optional smoke tests.  Would that be a good compromise?
> >>>>>>>
> >>>>>>> 3.  How will they be maintained?
> >>>>>>>
> >>>>>>> I'll certainly add myself to the maintainers list and will be
> >>>>>>> taking responsibility.  I'm happy to have others help as well if
> >>>>>>> anyone wants to
> >>>>>>> -- if not, that's cool, too.
> >>>>>>>
> >>>>>>> 4. Is anyone interested at all in discussing library APIs and
> >> designs?
> >>>>>>> What about internal interfaces and such?
> >>>>>>>
> >>>>>>>
> >>>>>>> My plan was to add at least one more data generator (weather
> >>>>>>> simulator)
> >>>>>> to
> >>>>>>> bigtop-data-generators in the short term.  However, given the
> >>>>>>> concerns raised by Cos (more discussion needed) and Olaf (don't
> >>>>>>> want to force data generators on unsuspecting users ;) ), I would
> >>>>>>> like to reach some
> >>>>>> consensus
> >>>>>>> on what people are concerned about and solutions.
> >>>>>>>
> >>>>>>> On Thu, Aug 27, 2015 at 12:38 PM, Konstantin Boudnik
> >>>>>>> <co...@apache.org>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>> Fine by me. I have linked this thread to the JIRA ticket that RJ
> >>>>>> created,
> >>>>>>>> so
> >>>>>>>> we have a way to connect one to another ;)
> >>>>>>>>
> >>>>>>>>> On Thu, Aug 27, 2015 at 01:02PM, Olaf Flebbe wrote:
> >>>>>>>>> Hi,
> >>>>>>>>>
> >>>>>>>>> I am not confident that moving important design discussions with
> >>>>>>>>> impact
> >>>>>>>> to
> >>>>>>>>> the whole project to jira is a good idea.
> >>>>>>>>>
> >>>>>>>>> In the current JIRA Traffic storm it is not easy to identify and
> >>>>>>>>> follow
> >>>>>>>> important tickets.
> >>>>>>>>>
> >>>>>>>>> Please keep discussions on the list or at least, please state on
> >>>>>>>>> this
> >>>>>>>> list which Ticket to follow ...
> >>>>>>>>>
> >>>>>>>>> Olaf
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> Am 26.08.2015 um 22:56 schrieb Konstantin Boudnik <
> >> cos@apache.org
> >>>> :
> >>>>>>>>>>
> >>>>>>>>>>> On Wed, Aug 26, 2015 at 10:38PM, Olaf Flebbe wrote:
> >>>>>>>>>>> Hi,
> >>>>>>>>>>>
> >>>>>>>>>>> Nive to have data generators in Bigtop.
> >>>>>>>>>>>
> >>>>>>>>>>> But please do not include it in bigtop_utils, since this
> >>>>>>>>>>> package is mandatory. Not everyone needs a data generator .
> >>>>>>>>>>
> >>>>>>>>>> Yup. And let's move further design discussion to the JIRA!
> >>>>>>>>>>
> >>>>>>>>>>> Olaf
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> Am 26.08.2015 um 11:25 schrieb Jay Vyas <
> >>>>>> jayunit100.apache@gmail.com
> >>>>>>>>> :
> >>>>>>>>>>>>
> >>>>>>>>>>>> Publishing the jar to bigtops maven is probably a good first
> >>>>>>>>>>>> step
> >>>>>>>> ,Then apps can just include it as needed...?.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I'm not against packaging if someone wants packages for this.
> >>>>>>>>>>>> Maybe
> >>>>>>>> even include it in bigtop util ?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Let's move to jira,
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On Aug 25, 2015, at 9:41 PM, Konstantin Boudnik
> >>>>>>>>>>>>> <co...@apache.org>
> >>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> It is pretty cool indeed!
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I wonder how it needs to be structured to be:
> >>>>>>>>>>>>> - easy to access/use from other components wherever it is
> >>>>>>>>>>>>> needed
> >>>>>>>>>>>>> - doesn't interfere with the rest of the stack
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I guess one possible way would be to implement the generator
> >>>>>>>>>>>>> as a
> >>>>>>>> set of maven
> >>>>>>>>>>>>> artifacts, that could be installed/consumed transparently by
> >>>>>>>>>>>>> just
> >>>>>>>> declaring a
> >>>>>>>>>>>>> dependency e.g as proposed via top-level component.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Another way is to have a new package like we do for
> >>>>>>>>>>>>> bigtop-utils
> >>>>>>>> and such.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Perhaps this discussion should be moved to JIRA or shall we
> >>>>>>>> continue on the
> >>>>>>>>>>>>> dev@ ??
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cos
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Sun, Aug 23, 2015 at 11:53AM, RJ Nowling wrote:
> >>>>>>>>>>>>>> Hi BigTop,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I had a discussion with Jay yesterday, we'd like to propose
> >>>>>>>>>>>>>> a new
> >>>>>>>> component
> >>>>>>>>>>>>>> for BigTop: BigTop Data Generators.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> BigTop Data Generators would consist of a common set of
> >>>>>>>>>>>>>> libraries
> >>>>>>>> for
> >>>>>>>>>>>>>> building data generators and three example data generators:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> * BigPetStore transaction generator (moved from
> >>>>>>>>>>>>>> BigPetStore)
> >>>>>>>>>>>>>> * BigTop Bazaar -- attendee movement and interactions with
> >>>>>>>>>>>>>> booths
> >>>>>>>> on a
> >>>>>>>>>>>>>> showroom floor, at a conference, or at a mall
> >>>>>>>>>>>>>> * BigTop Weatherman -- stochastic weather simulation
> >>>>>>>> (temperature, wind
> >>>>>>>>>>>>>> speed, wind chill, rainfall, etc.) per zip code.  (From a
> >>>>>>>>>>>>>> model
> >>>>>>>> trained on
> >>>>>>>>>>>>>> NOAA historical weather data)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> We believe that creating a common set of libraries will
> >>>>>>>>>>>>>> have
> >>>>>>>> several
> >>>>>>>>>>>>>> benefits including:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> * Easier for others to build their own data generators
> >>>>>>>>>>>>>> * Make data generators smaller and easier to maintain
> >>>>>>>>>>>>>> * Share improvements across the data generators
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> More details on the libraries are below.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> BigPetStore will be continue to focus on building  and
> >>>>>>>>>>>>>> maintaining blueprints, powered by the BigTop Data
> >> Generators.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Our vision is that we get all of Apache coming to BigTop
> >>>>>>>>>>>>>> for tools
> >>>>>>>> for
> >>>>>>>>>>>>>> building better, more comprehensive blueprints.  We want to
> >>>>>>>> support these
> >>>>>>>>>>>>>> efforts through data generators and the initial set of
> >>>>>>>>>>>>>> blueprint
> >>>>>>>> we've been
> >>>>>>>>>>>>>> building.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> If the community is generally in support of this, I can
> >>>>>>>>>>>>>> create a
> >>>>>>>> top-level
> >>>>>>>>>>>>>> "bigtop-data-generators" directory and put the data
> >>>>>>>>>>>>>> generators and libraries in there.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> RJ
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -------
> >>>>>>>>>>>>>> Library details:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> So far, I've extracted the following common libraries:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> * Samplers -- provides classes for PDFs and various
> >>>>>>>>>>>>>> samplers
> >>>>>>>>>>>>>> * Name generator -- data set and samplers for generating
> >>>>>>>>>>>>>> names
> >>>>>>>>>>>>>> * Location data set -- data set and classes for US zip
> >>>>>>>>>>>>>> codes,
> >>>>>>>> their
> >>>>>>>>>>>>>> GPS coordinates, median house hold incomes, and population
> >>>>>>>>>>>>>> sizes
> >>>>>>>>>>>>>> * Product generator -- library for enumerating products
> >>>>>>>>>>>>>> from a specification file.  Comes with default
> >>>>>>>>>>>>>> specifications for
> >>>>>>>> BigPetStore
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I also expect that I'll add libraries for:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> * Particle simulation -- customer movement in a room
> >>>>>>>>>>>>>> * Latent factor model generation -- generate latent
> >>>>>>>>>>>>>> factors and customer weights to create something like
> >>> MovieLens data.
> >>>>>>>>>>>>>> Used in
> >>>>>>>> Bazaar
> >>>>>>>>>>>>>> for booth preferences and potentially in BigPetStore for
> >>>>>>>>>>>>>> customer
> >>>>>>>> item
> >>>>>>>>>>>>>> preferences
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Most of these libraries came out of the BigPetStore data
> >>>>>>>>>>>>>> generator
> >>>>>>>> but the
> >>>>>>>>>>>>>> other generators have been refactored to be based off the
> >>>>>>>>>>>>>> standard
> >>>>>>>> set of
> >>>>>>>>>>>>>> libraries.
> >>>>
> >>>>
> >>>> --
> >>>> jay vyas
> >>
>

Re: Proposal for "BigTop Data Generators"

Posted by Jay Vyas <ja...@gmail.com>.
Rj can we abstract the command line so that we have "one cli to rule them all" into an interface?


> On Aug 31, 2015, at 10:40 AM, Evans Ye <ev...@apache.org> wrote:
> 
> I am very much like the shell script wrapper and docker image idea since
> that way we can integrate it directly with bigtop provisioner which yield a
> perfect ux for the whole things. I think its not too hard to do it both, we
> just need to add a parameter to turn the script into daemon mode. I see
> lots of image doing this way.
> 
> docker run bigtop/bigtop-data-gen --scheme weather --size 5GB --output
> data-dir --etc  foo --etc bar --daemon
> 2015年8月31日 下午9:06於 "RJ Nowling" <rn...@gmail.com>寫道:
> 
>> The BigPetStore, Bazaar, and weather data generators have single-threaded
>> command-line interfaces.  We could do the same with the smaller generators
>> (names, locations, etc.) if there is interest.
>> 
>> On Mon, Aug 31, 2015 at 5:24 AM, Jay Vyas <ja...@gmail.com>
>> wrote:
>> 
>>> Nate: Good idea to abstract the interface one level higher....
>>> 
>>> How about a docker run command ? That is probably the easiest way for
>>> Linux folks to run one off Java apps nowadays.
>>> 
>>> docker run bigtop/bigtop-data-gen --scheme weather --size 5GB --output
>>> data-dir --etc  foo --etc bar
>>> 
>>> I'm happy to curate such a docker image, I already am doing something
>> like
>>> this in kube for bigtop-transaction-queue, which continuously pumps data
>>> generator outputs into a REST endpoint or file
>>> Queue... So it could be extended to support other generators.
>>> 
>>> 
>>>> om> <na...@reactor8.com> wrote:
>>>> 
>>>> Could picture at some point supporting something like this for non-jvm
>>> folk just looking for test/demo data:
>>>> 
>>>> apt-get install bigtop-data-gen
>>>> ~/ $ bigtop-data-gen --scheme weather --size 5GB --output data-dir
>>> --etc  foo --etc bar
>>>> 
>>>> 
>>>> 
>>>> -----Original Message-----
>>>> From: jay vyas [mailto:jayunit100.apache@gmail.com]
>>>> Sent: Sunday, August 30, 2015 5:11 PM
>>>> To: dev@bigtop.apache.org
>>>> Subject: Re: Proposal for "BigTop Data Generators"
>>>> 
>>>> Hola nate.  Well, here are the Use cases I know of that I have used the
>>> data generators for.
>>>> 
>>>> Dockerfile:
>>>> 
>>>> (1) for testing kubernetes.  For this, I just use transaction-queue
>>> docker file.
>>>> (2) for testing GlusterFS small file workloads, maybe with other
>>> analytics tools...
>>>> 
>>>> Maven repo
>>>> 
>>>> (3) Java maprduce/ignite/spark applications, which can just add a mvn
>>> repo when compiling.  Java developers never add jars through RPM repos.
>>>> 
>>>> RPM/DEB packages:
>>>> 
>>>> I could see people using an RPM/DEB data generator, and I'm not against
>>> it.  But I simply don't know of any real world projects which *currently*
>>> need RPM/Deb packages, which is why I haven't bothered to propose it as a
>>> requirement.  Nevertheless linux packages are always a welcome addition
>> if
>>> someone wants to create em !
>>>> 
>>>> 
>>>> 
>>>> 
>>>>> On Sun, Aug 30, 2015 at 4:34 PM, <na...@reactor8.com> wrote:
>>>>> 
>>>>> Would container be in addition to deb/rpm, or instead of?  If latter
>>>>> can we do deb/rpm as base then have container either created from them
>>>>> or directly from artifacts?
>>>>> 
>>>>> On test usage side, seems could probably break up tests into
>>>>> base/required and then optional/add-on tests/test-suites.  Think
>>>>> remember seeing mention of certain tests that are failing at times on
>>>>> certain component(s) anyways in the core builds but don’t mean that
>>>>> the build is broken, so would make sense to have some clean up around
>>> those anyways.
>>>>> 
>>>>> -----Original Message-----
>>>>> From: RJ Nowling [mailto:rnowling@gmail.com]
>>>>> Sent: Sunday, August 30, 2015 1:11 PM
>>>>> To: dev@bigtop.apache.org
>>>>> Subject: Re: Proposal for "BigTop Data Generators"
>>>>> 
>>>>> I agree with the above. :)
>>>>> 
>>>>> On Sun, Aug 30, 2015 at 11:19 AM, Jay Vyas
>>>>> <ja...@gmail.com>
>>>>> wrote:
>>>>> 
>>>>>> Hi RJ.
>>>>>> 
>>>>>> Maven repositories and docker containers for the transaction queue
>>>>>> are good enough IMO.  That will give people a way to compose them in
>>>>>> different idioms (one for Java folks, another for broader Linux
>>>>>> audience
>>>>> ).
>>>>>> 
>>>>>> I think the lib designs are fairly intuitive.  I would say that we
>>>>>> should constrain them all to being written in Java or Groovy to keep
>>>>>> the bigtop theme of "JVM for everything" :).
>>>>>> 
>>>>>> Any particular questions you have around technical design can be
>>>>>> followed in a JIRA or else maybe a Readme spec that goes in a  top
>>>>>> level of the data-generators dir...
>>>>>> 
>>>>>>> On Aug 30, 2015, at 1:51 AM, RJ Nowling <rn...@gmail.com> wrote:
>>>>>>> 
>>>>>>> I'd like to keep this conversation going.
>>>>>>> 
>>>>>>> So here are a few discussion points:
>>>>>>> 
>>>>>>> 1. How do we want to make the data generators available?  Maven?
>>>>>>> RPMs
>>>>>> and
>>>>>>> Debs?
>>>>>>> 
>>>>>>> For now, I'm using a gradle multi-project build to easily build
>>>>>>> and
>>>>>> install
>>>>>>> the BPS data generators and its libraries into a local maven repo.
>>>>>>> This makes development easy.  Eventually, I would like to post
>>>>>>> binaries
>>>>>> through
>>>>>>> Maven for easy integration by users.  RPMs / Debs could be
>>>>>>> interesting since I use a pattern where the data generators are
>>>>>>> libraries (to support application integration / parallelization by
>>>>>>> the host framework) but also provide CLI drivers for local testing.
>>>>>>> 
>>>>>>> 2.  The idea of using the data generators as part of the smoke
>>>>>>> tests came up.  Since there is concern about making the data
>>>>>>> generators required, we could offer the blueprints (BigPetStore)
>>>>>>> as optional smoke tests.  Would that be a good compromise?
>>>>>>> 
>>>>>>> 3.  How will they be maintained?
>>>>>>> 
>>>>>>> I'll certainly add myself to the maintainers list and will be
>>>>>>> taking responsibility.  I'm happy to have others help as well if
>>>>>>> anyone wants to
>>>>>>> -- if not, that's cool, too.
>>>>>>> 
>>>>>>> 4. Is anyone interested at all in discussing library APIs and
>> designs?
>>>>>>> What about internal interfaces and such?
>>>>>>> 
>>>>>>> 
>>>>>>> My plan was to add at least one more data generator (weather
>>>>>>> simulator)
>>>>>> to
>>>>>>> bigtop-data-generators in the short term.  However, given the
>>>>>>> concerns raised by Cos (more discussion needed) and Olaf (don't
>>>>>>> want to force data generators on unsuspecting users ;) ), I would
>>>>>>> like to reach some
>>>>>> consensus
>>>>>>> on what people are concerned about and solutions.
>>>>>>> 
>>>>>>> On Thu, Aug 27, 2015 at 12:38 PM, Konstantin Boudnik
>>>>>>> <co...@apache.org>
>>>>>> wrote:
>>>>>>> 
>>>>>>>> Fine by me. I have linked this thread to the JIRA ticket that RJ
>>>>>> created,
>>>>>>>> so
>>>>>>>> we have a way to connect one to another ;)
>>>>>>>> 
>>>>>>>>> On Thu, Aug 27, 2015 at 01:02PM, Olaf Flebbe wrote:
>>>>>>>>> Hi,
>>>>>>>>> 
>>>>>>>>> I am not confident that moving important design discussions with
>>>>>>>>> impact
>>>>>>>> to
>>>>>>>>> the whole project to jira is a good idea.
>>>>>>>>> 
>>>>>>>>> In the current JIRA Traffic storm it is not easy to identify and
>>>>>>>>> follow
>>>>>>>> important tickets.
>>>>>>>>> 
>>>>>>>>> Please keep discussions on the list or at least, please state on
>>>>>>>>> this
>>>>>>>> list which Ticket to follow ...
>>>>>>>>> 
>>>>>>>>> Olaf
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> Am 26.08.2015 um 22:56 schrieb Konstantin Boudnik <
>> cos@apache.org
>>>> :
>>>>>>>>>> 
>>>>>>>>>>> On Wed, Aug 26, 2015 at 10:38PM, Olaf Flebbe wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>> 
>>>>>>>>>>> Nive to have data generators in Bigtop.
>>>>>>>>>>> 
>>>>>>>>>>> But please do not include it in bigtop_utils, since this
>>>>>>>>>>> package is mandatory. Not everyone needs a data generator .
>>>>>>>>>> 
>>>>>>>>>> Yup. And let's move further design discussion to the JIRA!
>>>>>>>>>> 
>>>>>>>>>>> Olaf
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> Am 26.08.2015 um 11:25 schrieb Jay Vyas <
>>>>>> jayunit100.apache@gmail.com
>>>>>>>>> :
>>>>>>>>>>>> 
>>>>>>>>>>>> Publishing the jar to bigtops maven is probably a good first
>>>>>>>>>>>> step
>>>>>>>> ,Then apps can just include it as needed...?.
>>>>>>>>>>>> 
>>>>>>>>>>>> I'm not against packaging if someone wants packages for this.
>>>>>>>>>>>> Maybe
>>>>>>>> even include it in bigtop util ?
>>>>>>>>>>>> 
>>>>>>>>>>>> Let's move to jira,
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Aug 25, 2015, at 9:41 PM, Konstantin Boudnik
>>>>>>>>>>>>> <co...@apache.org>
>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> It is pretty cool indeed!
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I wonder how it needs to be structured to be:
>>>>>>>>>>>>> - easy to access/use from other components wherever it is
>>>>>>>>>>>>> needed
>>>>>>>>>>>>> - doesn't interfere with the rest of the stack
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I guess one possible way would be to implement the generator
>>>>>>>>>>>>> as a
>>>>>>>> set of maven
>>>>>>>>>>>>> artifacts, that could be installed/consumed transparently by
>>>>>>>>>>>>> just
>>>>>>>> declaring a
>>>>>>>>>>>>> dependency e.g as proposed via top-level component.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Another way is to have a new package like we do for
>>>>>>>>>>>>> bigtop-utils
>>>>>>>> and such.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Perhaps this discussion should be moved to JIRA or shall we
>>>>>>>> continue on the
>>>>>>>>>>>>> dev@ ??
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Cos
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sun, Aug 23, 2015 at 11:53AM, RJ Nowling wrote:
>>>>>>>>>>>>>> Hi BigTop,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I had a discussion with Jay yesterday, we'd like to propose
>>>>>>>>>>>>>> a new
>>>>>>>> component
>>>>>>>>>>>>>> for BigTop: BigTop Data Generators.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> BigTop Data Generators would consist of a common set of
>>>>>>>>>>>>>> libraries
>>>>>>>> for
>>>>>>>>>>>>>> building data generators and three example data generators:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> * BigPetStore transaction generator (moved from
>>>>>>>>>>>>>> BigPetStore)
>>>>>>>>>>>>>> * BigTop Bazaar -- attendee movement and interactions with
>>>>>>>>>>>>>> booths
>>>>>>>> on a
>>>>>>>>>>>>>> showroom floor, at a conference, or at a mall
>>>>>>>>>>>>>> * BigTop Weatherman -- stochastic weather simulation
>>>>>>>> (temperature, wind
>>>>>>>>>>>>>> speed, wind chill, rainfall, etc.) per zip code.  (From a
>>>>>>>>>>>>>> model
>>>>>>>> trained on
>>>>>>>>>>>>>> NOAA historical weather data)
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> We believe that creating a common set of libraries will
>>>>>>>>>>>>>> have
>>>>>>>> several
>>>>>>>>>>>>>> benefits including:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> * Easier for others to build their own data generators
>>>>>>>>>>>>>> * Make data generators smaller and easier to maintain
>>>>>>>>>>>>>> * Share improvements across the data generators
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> More details on the libraries are below.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> BigPetStore will be continue to focus on building  and
>>>>>>>>>>>>>> maintaining blueprints, powered by the BigTop Data
>> Generators.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Our vision is that we get all of Apache coming to BigTop
>>>>>>>>>>>>>> for tools
>>>>>>>> for
>>>>>>>>>>>>>> building better, more comprehensive blueprints.  We want to
>>>>>>>> support these
>>>>>>>>>>>>>> efforts through data generators and the initial set of
>>>>>>>>>>>>>> blueprint
>>>>>>>> we've been
>>>>>>>>>>>>>> building.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> If the community is generally in support of this, I can
>>>>>>>>>>>>>> create a
>>>>>>>> top-level
>>>>>>>>>>>>>> "bigtop-data-generators" directory and put the data
>>>>>>>>>>>>>> generators and libraries in there.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> RJ
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> -------
>>>>>>>>>>>>>> Library details:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> So far, I've extracted the following common libraries:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> * Samplers -- provides classes for PDFs and various
>>>>>>>>>>>>>> samplers
>>>>>>>>>>>>>> * Name generator -- data set and samplers for generating
>>>>>>>>>>>>>> names
>>>>>>>>>>>>>> * Location data set -- data set and classes for US zip
>>>>>>>>>>>>>> codes,
>>>>>>>> their
>>>>>>>>>>>>>> GPS coordinates, median house hold incomes, and population
>>>>>>>>>>>>>> sizes
>>>>>>>>>>>>>> * Product generator -- library for enumerating products
>>>>>>>>>>>>>> from a specification file.  Comes with default
>>>>>>>>>>>>>> specifications for
>>>>>>>> BigPetStore
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I also expect that I'll add libraries for:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> * Particle simulation -- customer movement in a room
>>>>>>>>>>>>>> * Latent factor model generation -- generate latent
>>>>>>>>>>>>>> factors and customer weights to create something like
>>> MovieLens data.
>>>>>>>>>>>>>> Used in
>>>>>>>> Bazaar
>>>>>>>>>>>>>> for booth preferences and potentially in BigPetStore for
>>>>>>>>>>>>>> customer
>>>>>>>> item
>>>>>>>>>>>>>> preferences
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Most of these libraries came out of the BigPetStore data
>>>>>>>>>>>>>> generator
>>>>>>>> but the
>>>>>>>>>>>>>> other generators have been refactored to be based off the
>>>>>>>>>>>>>> standard
>>>>>>>> set of
>>>>>>>>>>>>>> libraries.
>>>> 
>>>> 
>>>> --
>>>> jay vyas
>> 

Re: Proposal for "BigTop Data Generators"

Posted by Olaf Flebbe <of...@oflebbe.de>.
+1 to the CLI /shell script interface.

If I can choose I like to have a apt-get install bigtop-datagenerator , running for instance

bigtop-data-generatoroutputDir nStores nCustomers nPurchasingModels simulationLength seed

I can help out with packaging if needed.

Why should we use the docker indirection for a plain CLI file ? Of course, We can provide a trivial Dockerfile to create a container supplying a JVM and running the CLI ... But I do not like to depend our services on docker registry more than we do now.

Olaf



> Am 31.08.2015 um 16:40 schrieb Evans Ye <ev...@apache.org>:
> 
> I am very much like the shell script wrapper and docker image idea since
> that way we can integrate it directly with bigtop provisioner which yield a
> perfect ux for the whole things. I think its not too hard to do it both, we
> just need to add a parameter to turn the script into daemon mode. I see
> lots of image doing this way.
> 
> docker run bigtop/bigtop-data-gen --scheme weather --size 5GB --output
> data-dir --etc  foo --etc bar --daemon
> 2015年8月31日 下午9:06於 "RJ Nowling" <rn...@gmail.com>寫道:
> 
>> The BigPetStore, Bazaar, and weather data generators have single-threaded
>> command-line interfaces.  We could do the same with the smaller generators
>> (names, locations, etc.) if there is interest.
>> 
>> On Mon, Aug 31, 2015 at 5:24 AM, Jay Vyas <ja...@gmail.com>
>> wrote:
>> 
>>> Nate: Good idea to abstract the interface one level higher....
>>> 
>>> How about a docker run command ? That is probably the easiest way for
>>> Linux folks to run one off Java apps nowadays.
>>> 
>>> docker run bigtop/bigtop-data-gen --scheme weather --size 5GB --output
>>> data-dir --etc  foo --etc bar
>>> 
>>> I'm happy to curate such a docker image, I already am doing something
>> like
>>> this in kube for bigtop-transaction-queue, which continuously pumps data
>>> generator outputs into a REST endpoint or file
>>> Queue... So it could be extended to support other generators.
>>> 
>>> 
>>>> om> <na...@reactor8.com> wrote:
>>>> 
>>>> Could picture at some point supporting something like this for non-jvm
>>> folk just looking for test/demo data:
>>>> 
>>>> apt-get install bigtop-data-gen
>>>> ~/ $ bigtop-data-gen --scheme weather --size 5GB --output data-dir
>>> --etc  foo --etc bar
>>>> 
>>>> 
>>>> 
>>>> -----Original Message-----
>>>> From: jay vyas [mailto:jayunit100.apache@gmail.com]
>>>> Sent: Sunday, August 30, 2015 5:11 PM
>>>> To: dev@bigtop.apache.org
>>>> Subject: Re: Proposal for "BigTop Data Generators"
>>>> 
>>>> Hola nate.  Well, here are the Use cases I know of that I have used the
>>> data generators for.
>>>> 
>>>> Dockerfile:
>>>> 
>>>> (1) for testing kubernetes.  For this, I just use transaction-queue
>>> docker file.
>>>> (2) for testing GlusterFS small file workloads, maybe with other
>>> analytics tools...
>>>> 
>>>> Maven repo
>>>> 
>>>> (3) Java maprduce/ignite/spark applications, which can just add a mvn
>>> repo when compiling.  Java developers never add jars through RPM repos.
>>>> 
>>>> RPM/DEB packages:
>>>> 
>>>> I could see people using an RPM/DEB data generator, and I'm not against
>>> it.  But I simply don't know of any real world projects which *currently*
>>> need RPM/Deb packages, which is why I haven't bothered to propose it as a
>>> requirement.  Nevertheless linux packages are always a welcome addition
>> if
>>> someone wants to create em !
>>>> 
>>>> 
>>>> 
>>>> 
>>>>> On Sun, Aug 30, 2015 at 4:34 PM, <na...@reactor8.com> wrote:
>>>>> 
>>>>> Would container be in addition to deb/rpm, or instead of?  If latter
>>>>> can we do deb/rpm as base then have container either created from them
>>>>> or directly from artifacts?
>>>>> 
>>>>> On test usage side, seems could probably break up tests into
>>>>> base/required and then optional/add-on tests/test-suites.  Think
>>>>> remember seeing mention of certain tests that are failing at times on
>>>>> certain component(s) anyways in the core builds but don’t mean that
>>>>> the build is broken, so would make sense to have some clean up around
>>> those anyways.
>>>>> 
>>>>> -----Original Message-----
>>>>> From: RJ Nowling [mailto:rnowling@gmail.com]
>>>>> Sent: Sunday, August 30, 2015 1:11 PM
>>>>> To: dev@bigtop.apache.org
>>>>> Subject: Re: Proposal for "BigTop Data Generators"
>>>>> 
>>>>> I agree with the above. :)
>>>>> 
>>>>> On Sun, Aug 30, 2015 at 11:19 AM, Jay Vyas
>>>>> <ja...@gmail.com>
>>>>> wrote:
>>>>> 
>>>>>> Hi RJ.
>>>>>> 
>>>>>> Maven repositories and docker containers for the transaction queue
>>>>>> are good enough IMO.  That will give people a way to compose them in
>>>>>> different idioms (one for Java folks, another for broader Linux
>>>>>> audience
>>>>> ).
>>>>>> 
>>>>>> I think the lib designs are fairly intuitive.  I would say that we
>>>>>> should constrain them all to being written in Java or Groovy to keep
>>>>>> the bigtop theme of "JVM for everything" :).
>>>>>> 
>>>>>> Any particular questions you have around technical design can be
>>>>>> followed in a JIRA or else maybe a Readme spec that goes in a  top
>>>>>> level of the data-generators dir...
>>>>>> 
>>>>>>> On Aug 30, 2015, at 1:51 AM, RJ Nowling <rn...@gmail.com> wrote:
>>>>>>> 
>>>>>>> I'd like to keep this conversation going.
>>>>>>> 
>>>>>>> So here are a few discussion points:
>>>>>>> 
>>>>>>> 1. How do we want to make the data generators available?  Maven?
>>>>>>> RPMs
>>>>>> and
>>>>>>> Debs?
>>>>>>> 
>>>>>>> For now, I'm using a gradle multi-project build to easily build
>>>>>>> and
>>>>>> install
>>>>>>> the BPS data generators and its libraries into a local maven repo.
>>>>>>> This makes development easy.  Eventually, I would like to post
>>>>>>> binaries
>>>>>> through
>>>>>>> Maven for easy integration by users.  RPMs / Debs could be
>>>>>>> interesting since I use a pattern where the data generators are
>>>>>>> libraries (to support application integration / parallelization by
>>>>>>> the host framework) but also provide CLI drivers for local testing.
>>>>>>> 
>>>>>>> 2.  The idea of using the data generators as part of the smoke
>>>>>>> tests came up.  Since there is concern about making the data
>>>>>>> generators required, we could offer the blueprints (BigPetStore)
>>>>>>> as optional smoke tests.  Would that be a good compromise?
>>>>>>> 
>>>>>>> 3.  How will they be maintained?
>>>>>>> 
>>>>>>> I'll certainly add myself to the maintainers list and will be
>>>>>>> taking responsibility.  I'm happy to have others help as well if
>>>>>>> anyone wants to
>>>>>>> -- if not, that's cool, too.
>>>>>>> 
>>>>>>> 4. Is anyone interested at all in discussing library APIs and
>> designs?
>>>>>>> What about internal interfaces and such?
>>>>>>> 
>>>>>>> 
>>>>>>> My plan was to add at least one more data generator (weather
>>>>>>> simulator)
>>>>>> to
>>>>>>> bigtop-data-generators in the short term.  However, given the
>>>>>>> concerns raised by Cos (more discussion needed) and Olaf (don't
>>>>>>> want to force data generators on unsuspecting users ;) ), I would
>>>>>>> like to reach some
>>>>>> consensus
>>>>>>> on what people are concerned about and solutions.
>>>>>>> 
>>>>>>> On Thu, Aug 27, 2015 at 12:38 PM, Konstantin Boudnik
>>>>>>> <co...@apache.org>
>>>>>> wrote:
>>>>>>> 
>>>>>>>> Fine by me. I have linked this thread to the JIRA ticket that RJ
>>>>>> created,
>>>>>>>> so
>>>>>>>> we have a way to connect one to another ;)
>>>>>>>> 
>>>>>>>>> On Thu, Aug 27, 2015 at 01:02PM, Olaf Flebbe wrote:
>>>>>>>>> Hi,
>>>>>>>>> 
>>>>>>>>> I am not confident that moving important design discussions with
>>>>>>>>> impact
>>>>>>>> to
>>>>>>>>> the whole project to jira is a good idea.
>>>>>>>>> 
>>>>>>>>> In the current JIRA Traffic storm it is not easy to identify and
>>>>>>>>> follow
>>>>>>>> important tickets.
>>>>>>>>> 
>>>>>>>>> Please keep discussions on the list or at least, please state on
>>>>>>>>> this
>>>>>>>> list which Ticket to follow ...
>>>>>>>>> 
>>>>>>>>> Olaf
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> Am 26.08.2015 um 22:56 schrieb Konstantin Boudnik <
>> cos@apache.org
>>>> :
>>>>>>>>>> 
>>>>>>>>>> On Wed, Aug 26, 2015 at 10:38PM, Olaf Flebbe wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>> 
>>>>>>>>>>> Nive to have data generators in Bigtop.
>>>>>>>>>>> 
>>>>>>>>>>> But please do not include it in bigtop_utils, since this
>>>>>>>>>>> package is mandatory. Not everyone needs a data generator .
>>>>>>>>>> 
>>>>>>>>>> Yup. And let's move further design discussion to the JIRA!
>>>>>>>>>> 
>>>>>>>>>>> Olaf
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> Am 26.08.2015 um 11:25 schrieb Jay Vyas <
>>>>>> jayunit100.apache@gmail.com
>>>>>>>>> :
>>>>>>>>>>>> 
>>>>>>>>>>>> Publishing the jar to bigtops maven is probably a good first
>>>>>>>>>>>> step
>>>>>>>> ,Then apps can just include it as needed...?.
>>>>>>>>>>>> 
>>>>>>>>>>>> I'm not against packaging if someone wants packages for this.
>>>>>>>>>>>> Maybe
>>>>>>>> even include it in bigtop util ?
>>>>>>>>>>>> 
>>>>>>>>>>>> Let's move to jira,
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Aug 25, 2015, at 9:41 PM, Konstantin Boudnik
>>>>>>>>>>>>> <co...@apache.org>
>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> It is pretty cool indeed!
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I wonder how it needs to be structured to be:
>>>>>>>>>>>>> - easy to access/use from other components wherever it is
>>>>>>>>>>>>> needed
>>>>>>>>>>>>> - doesn't interfere with the rest of the stack
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I guess one possible way would be to implement the generator
>>>>>>>>>>>>> as a
>>>>>>>> set of maven
>>>>>>>>>>>>> artifacts, that could be installed/consumed transparently by
>>>>>>>>>>>>> just
>>>>>>>> declaring a
>>>>>>>>>>>>> dependency e.g as proposed via top-level component.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Another way is to have a new package like we do for
>>>>>>>>>>>>> bigtop-utils
>>>>>>>> and such.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Perhaps this discussion should be moved to JIRA or shall we
>>>>>>>> continue on the
>>>>>>>>>>>>> dev@ ??
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Cos
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sun, Aug 23, 2015 at 11:53AM, RJ Nowling wrote:
>>>>>>>>>>>>>> Hi BigTop,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I had a discussion with Jay yesterday, we'd like to propose
>>>>>>>>>>>>>> a new
>>>>>>>> component
>>>>>>>>>>>>>> for BigTop: BigTop Data Generators.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> BigTop Data Generators would consist of a common set of
>>>>>>>>>>>>>> libraries
>>>>>>>> for
>>>>>>>>>>>>>> building data generators and three example data generators:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> * BigPetStore transaction generator (moved from
>>>>>>>>>>>>>> BigPetStore)
>>>>>>>>>>>>>> * BigTop Bazaar -- attendee movement and interactions with
>>>>>>>>>>>>>> booths
>>>>>>>> on a
>>>>>>>>>>>>>> showroom floor, at a conference, or at a mall
>>>>>>>>>>>>>> * BigTop Weatherman -- stochastic weather simulation
>>>>>>>> (temperature, wind
>>>>>>>>>>>>>> speed, wind chill, rainfall, etc.) per zip code.  (From a
>>>>>>>>>>>>>> model
>>>>>>>> trained on
>>>>>>>>>>>>>> NOAA historical weather data)
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> We believe that creating a common set of libraries will
>>>>>>>>>>>>>> have
>>>>>>>> several
>>>>>>>>>>>>>> benefits including:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> * Easier for others to build their own data generators
>>>>>>>>>>>>>> * Make data generators smaller and easier to maintain
>>>>>>>>>>>>>> * Share improvements across the data generators
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> More details on the libraries are below.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> BigPetStore will be continue to focus on building  and
>>>>>>>>>>>>>> maintaining blueprints, powered by the BigTop Data
>> Generators.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Our vision is that we get all of Apache coming to BigTop
>>>>>>>>>>>>>> for tools
>>>>>>>> for
>>>>>>>>>>>>>> building better, more comprehensive blueprints.  We want to
>>>>>>>> support these
>>>>>>>>>>>>>> efforts through data generators and the initial set of
>>>>>>>>>>>>>> blueprint
>>>>>>>> we've been
>>>>>>>>>>>>>> building.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> If the community is generally in support of this, I can
>>>>>>>>>>>>>> create a
>>>>>>>> top-level
>>>>>>>>>>>>>> "bigtop-data-generators" directory and put the data
>>>>>>>>>>>>>> generators and libraries in there.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> RJ
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> -------
>>>>>>>>>>>>>> Library details:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> So far, I've extracted the following common libraries:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> * Samplers -- provides classes for PDFs and various
>>>>>>>>>>>>>> samplers
>>>>>>>>>>>>>> * Name generator -- data set and samplers for generating
>>>>>>>>>>>>>> names
>>>>>>>>>>>>>> * Location data set -- data set and classes for US zip
>>>>>>>>>>>>>> codes,
>>>>>>>> their
>>>>>>>>>>>>>> GPS coordinates, median house hold incomes, and population
>>>>>>>>>>>>>> sizes
>>>>>>>>>>>>>> * Product generator -- library for enumerating products
>>>>>>>>>>>>>> from a specification file.  Comes with default
>>>>>>>>>>>>>> specifications for
>>>>>>>> BigPetStore
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I also expect that I'll add libraries for:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> * Particle simulation -- customer movement in a room
>>>>>>>>>>>>>> * Latent factor model generation -- generate latent
>>>>>>>>>>>>>> factors and customer weights to create something like
>>> MovieLens data.
>>>>>>>>>>>>>> Used in
>>>>>>>> Bazaar
>>>>>>>>>>>>>> for booth preferences and potentially in BigPetStore for
>>>>>>>>>>>>>> customer
>>>>>>>> item
>>>>>>>>>>>>>> preferences
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Most of these libraries came out of the BigPetStore data
>>>>>>>>>>>>>> generator
>>>>>>>> but the
>>>>>>>>>>>>>> other generators have been refactored to be based off the
>>>>>>>>>>>>>> standard
>>>>>>>> set of
>>>>>>>>>>>>>> libraries.
>>>> 
>>>> 
>>>> --
>>>> jay vyas
>>>> 
>>> 
>> 


Re: Proposal for "BigTop Data Generators"

Posted by Evans Ye <ev...@apache.org>.
I am very much like the shell script wrapper and docker image idea since
that way we can integrate it directly with bigtop provisioner which yield a
perfect ux for the whole things. I think its not too hard to do it both, we
just need to add a parameter to turn the script into daemon mode. I see
lots of image doing this way.

docker run bigtop/bigtop-data-gen --scheme weather --size 5GB --output
data-dir --etc  foo --etc bar --daemon
2015年8月31日 下午9:06於 "RJ Nowling" <rn...@gmail.com>寫道:

> The BigPetStore, Bazaar, and weather data generators have single-threaded
> command-line interfaces.  We could do the same with the smaller generators
> (names, locations, etc.) if there is interest.
>
> On Mon, Aug 31, 2015 at 5:24 AM, Jay Vyas <ja...@gmail.com>
> wrote:
>
> > Nate: Good idea to abstract the interface one level higher....
> >
> > How about a docker run command ? That is probably the easiest way for
> > Linux folks to run one off Java apps nowadays.
> >
> > docker run bigtop/bigtop-data-gen --scheme weather --size 5GB --output
> > data-dir --etc  foo --etc bar
> >
> > I'm happy to curate such a docker image, I already am doing something
> like
> > this in kube for bigtop-transaction-queue, which continuously pumps data
> > generator outputs into a REST endpoint or file
> > Queue... So it could be extended to support other generators.
> >
> >
> > > om> <na...@reactor8.com> wrote:
> > >
> > > Could picture at some point supporting something like this for non-jvm
> > folk just looking for test/demo data:
> > >
> > > apt-get install bigtop-data-gen
> > > ~/ $ bigtop-data-gen --scheme weather --size 5GB --output data-dir
> > --etc  foo --etc bar
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: jay vyas [mailto:jayunit100.apache@gmail.com]
> > > Sent: Sunday, August 30, 2015 5:11 PM
> > > To: dev@bigtop.apache.org
> > > Subject: Re: Proposal for "BigTop Data Generators"
> > >
> > > Hola nate.  Well, here are the Use cases I know of that I have used the
> > data generators for.
> > >
> > > Dockerfile:
> > >
> > > (1) for testing kubernetes.  For this, I just use transaction-queue
> > docker file.
> > > (2) for testing GlusterFS small file workloads, maybe with other
> > analytics tools...
> > >
> > > Maven repo
> > >
> > > (3) Java maprduce/ignite/spark applications, which can just add a mvn
> > repo when compiling.  Java developers never add jars through RPM repos.
> > >
> > > RPM/DEB packages:
> > >
> > > I could see people using an RPM/DEB data generator, and I'm not against
> > it.  But I simply don't know of any real world projects which *currently*
> > need RPM/Deb packages, which is why I haven't bothered to propose it as a
> > requirement.  Nevertheless linux packages are always a welcome addition
> if
> > someone wants to create em !
> > >
> > >
> > >
> > >
> > >> On Sun, Aug 30, 2015 at 4:34 PM, <na...@reactor8.com> wrote:
> > >>
> > >> Would container be in addition to deb/rpm, or instead of?  If latter
> > >> can we do deb/rpm as base then have container either created from them
> > >> or directly from artifacts?
> > >>
> > >> On test usage side, seems could probably break up tests into
> > >> base/required and then optional/add-on tests/test-suites.  Think
> > >> remember seeing mention of certain tests that are failing at times on
> > >> certain component(s) anyways in the core builds but don’t mean that
> > >> the build is broken, so would make sense to have some clean up around
> > those anyways.
> > >>
> > >> -----Original Message-----
> > >> From: RJ Nowling [mailto:rnowling@gmail.com]
> > >> Sent: Sunday, August 30, 2015 1:11 PM
> > >> To: dev@bigtop.apache.org
> > >> Subject: Re: Proposal for "BigTop Data Generators"
> > >>
> > >> I agree with the above. :)
> > >>
> > >> On Sun, Aug 30, 2015 at 11:19 AM, Jay Vyas
> > >> <ja...@gmail.com>
> > >> wrote:
> > >>
> > >>> Hi RJ.
> > >>>
> > >>> Maven repositories and docker containers for the transaction queue
> > >>> are good enough IMO.  That will give people a way to compose them in
> > >>> different idioms (one for Java folks, another for broader Linux
> > >>> audience
> > >> ).
> > >>>
> > >>> I think the lib designs are fairly intuitive.  I would say that we
> > >>> should constrain them all to being written in Java or Groovy to keep
> > >>> the bigtop theme of "JVM for everything" :).
> > >>>
> > >>> Any particular questions you have around technical design can be
> > >>> followed in a JIRA or else maybe a Readme spec that goes in a  top
> > >>> level of the data-generators dir...
> > >>>
> > >>>> On Aug 30, 2015, at 1:51 AM, RJ Nowling <rn...@gmail.com> wrote:
> > >>>>
> > >>>> I'd like to keep this conversation going.
> > >>>>
> > >>>> So here are a few discussion points:
> > >>>>
> > >>>> 1. How do we want to make the data generators available?  Maven?
> > >>>> RPMs
> > >>> and
> > >>>> Debs?
> > >>>>
> > >>>> For now, I'm using a gradle multi-project build to easily build
> > >>>> and
> > >>> install
> > >>>> the BPS data generators and its libraries into a local maven repo.
> > >>>> This makes development easy.  Eventually, I would like to post
> > >>>> binaries
> > >>> through
> > >>>> Maven for easy integration by users.  RPMs / Debs could be
> > >>>> interesting since I use a pattern where the data generators are
> > >>>> libraries (to support application integration / parallelization by
> > >>>> the host framework) but also provide CLI drivers for local testing.
> > >>>>
> > >>>> 2.  The idea of using the data generators as part of the smoke
> > >>>> tests came up.  Since there is concern about making the data
> > >>>> generators required, we could offer the blueprints (BigPetStore)
> > >>>> as optional smoke tests.  Would that be a good compromise?
> > >>>>
> > >>>> 3.  How will they be maintained?
> > >>>>
> > >>>> I'll certainly add myself to the maintainers list and will be
> > >>>> taking responsibility.  I'm happy to have others help as well if
> > >>>> anyone wants to
> > >>>> -- if not, that's cool, too.
> > >>>>
> > >>>> 4. Is anyone interested at all in discussing library APIs and
> designs?
> > >>>> What about internal interfaces and such?
> > >>>>
> > >>>>
> > >>>> My plan was to add at least one more data generator (weather
> > >>>> simulator)
> > >>> to
> > >>>> bigtop-data-generators in the short term.  However, given the
> > >>>> concerns raised by Cos (more discussion needed) and Olaf (don't
> > >>>> want to force data generators on unsuspecting users ;) ), I would
> > >>>> like to reach some
> > >>> consensus
> > >>>> on what people are concerned about and solutions.
> > >>>>
> > >>>> On Thu, Aug 27, 2015 at 12:38 PM, Konstantin Boudnik
> > >>>> <co...@apache.org>
> > >>> wrote:
> > >>>>
> > >>>>> Fine by me. I have linked this thread to the JIRA ticket that RJ
> > >>> created,
> > >>>>> so
> > >>>>> we have a way to connect one to another ;)
> > >>>>>
> > >>>>>> On Thu, Aug 27, 2015 at 01:02PM, Olaf Flebbe wrote:
> > >>>>>> Hi,
> > >>>>>>
> > >>>>>> I am not confident that moving important design discussions with
> > >>>>>> impact
> > >>>>> to
> > >>>>>> the whole project to jira is a good idea.
> > >>>>>>
> > >>>>>> In the current JIRA Traffic storm it is not easy to identify and
> > >>>>>> follow
> > >>>>> important tickets.
> > >>>>>>
> > >>>>>> Please keep discussions on the list or at least, please state on
> > >>>>>> this
> > >>>>> list which Ticket to follow ...
> > >>>>>>
> > >>>>>> Olaf
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>> Am 26.08.2015 um 22:56 schrieb Konstantin Boudnik <
> cos@apache.org
> > >:
> > >>>>>>>
> > >>>>>>> On Wed, Aug 26, 2015 at 10:38PM, Olaf Flebbe wrote:
> > >>>>>>>> Hi,
> > >>>>>>>>
> > >>>>>>>> Nive to have data generators in Bigtop.
> > >>>>>>>>
> > >>>>>>>> But please do not include it in bigtop_utils, since this
> > >>>>>>>> package is mandatory. Not everyone needs a data generator .
> > >>>>>>>
> > >>>>>>> Yup. And let's move further design discussion to the JIRA!
> > >>>>>>>
> > >>>>>>>> Olaf
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> Am 26.08.2015 um 11:25 schrieb Jay Vyas <
> > >>> jayunit100.apache@gmail.com
> > >>>>>> :
> > >>>>>>>>>
> > >>>>>>>>> Publishing the jar to bigtops maven is probably a good first
> > >>>>>>>>> step
> > >>>>> ,Then apps can just include it as needed...?.
> > >>>>>>>>>
> > >>>>>>>>> I'm not against packaging if someone wants packages for this.
> > >>>>>>>>> Maybe
> > >>>>> even include it in bigtop util ?
> > >>>>>>>>>
> > >>>>>>>>> Let's move to jira,
> > >>>>>>>>>
> > >>>>>>>>>> On Aug 25, 2015, at 9:41 PM, Konstantin Boudnik
> > >>>>>>>>>> <co...@apache.org>
> > >>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> It is pretty cool indeed!
> > >>>>>>>>>>
> > >>>>>>>>>> I wonder how it needs to be structured to be:
> > >>>>>>>>>> - easy to access/use from other components wherever it is
> > >>>>>>>>>> needed
> > >>>>>>>>>> - doesn't interfere with the rest of the stack
> > >>>>>>>>>>
> > >>>>>>>>>> I guess one possible way would be to implement the generator
> > >>>>>>>>>> as a
> > >>>>> set of maven
> > >>>>>>>>>> artifacts, that could be installed/consumed transparently by
> > >>>>>>>>>> just
> > >>>>> declaring a
> > >>>>>>>>>> dependency e.g as proposed via top-level component.
> > >>>>>>>>>>
> > >>>>>>>>>> Another way is to have a new package like we do for
> > >>>>>>>>>> bigtop-utils
> > >>>>> and such.
> > >>>>>>>>>>
> > >>>>>>>>>> Perhaps this discussion should be moved to JIRA or shall we
> > >>>>> continue on the
> > >>>>>>>>>> dev@ ??
> > >>>>>>>>>>
> > >>>>>>>>>> Cos
> > >>>>>>>>>>
> > >>>>>>>>>>> On Sun, Aug 23, 2015 at 11:53AM, RJ Nowling wrote:
> > >>>>>>>>>>> Hi BigTop,
> > >>>>>>>>>>>
> > >>>>>>>>>>> I had a discussion with Jay yesterday, we'd like to propose
> > >>>>>>>>>>> a new
> > >>>>> component
> > >>>>>>>>>>> for BigTop: BigTop Data Generators.
> > >>>>>>>>>>>
> > >>>>>>>>>>> BigTop Data Generators would consist of a common set of
> > >>>>>>>>>>> libraries
> > >>>>> for
> > >>>>>>>>>>> building data generators and three example data generators:
> > >>>>>>>>>>>
> > >>>>>>>>>>> * BigPetStore transaction generator (moved from
> > >>>>>>>>>>> BigPetStore)
> > >>>>>>>>>>> * BigTop Bazaar -- attendee movement and interactions with
> > >>>>>>>>>>> booths
> > >>>>> on a
> > >>>>>>>>>>> showroom floor, at a conference, or at a mall
> > >>>>>>>>>>> * BigTop Weatherman -- stochastic weather simulation
> > >>>>> (temperature, wind
> > >>>>>>>>>>> speed, wind chill, rainfall, etc.) per zip code.  (From a
> > >>>>>>>>>>> model
> > >>>>> trained on
> > >>>>>>>>>>> NOAA historical weather data)
> > >>>>>>>>>>>
> > >>>>>>>>>>> We believe that creating a common set of libraries will
> > >>>>>>>>>>> have
> > >>>>> several
> > >>>>>>>>>>> benefits including:
> > >>>>>>>>>>>
> > >>>>>>>>>>> * Easier for others to build their own data generators
> > >>>>>>>>>>> * Make data generators smaller and easier to maintain
> > >>>>>>>>>>> * Share improvements across the data generators
> > >>>>>>>>>>>
> > >>>>>>>>>>> More details on the libraries are below.
> > >>>>>>>>>>>
> > >>>>>>>>>>> BigPetStore will be continue to focus on building  and
> > >>>>>>>>>>> maintaining blueprints, powered by the BigTop Data
> Generators.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Our vision is that we get all of Apache coming to BigTop
> > >>>>>>>>>>> for tools
> > >>>>> for
> > >>>>>>>>>>> building better, more comprehensive blueprints.  We want to
> > >>>>> support these
> > >>>>>>>>>>> efforts through data generators and the initial set of
> > >>>>>>>>>>> blueprint
> > >>>>> we've been
> > >>>>>>>>>>> building.
> > >>>>>>>>>>>
> > >>>>>>>>>>> If the community is generally in support of this, I can
> > >>>>>>>>>>> create a
> > >>>>> top-level
> > >>>>>>>>>>> "bigtop-data-generators" directory and put the data
> > >>>>>>>>>>> generators and libraries in there.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks!
> > >>>>>>>>>>>
> > >>>>>>>>>>> RJ
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> -------
> > >>>>>>>>>>> Library details:
> > >>>>>>>>>>>
> > >>>>>>>>>>> So far, I've extracted the following common libraries:
> > >>>>>>>>>>>
> > >>>>>>>>>>> * Samplers -- provides classes for PDFs and various
> > >>>>>>>>>>> samplers
> > >>>>>>>>>>> * Name generator -- data set and samplers for generating
> > >>>>>>>>>>> names
> > >>>>>>>>>>> * Location data set -- data set and classes for US zip
> > >>>>>>>>>>> codes,
> > >>>>> their
> > >>>>>>>>>>> GPS coordinates, median house hold incomes, and population
> > >>>>>>>>>>> sizes
> > >>>>>>>>>>> * Product generator -- library for enumerating products
> > >>>>>>>>>>> from a specification file.  Comes with default
> > >>>>>>>>>>> specifications for
> > >>>>> BigPetStore
> > >>>>>>>>>>>
> > >>>>>>>>>>> I also expect that I'll add libraries for:
> > >>>>>>>>>>>
> > >>>>>>>>>>>  * Particle simulation -- customer movement in a room
> > >>>>>>>>>>>  * Latent factor model generation -- generate latent
> > >>>>>>>>>>> factors and customer weights to create something like
> > MovieLens data.
> > >>>>>>>>>>> Used in
> > >>>>> Bazaar
> > >>>>>>>>>>> for booth preferences and potentially in BigPetStore for
> > >>>>>>>>>>> customer
> > >>>>> item
> > >>>>>>>>>>> preferences
> > >>>>>>>>>>>
> > >>>>>>>>>>> Most of these libraries came out of the BigPetStore data
> > >>>>>>>>>>> generator
> > >>>>> but the
> > >>>>>>>>>>> other generators have been refactored to be based off the
> > >>>>>>>>>>> standard
> > >>>>> set of
> > >>>>>>>>>>> libraries.
> > >
> > >
> > > --
> > > jay vyas
> > >
> >
>

Re: Proposal for "BigTop Data Generators"

Posted by RJ Nowling <rn...@gmail.com>.
The BigPetStore, Bazaar, and weather data generators have single-threaded
command-line interfaces.  We could do the same with the smaller generators
(names, locations, etc.) if there is interest.

On Mon, Aug 31, 2015 at 5:24 AM, Jay Vyas <ja...@gmail.com>
wrote:

> Nate: Good idea to abstract the interface one level higher....
>
> How about a docker run command ? That is probably the easiest way for
> Linux folks to run one off Java apps nowadays.
>
> docker run bigtop/bigtop-data-gen --scheme weather --size 5GB --output
> data-dir --etc  foo --etc bar
>
> I'm happy to curate such a docker image, I already am doing something like
> this in kube for bigtop-transaction-queue, which continuously pumps data
> generator outputs into a REST endpoint or file
> Queue... So it could be extended to support other generators.
>
>
> > om> <na...@reactor8.com> wrote:
> >
> > Could picture at some point supporting something like this for non-jvm
> folk just looking for test/demo data:
> >
> > apt-get install bigtop-data-gen
> > ~/ $ bigtop-data-gen --scheme weather --size 5GB --output data-dir
> --etc  foo --etc bar
> >
> >
> >
> > -----Original Message-----
> > From: jay vyas [mailto:jayunit100.apache@gmail.com]
> > Sent: Sunday, August 30, 2015 5:11 PM
> > To: dev@bigtop.apache.org
> > Subject: Re: Proposal for "BigTop Data Generators"
> >
> > Hola nate.  Well, here are the Use cases I know of that I have used the
> data generators for.
> >
> > Dockerfile:
> >
> > (1) for testing kubernetes.  For this, I just use transaction-queue
> docker file.
> > (2) for testing GlusterFS small file workloads, maybe with other
> analytics tools...
> >
> > Maven repo
> >
> > (3) Java maprduce/ignite/spark applications, which can just add a mvn
> repo when compiling.  Java developers never add jars through RPM repos.
> >
> > RPM/DEB packages:
> >
> > I could see people using an RPM/DEB data generator, and I'm not against
> it.  But I simply don't know of any real world projects which *currently*
> need RPM/Deb packages, which is why I haven't bothered to propose it as a
> requirement.  Nevertheless linux packages are always a welcome addition if
> someone wants to create em !
> >
> >
> >
> >
> >> On Sun, Aug 30, 2015 at 4:34 PM, <na...@reactor8.com> wrote:
> >>
> >> Would container be in addition to deb/rpm, or instead of?  If latter
> >> can we do deb/rpm as base then have container either created from them
> >> or directly from artifacts?
> >>
> >> On test usage side, seems could probably break up tests into
> >> base/required and then optional/add-on tests/test-suites.  Think
> >> remember seeing mention of certain tests that are failing at times on
> >> certain component(s) anyways in the core builds but don’t mean that
> >> the build is broken, so would make sense to have some clean up around
> those anyways.
> >>
> >> -----Original Message-----
> >> From: RJ Nowling [mailto:rnowling@gmail.com]
> >> Sent: Sunday, August 30, 2015 1:11 PM
> >> To: dev@bigtop.apache.org
> >> Subject: Re: Proposal for "BigTop Data Generators"
> >>
> >> I agree with the above. :)
> >>
> >> On Sun, Aug 30, 2015 at 11:19 AM, Jay Vyas
> >> <ja...@gmail.com>
> >> wrote:
> >>
> >>> Hi RJ.
> >>>
> >>> Maven repositories and docker containers for the transaction queue
> >>> are good enough IMO.  That will give people a way to compose them in
> >>> different idioms (one for Java folks, another for broader Linux
> >>> audience
> >> ).
> >>>
> >>> I think the lib designs are fairly intuitive.  I would say that we
> >>> should constrain them all to being written in Java or Groovy to keep
> >>> the bigtop theme of "JVM for everything" :).
> >>>
> >>> Any particular questions you have around technical design can be
> >>> followed in a JIRA or else maybe a Readme spec that goes in a  top
> >>> level of the data-generators dir...
> >>>
> >>>> On Aug 30, 2015, at 1:51 AM, RJ Nowling <rn...@gmail.com> wrote:
> >>>>
> >>>> I'd like to keep this conversation going.
> >>>>
> >>>> So here are a few discussion points:
> >>>>
> >>>> 1. How do we want to make the data generators available?  Maven?
> >>>> RPMs
> >>> and
> >>>> Debs?
> >>>>
> >>>> For now, I'm using a gradle multi-project build to easily build
> >>>> and
> >>> install
> >>>> the BPS data generators and its libraries into a local maven repo.
> >>>> This makes development easy.  Eventually, I would like to post
> >>>> binaries
> >>> through
> >>>> Maven for easy integration by users.  RPMs / Debs could be
> >>>> interesting since I use a pattern where the data generators are
> >>>> libraries (to support application integration / parallelization by
> >>>> the host framework) but also provide CLI drivers for local testing.
> >>>>
> >>>> 2.  The idea of using the data generators as part of the smoke
> >>>> tests came up.  Since there is concern about making the data
> >>>> generators required, we could offer the blueprints (BigPetStore)
> >>>> as optional smoke tests.  Would that be a good compromise?
> >>>>
> >>>> 3.  How will they be maintained?
> >>>>
> >>>> I'll certainly add myself to the maintainers list and will be
> >>>> taking responsibility.  I'm happy to have others help as well if
> >>>> anyone wants to
> >>>> -- if not, that's cool, too.
> >>>>
> >>>> 4. Is anyone interested at all in discussing library APIs and designs?
> >>>> What about internal interfaces and such?
> >>>>
> >>>>
> >>>> My plan was to add at least one more data generator (weather
> >>>> simulator)
> >>> to
> >>>> bigtop-data-generators in the short term.  However, given the
> >>>> concerns raised by Cos (more discussion needed) and Olaf (don't
> >>>> want to force data generators on unsuspecting users ;) ), I would
> >>>> like to reach some
> >>> consensus
> >>>> on what people are concerned about and solutions.
> >>>>
> >>>> On Thu, Aug 27, 2015 at 12:38 PM, Konstantin Boudnik
> >>>> <co...@apache.org>
> >>> wrote:
> >>>>
> >>>>> Fine by me. I have linked this thread to the JIRA ticket that RJ
> >>> created,
> >>>>> so
> >>>>> we have a way to connect one to another ;)
> >>>>>
> >>>>>> On Thu, Aug 27, 2015 at 01:02PM, Olaf Flebbe wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> I am not confident that moving important design discussions with
> >>>>>> impact
> >>>>> to
> >>>>>> the whole project to jira is a good idea.
> >>>>>>
> >>>>>> In the current JIRA Traffic storm it is not easy to identify and
> >>>>>> follow
> >>>>> important tickets.
> >>>>>>
> >>>>>> Please keep discussions on the list or at least, please state on
> >>>>>> this
> >>>>> list which Ticket to follow ...
> >>>>>>
> >>>>>> Olaf
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> Am 26.08.2015 um 22:56 schrieb Konstantin Boudnik <cos@apache.org
> >:
> >>>>>>>
> >>>>>>> On Wed, Aug 26, 2015 at 10:38PM, Olaf Flebbe wrote:
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> Nive to have data generators in Bigtop.
> >>>>>>>>
> >>>>>>>> But please do not include it in bigtop_utils, since this
> >>>>>>>> package is mandatory. Not everyone needs a data generator .
> >>>>>>>
> >>>>>>> Yup. And let's move further design discussion to the JIRA!
> >>>>>>>
> >>>>>>>> Olaf
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> Am 26.08.2015 um 11:25 schrieb Jay Vyas <
> >>> jayunit100.apache@gmail.com
> >>>>>> :
> >>>>>>>>>
> >>>>>>>>> Publishing the jar to bigtops maven is probably a good first
> >>>>>>>>> step
> >>>>> ,Then apps can just include it as needed...?.
> >>>>>>>>>
> >>>>>>>>> I'm not against packaging if someone wants packages for this.
> >>>>>>>>> Maybe
> >>>>> even include it in bigtop util ?
> >>>>>>>>>
> >>>>>>>>> Let's move to jira,
> >>>>>>>>>
> >>>>>>>>>> On Aug 25, 2015, at 9:41 PM, Konstantin Boudnik
> >>>>>>>>>> <co...@apache.org>
> >>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> It is pretty cool indeed!
> >>>>>>>>>>
> >>>>>>>>>> I wonder how it needs to be structured to be:
> >>>>>>>>>> - easy to access/use from other components wherever it is
> >>>>>>>>>> needed
> >>>>>>>>>> - doesn't interfere with the rest of the stack
> >>>>>>>>>>
> >>>>>>>>>> I guess one possible way would be to implement the generator
> >>>>>>>>>> as a
> >>>>> set of maven
> >>>>>>>>>> artifacts, that could be installed/consumed transparently by
> >>>>>>>>>> just
> >>>>> declaring a
> >>>>>>>>>> dependency e.g as proposed via top-level component.
> >>>>>>>>>>
> >>>>>>>>>> Another way is to have a new package like we do for
> >>>>>>>>>> bigtop-utils
> >>>>> and such.
> >>>>>>>>>>
> >>>>>>>>>> Perhaps this discussion should be moved to JIRA or shall we
> >>>>> continue on the
> >>>>>>>>>> dev@ ??
> >>>>>>>>>>
> >>>>>>>>>> Cos
> >>>>>>>>>>
> >>>>>>>>>>> On Sun, Aug 23, 2015 at 11:53AM, RJ Nowling wrote:
> >>>>>>>>>>> Hi BigTop,
> >>>>>>>>>>>
> >>>>>>>>>>> I had a discussion with Jay yesterday, we'd like to propose
> >>>>>>>>>>> a new
> >>>>> component
> >>>>>>>>>>> for BigTop: BigTop Data Generators.
> >>>>>>>>>>>
> >>>>>>>>>>> BigTop Data Generators would consist of a common set of
> >>>>>>>>>>> libraries
> >>>>> for
> >>>>>>>>>>> building data generators and three example data generators:
> >>>>>>>>>>>
> >>>>>>>>>>> * BigPetStore transaction generator (moved from
> >>>>>>>>>>> BigPetStore)
> >>>>>>>>>>> * BigTop Bazaar -- attendee movement and interactions with
> >>>>>>>>>>> booths
> >>>>> on a
> >>>>>>>>>>> showroom floor, at a conference, or at a mall
> >>>>>>>>>>> * BigTop Weatherman -- stochastic weather simulation
> >>>>> (temperature, wind
> >>>>>>>>>>> speed, wind chill, rainfall, etc.) per zip code.  (From a
> >>>>>>>>>>> model
> >>>>> trained on
> >>>>>>>>>>> NOAA historical weather data)
> >>>>>>>>>>>
> >>>>>>>>>>> We believe that creating a common set of libraries will
> >>>>>>>>>>> have
> >>>>> several
> >>>>>>>>>>> benefits including:
> >>>>>>>>>>>
> >>>>>>>>>>> * Easier for others to build their own data generators
> >>>>>>>>>>> * Make data generators smaller and easier to maintain
> >>>>>>>>>>> * Share improvements across the data generators
> >>>>>>>>>>>
> >>>>>>>>>>> More details on the libraries are below.
> >>>>>>>>>>>
> >>>>>>>>>>> BigPetStore will be continue to focus on building  and
> >>>>>>>>>>> maintaining blueprints, powered by the BigTop Data Generators.
> >>>>>>>>>>>
> >>>>>>>>>>> Our vision is that we get all of Apache coming to BigTop
> >>>>>>>>>>> for tools
> >>>>> for
> >>>>>>>>>>> building better, more comprehensive blueprints.  We want to
> >>>>> support these
> >>>>>>>>>>> efforts through data generators and the initial set of
> >>>>>>>>>>> blueprint
> >>>>> we've been
> >>>>>>>>>>> building.
> >>>>>>>>>>>
> >>>>>>>>>>> If the community is generally in support of this, I can
> >>>>>>>>>>> create a
> >>>>> top-level
> >>>>>>>>>>> "bigtop-data-generators" directory and put the data
> >>>>>>>>>>> generators and libraries in there.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks!
> >>>>>>>>>>>
> >>>>>>>>>>> RJ
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> -------
> >>>>>>>>>>> Library details:
> >>>>>>>>>>>
> >>>>>>>>>>> So far, I've extracted the following common libraries:
> >>>>>>>>>>>
> >>>>>>>>>>> * Samplers -- provides classes for PDFs and various
> >>>>>>>>>>> samplers
> >>>>>>>>>>> * Name generator -- data set and samplers for generating
> >>>>>>>>>>> names
> >>>>>>>>>>> * Location data set -- data set and classes for US zip
> >>>>>>>>>>> codes,
> >>>>> their
> >>>>>>>>>>> GPS coordinates, median house hold incomes, and population
> >>>>>>>>>>> sizes
> >>>>>>>>>>> * Product generator -- library for enumerating products
> >>>>>>>>>>> from a specification file.  Comes with default
> >>>>>>>>>>> specifications for
> >>>>> BigPetStore
> >>>>>>>>>>>
> >>>>>>>>>>> I also expect that I'll add libraries for:
> >>>>>>>>>>>
> >>>>>>>>>>>  * Particle simulation -- customer movement in a room
> >>>>>>>>>>>  * Latent factor model generation -- generate latent
> >>>>>>>>>>> factors and customer weights to create something like
> MovieLens data.
> >>>>>>>>>>> Used in
> >>>>> Bazaar
> >>>>>>>>>>> for booth preferences and potentially in BigPetStore for
> >>>>>>>>>>> customer
> >>>>> item
> >>>>>>>>>>> preferences
> >>>>>>>>>>>
> >>>>>>>>>>> Most of these libraries came out of the BigPetStore data
> >>>>>>>>>>> generator
> >>>>> but the
> >>>>>>>>>>> other generators have been refactored to be based off the
> >>>>>>>>>>> standard
> >>>>> set of
> >>>>>>>>>>> libraries.
> >
> >
> > --
> > jay vyas
> >
>

Re: Proposal for "BigTop Data Generators"

Posted by Jay Vyas <ja...@gmail.com>.
Nate: Good idea to abstract the interface one level higher....

How about a docker run command ? That is probably the easiest way for Linux folks to run one off Java apps nowadays.  

docker run bigtop/bigtop-data-gen --scheme weather --size 5GB --output data-dir --etc  foo --etc bar

I'm happy to curate such a docker image, I already am doing something like this in kube for bigtop-transaction-queue, which continuously pumps data generator outputs into a REST endpoint or file
Queue... So it could be extended to support other generators.


> om> <na...@reactor8.com> wrote:
> 
> Could picture at some point supporting something like this for non-jvm folk just looking for test/demo data:
> 
> apt-get install bigtop-data-gen
> ~/ $ bigtop-data-gen --scheme weather --size 5GB --output data-dir --etc  foo --etc bar
> 
> 
> 
> -----Original Message-----
> From: jay vyas [mailto:jayunit100.apache@gmail.com] 
> Sent: Sunday, August 30, 2015 5:11 PM
> To: dev@bigtop.apache.org
> Subject: Re: Proposal for "BigTop Data Generators"
> 
> Hola nate.  Well, here are the Use cases I know of that I have used the data generators for.
> 
> Dockerfile:
> 
> (1) for testing kubernetes.  For this, I just use transaction-queue docker file.
> (2) for testing GlusterFS small file workloads, maybe with other analytics tools...
> 
> Maven repo
> 
> (3) Java maprduce/ignite/spark applications, which can just add a mvn repo when compiling.  Java developers never add jars through RPM repos.
> 
> RPM/DEB packages:
> 
> I could see people using an RPM/DEB data generator, and I'm not against it.  But I simply don't know of any real world projects which *currently* need RPM/Deb packages, which is why I haven't bothered to propose it as a requirement.  Nevertheless linux packages are always a welcome addition if  someone wants to create em !
> 
> 
> 
> 
>> On Sun, Aug 30, 2015 at 4:34 PM, <na...@reactor8.com> wrote:
>> 
>> Would container be in addition to deb/rpm, or instead of?  If latter 
>> can we do deb/rpm as base then have container either created from them 
>> or directly from artifacts?
>> 
>> On test usage side, seems could probably break up tests into 
>> base/required and then optional/add-on tests/test-suites.  Think 
>> remember seeing mention of certain tests that are failing at times on 
>> certain component(s) anyways in the core builds but don’t mean that 
>> the build is broken, so would make sense to have some clean up around those anyways.
>> 
>> -----Original Message-----
>> From: RJ Nowling [mailto:rnowling@gmail.com]
>> Sent: Sunday, August 30, 2015 1:11 PM
>> To: dev@bigtop.apache.org
>> Subject: Re: Proposal for "BigTop Data Generators"
>> 
>> I agree with the above. :)
>> 
>> On Sun, Aug 30, 2015 at 11:19 AM, Jay Vyas 
>> <ja...@gmail.com>
>> wrote:
>> 
>>> Hi RJ.
>>> 
>>> Maven repositories and docker containers for the transaction queue 
>>> are good enough IMO.  That will give people a way to compose them in 
>>> different idioms (one for Java folks, another for broader Linux 
>>> audience
>> ).
>>> 
>>> I think the lib designs are fairly intuitive.  I would say that we 
>>> should constrain them all to being written in Java or Groovy to keep 
>>> the bigtop theme of "JVM for everything" :).
>>> 
>>> Any particular questions you have around technical design can be 
>>> followed in a JIRA or else maybe a Readme spec that goes in a  top 
>>> level of the data-generators dir...
>>> 
>>>> On Aug 30, 2015, at 1:51 AM, RJ Nowling <rn...@gmail.com> wrote:
>>>> 
>>>> I'd like to keep this conversation going.
>>>> 
>>>> So here are a few discussion points:
>>>> 
>>>> 1. How do we want to make the data generators available?  Maven?
>>>> RPMs
>>> and
>>>> Debs?
>>>> 
>>>> For now, I'm using a gradle multi-project build to easily build 
>>>> and
>>> install
>>>> the BPS data generators and its libraries into a local maven repo.
>>>> This makes development easy.  Eventually, I would like to post 
>>>> binaries
>>> through
>>>> Maven for easy integration by users.  RPMs / Debs could be 
>>>> interesting since I use a pattern where the data generators are 
>>>> libraries (to support application integration / parallelization by 
>>>> the host framework) but also provide CLI drivers for local testing.
>>>> 
>>>> 2.  The idea of using the data generators as part of the smoke 
>>>> tests came up.  Since there is concern about making the data 
>>>> generators required, we could offer the blueprints (BigPetStore) 
>>>> as optional smoke tests.  Would that be a good compromise?
>>>> 
>>>> 3.  How will they be maintained?
>>>> 
>>>> I'll certainly add myself to the maintainers list and will be 
>>>> taking responsibility.  I'm happy to have others help as well if 
>>>> anyone wants to
>>>> -- if not, that's cool, too.
>>>> 
>>>> 4. Is anyone interested at all in discussing library APIs and designs?
>>>> What about internal interfaces and such?
>>>> 
>>>> 
>>>> My plan was to add at least one more data generator (weather
>>>> simulator)
>>> to
>>>> bigtop-data-generators in the short term.  However, given the 
>>>> concerns raised by Cos (more discussion needed) and Olaf (don't 
>>>> want to force data generators on unsuspecting users ;) ), I would 
>>>> like to reach some
>>> consensus
>>>> on what people are concerned about and solutions.
>>>> 
>>>> On Thu, Aug 27, 2015 at 12:38 PM, Konstantin Boudnik 
>>>> <co...@apache.org>
>>> wrote:
>>>> 
>>>>> Fine by me. I have linked this thread to the JIRA ticket that RJ
>>> created,
>>>>> so
>>>>> we have a way to connect one to another ;)
>>>>> 
>>>>>> On Thu, Aug 27, 2015 at 01:02PM, Olaf Flebbe wrote:
>>>>>> Hi,
>>>>>> 
>>>>>> I am not confident that moving important design discussions with 
>>>>>> impact
>>>>> to
>>>>>> the whole project to jira is a good idea.
>>>>>> 
>>>>>> In the current JIRA Traffic storm it is not easy to identify and 
>>>>>> follow
>>>>> important tickets.
>>>>>> 
>>>>>> Please keep discussions on the list or at least, please state on 
>>>>>> this
>>>>> list which Ticket to follow ...
>>>>>> 
>>>>>> Olaf
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> Am 26.08.2015 um 22:56 schrieb Konstantin Boudnik <co...@apache.org>:
>>>>>>> 
>>>>>>> On Wed, Aug 26, 2015 at 10:38PM, Olaf Flebbe wrote:
>>>>>>>> Hi,
>>>>>>>> 
>>>>>>>> Nive to have data generators in Bigtop.
>>>>>>>> 
>>>>>>>> But please do not include it in bigtop_utils, since this 
>>>>>>>> package is mandatory. Not everyone needs a data generator .
>>>>>>> 
>>>>>>> Yup. And let's move further design discussion to the JIRA!
>>>>>>> 
>>>>>>>> Olaf
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> Am 26.08.2015 um 11:25 schrieb Jay Vyas <
>>> jayunit100.apache@gmail.com
>>>>>> :
>>>>>>>>> 
>>>>>>>>> Publishing the jar to bigtops maven is probably a good first 
>>>>>>>>> step
>>>>> ,Then apps can just include it as needed...?.
>>>>>>>>> 
>>>>>>>>> I'm not against packaging if someone wants packages for this.
>>>>>>>>> Maybe
>>>>> even include it in bigtop util ?
>>>>>>>>> 
>>>>>>>>> Let's move to jira,
>>>>>>>>> 
>>>>>>>>>> On Aug 25, 2015, at 9:41 PM, Konstantin Boudnik 
>>>>>>>>>> <co...@apache.org>
>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> It is pretty cool indeed!
>>>>>>>>>> 
>>>>>>>>>> I wonder how it needs to be structured to be:
>>>>>>>>>> - easy to access/use from other components wherever it is 
>>>>>>>>>> needed
>>>>>>>>>> - doesn't interfere with the rest of the stack
>>>>>>>>>> 
>>>>>>>>>> I guess one possible way would be to implement the generator 
>>>>>>>>>> as a
>>>>> set of maven
>>>>>>>>>> artifacts, that could be installed/consumed transparently by 
>>>>>>>>>> just
>>>>> declaring a
>>>>>>>>>> dependency e.g as proposed via top-level component.
>>>>>>>>>> 
>>>>>>>>>> Another way is to have a new package like we do for 
>>>>>>>>>> bigtop-utils
>>>>> and such.
>>>>>>>>>> 
>>>>>>>>>> Perhaps this discussion should be moved to JIRA or shall we
>>>>> continue on the
>>>>>>>>>> dev@ ??
>>>>>>>>>> 
>>>>>>>>>> Cos
>>>>>>>>>> 
>>>>>>>>>>> On Sun, Aug 23, 2015 at 11:53AM, RJ Nowling wrote:
>>>>>>>>>>> Hi BigTop,
>>>>>>>>>>> 
>>>>>>>>>>> I had a discussion with Jay yesterday, we'd like to propose 
>>>>>>>>>>> a new
>>>>> component
>>>>>>>>>>> for BigTop: BigTop Data Generators.
>>>>>>>>>>> 
>>>>>>>>>>> BigTop Data Generators would consist of a common set of 
>>>>>>>>>>> libraries
>>>>> for
>>>>>>>>>>> building data generators and three example data generators:
>>>>>>>>>>> 
>>>>>>>>>>> * BigPetStore transaction generator (moved from 
>>>>>>>>>>> BigPetStore)
>>>>>>>>>>> * BigTop Bazaar -- attendee movement and interactions with 
>>>>>>>>>>> booths
>>>>> on a
>>>>>>>>>>> showroom floor, at a conference, or at a mall
>>>>>>>>>>> * BigTop Weatherman -- stochastic weather simulation
>>>>> (temperature, wind
>>>>>>>>>>> speed, wind chill, rainfall, etc.) per zip code.  (From a 
>>>>>>>>>>> model
>>>>> trained on
>>>>>>>>>>> NOAA historical weather data)
>>>>>>>>>>> 
>>>>>>>>>>> We believe that creating a common set of libraries will 
>>>>>>>>>>> have
>>>>> several
>>>>>>>>>>> benefits including:
>>>>>>>>>>> 
>>>>>>>>>>> * Easier for others to build their own data generators
>>>>>>>>>>> * Make data generators smaller and easier to maintain
>>>>>>>>>>> * Share improvements across the data generators
>>>>>>>>>>> 
>>>>>>>>>>> More details on the libraries are below.
>>>>>>>>>>> 
>>>>>>>>>>> BigPetStore will be continue to focus on building  and 
>>>>>>>>>>> maintaining blueprints, powered by the BigTop Data Generators.
>>>>>>>>>>> 
>>>>>>>>>>> Our vision is that we get all of Apache coming to BigTop 
>>>>>>>>>>> for tools
>>>>> for
>>>>>>>>>>> building better, more comprehensive blueprints.  We want to
>>>>> support these
>>>>>>>>>>> efforts through data generators and the initial set of 
>>>>>>>>>>> blueprint
>>>>> we've been
>>>>>>>>>>> building.
>>>>>>>>>>> 
>>>>>>>>>>> If the community is generally in support of this, I can 
>>>>>>>>>>> create a
>>>>> top-level
>>>>>>>>>>> "bigtop-data-generators" directory and put the data 
>>>>>>>>>>> generators and libraries in there.
>>>>>>>>>>> 
>>>>>>>>>>> Thanks!
>>>>>>>>>>> 
>>>>>>>>>>> RJ
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> -------
>>>>>>>>>>> Library details:
>>>>>>>>>>> 
>>>>>>>>>>> So far, I've extracted the following common libraries:
>>>>>>>>>>> 
>>>>>>>>>>> * Samplers -- provides classes for PDFs and various 
>>>>>>>>>>> samplers
>>>>>>>>>>> * Name generator -- data set and samplers for generating 
>>>>>>>>>>> names
>>>>>>>>>>> * Location data set -- data set and classes for US zip 
>>>>>>>>>>> codes,
>>>>> their
>>>>>>>>>>> GPS coordinates, median house hold incomes, and population 
>>>>>>>>>>> sizes
>>>>>>>>>>> * Product generator -- library for enumerating products 
>>>>>>>>>>> from a specification file.  Comes with default 
>>>>>>>>>>> specifications for
>>>>> BigPetStore
>>>>>>>>>>> 
>>>>>>>>>>> I also expect that I'll add libraries for:
>>>>>>>>>>> 
>>>>>>>>>>>  * Particle simulation -- customer movement in a room
>>>>>>>>>>>  * Latent factor model generation -- generate latent 
>>>>>>>>>>> factors and customer weights to create something like MovieLens data.
>>>>>>>>>>> Used in
>>>>> Bazaar
>>>>>>>>>>> for booth preferences and potentially in BigPetStore for 
>>>>>>>>>>> customer
>>>>> item
>>>>>>>>>>> preferences
>>>>>>>>>>> 
>>>>>>>>>>> Most of these libraries came out of the BigPetStore data 
>>>>>>>>>>> generator
>>>>> but the
>>>>>>>>>>> other generators have been refactored to be based off the 
>>>>>>>>>>> standard
>>>>> set of
>>>>>>>>>>> libraries.
> 
> 
> --
> jay vyas
> 

RE: Proposal for "BigTop Data Generators"

Posted by na...@reactor8.com.
Could picture at some point supporting something like this for non-jvm folk just looking for test/demo data:

apt-get install bigtop-data-gen
~/ $ bigtop-data-gen --scheme weather --size 5GB --output data-dir --etc  foo --etc bar



-----Original Message-----
From: jay vyas [mailto:jayunit100.apache@gmail.com] 
Sent: Sunday, August 30, 2015 5:11 PM
To: dev@bigtop.apache.org
Subject: Re: Proposal for "BigTop Data Generators"

Hola nate.  Well, here are the Use cases I know of that I have used the data generators for.

Dockerfile:

(1) for testing kubernetes.  For this, I just use transaction-queue docker file.
(2) for testing GlusterFS small file workloads, maybe with other analytics tools...

Maven repo

(3) Java maprduce/ignite/spark applications, which can just add a mvn repo when compiling.  Java developers never add jars through RPM repos.

RPM/DEB packages:

I could see people using an RPM/DEB data generator, and I'm not against it.  But I simply don't know of any real world projects which *currently* need RPM/Deb packages, which is why I haven't bothered to propose it as a requirement.  Nevertheless linux packages are always a welcome addition if  someone wants to create em !




On Sun, Aug 30, 2015 at 4:34 PM, <na...@reactor8.com> wrote:

> Would container be in addition to deb/rpm, or instead of?  If latter 
> can we do deb/rpm as base then have container either created from them 
> or directly from artifacts?
>
> On test usage side, seems could probably break up tests into 
> base/required and then optional/add-on tests/test-suites.  Think 
> remember seeing mention of certain tests that are failing at times on 
> certain component(s) anyways in the core builds but don’t mean that 
> the build is broken, so would make sense to have some clean up around those anyways.
>
> -----Original Message-----
> From: RJ Nowling [mailto:rnowling@gmail.com]
> Sent: Sunday, August 30, 2015 1:11 PM
> To: dev@bigtop.apache.org
> Subject: Re: Proposal for "BigTop Data Generators"
>
> I agree with the above. :)
>
> On Sun, Aug 30, 2015 at 11:19 AM, Jay Vyas 
> <ja...@gmail.com>
> wrote:
>
> > Hi RJ.
> >
> > Maven repositories and docker containers for the transaction queue 
> > are good enough IMO.  That will give people a way to compose them in 
> > different idioms (one for Java folks, another for broader Linux 
> > audience
> ).
> >
> > I think the lib designs are fairly intuitive.  I would say that we 
> > should constrain them all to being written in Java or Groovy to keep 
> > the bigtop theme of "JVM for everything" :).
> >
> > Any particular questions you have around technical design can be 
> > followed in a JIRA or else maybe a Readme spec that goes in a  top 
> > level of the data-generators dir...
> >
> > > On Aug 30, 2015, at 1:51 AM, RJ Nowling <rn...@gmail.com> wrote:
> > >
> > > I'd like to keep this conversation going.
> > >
> > > So here are a few discussion points:
> > >
> > > 1. How do we want to make the data generators available?  Maven?
> > > RPMs
> > and
> > > Debs?
> > >
> > > For now, I'm using a gradle multi-project build to easily build 
> > > and
> > install
> > > the BPS data generators and its libraries into a local maven repo.
> > > This makes development easy.  Eventually, I would like to post 
> > > binaries
> > through
> > > Maven for easy integration by users.  RPMs / Debs could be 
> > > interesting since I use a pattern where the data generators are 
> > > libraries (to support application integration / parallelization by 
> > > the host framework) but also provide CLI drivers for local testing.
> > >
> > > 2.  The idea of using the data generators as part of the smoke 
> > > tests came up.  Since there is concern about making the data 
> > > generators required, we could offer the blueprints (BigPetStore) 
> > > as optional smoke tests.  Would that be a good compromise?
> > >
> > > 3.  How will they be maintained?
> > >
> > > I'll certainly add myself to the maintainers list and will be 
> > > taking responsibility.  I'm happy to have others help as well if 
> > > anyone wants to
> > > -- if not, that's cool, too.
> > >
> > > 4. Is anyone interested at all in discussing library APIs and designs?
> > > What about internal interfaces and such?
> > >
> > >
> > > My plan was to add at least one more data generator (weather
> > > simulator)
> > to
> > > bigtop-data-generators in the short term.  However, given the 
> > > concerns raised by Cos (more discussion needed) and Olaf (don't 
> > > want to force data generators on unsuspecting users ;) ), I would 
> > > like to reach some
> > consensus
> > > on what people are concerned about and solutions.
> > >
> > > On Thu, Aug 27, 2015 at 12:38 PM, Konstantin Boudnik 
> > > <co...@apache.org>
> > wrote:
> > >
> > >> Fine by me. I have linked this thread to the JIRA ticket that RJ
> > created,
> > >> so
> > >> we have a way to connect one to another ;)
> > >>
> > >>> On Thu, Aug 27, 2015 at 01:02PM, Olaf Flebbe wrote:
> > >>> Hi,
> > >>>
> > >>> I am not confident that moving important design discussions with 
> > >>> impact
> > >> to
> > >>> the whole project to jira is a good idea.
> > >>>
> > >>> In the current JIRA Traffic storm it is not easy to identify and 
> > >>> follow
> > >> important tickets.
> > >>>
> > >>> Please keep discussions on the list or at least, please state on 
> > >>> this
> > >> list which Ticket to follow ...
> > >>>
> > >>> Olaf
> > >>>
> > >>>
> > >>>
> > >>>> Am 26.08.2015 um 22:56 schrieb Konstantin Boudnik <co...@apache.org>:
> > >>>>
> > >>>> On Wed, Aug 26, 2015 at 10:38PM, Olaf Flebbe wrote:
> > >>>>> Hi,
> > >>>>>
> > >>>>> Nive to have data generators in Bigtop.
> > >>>>>
> > >>>>> But please do not include it in bigtop_utils, since this 
> > >>>>> package is mandatory. Not everyone needs a data generator .
> > >>>>
> > >>>> Yup. And let's move further design discussion to the JIRA!
> > >>>>
> > >>>>> Olaf
> > >>>>>
> > >>>>>
> > >>>>>> Am 26.08.2015 um 11:25 schrieb Jay Vyas <
> > jayunit100.apache@gmail.com
> > >>> :
> > >>>>>>
> > >>>>>> Publishing the jar to bigtops maven is probably a good first 
> > >>>>>> step
> > >> ,Then apps can just include it as needed...?.
> > >>>>>>
> > >>>>>> I'm not against packaging if someone wants packages for this.
> > >>>>>> Maybe
> > >> even include it in bigtop util ?
> > >>>>>>
> > >>>>>> Let's move to jira,
> > >>>>>>
> > >>>>>>> On Aug 25, 2015, at 9:41 PM, Konstantin Boudnik 
> > >>>>>>> <co...@apache.org>
> > >> wrote:
> > >>>>>>>
> > >>>>>>> It is pretty cool indeed!
> > >>>>>>>
> > >>>>>>> I wonder how it needs to be structured to be:
> > >>>>>>> - easy to access/use from other components wherever it is 
> > >>>>>>> needed
> > >>>>>>> - doesn't interfere with the rest of the stack
> > >>>>>>>
> > >>>>>>> I guess one possible way would be to implement the generator 
> > >>>>>>> as a
> > >> set of maven
> > >>>>>>> artifacts, that could be installed/consumed transparently by 
> > >>>>>>> just
> > >> declaring a
> > >>>>>>> dependency e.g as proposed via top-level component.
> > >>>>>>>
> > >>>>>>> Another way is to have a new package like we do for 
> > >>>>>>> bigtop-utils
> > >> and such.
> > >>>>>>>
> > >>>>>>> Perhaps this discussion should be moved to JIRA or shall we
> > >> continue on the
> > >>>>>>> dev@ ??
> > >>>>>>>
> > >>>>>>> Cos
> > >>>>>>>
> > >>>>>>>> On Sun, Aug 23, 2015 at 11:53AM, RJ Nowling wrote:
> > >>>>>>>> Hi BigTop,
> > >>>>>>>>
> > >>>>>>>> I had a discussion with Jay yesterday, we'd like to propose 
> > >>>>>>>> a new
> > >> component
> > >>>>>>>> for BigTop: BigTop Data Generators.
> > >>>>>>>>
> > >>>>>>>> BigTop Data Generators would consist of a common set of 
> > >>>>>>>> libraries
> > >> for
> > >>>>>>>> building data generators and three example data generators:
> > >>>>>>>>
> > >>>>>>>> * BigPetStore transaction generator (moved from 
> > >>>>>>>> BigPetStore)
> > >>>>>>>> * BigTop Bazaar -- attendee movement and interactions with 
> > >>>>>>>> booths
> > >> on a
> > >>>>>>>> showroom floor, at a conference, or at a mall
> > >>>>>>>> * BigTop Weatherman -- stochastic weather simulation
> > >> (temperature, wind
> > >>>>>>>> speed, wind chill, rainfall, etc.) per zip code.  (From a 
> > >>>>>>>> model
> > >> trained on
> > >>>>>>>> NOAA historical weather data)
> > >>>>>>>>
> > >>>>>>>> We believe that creating a common set of libraries will 
> > >>>>>>>> have
> > >> several
> > >>>>>>>> benefits including:
> > >>>>>>>>
> > >>>>>>>>  * Easier for others to build their own data generators
> > >>>>>>>>  * Make data generators smaller and easier to maintain
> > >>>>>>>>  * Share improvements across the data generators
> > >>>>>>>>
> > >>>>>>>> More details on the libraries are below.
> > >>>>>>>>
> > >>>>>>>> BigPetStore will be continue to focus on building  and 
> > >>>>>>>> maintaining blueprints, powered by the BigTop Data Generators.
> > >>>>>>>>
> > >>>>>>>> Our vision is that we get all of Apache coming to BigTop 
> > >>>>>>>> for tools
> > >> for
> > >>>>>>>> building better, more comprehensive blueprints.  We want to
> > >> support these
> > >>>>>>>> efforts through data generators and the initial set of 
> > >>>>>>>> blueprint
> > >> we've been
> > >>>>>>>> building.
> > >>>>>>>>
> > >>>>>>>> If the community is generally in support of this, I can 
> > >>>>>>>> create a
> > >> top-level
> > >>>>>>>> "bigtop-data-generators" directory and put the data 
> > >>>>>>>> generators and libraries in there.
> > >>>>>>>>
> > >>>>>>>> Thanks!
> > >>>>>>>>
> > >>>>>>>> RJ
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> -------
> > >>>>>>>> Library details:
> > >>>>>>>>
> > >>>>>>>> So far, I've extracted the following common libraries:
> > >>>>>>>>
> > >>>>>>>>  * Samplers -- provides classes for PDFs and various 
> > >>>>>>>> samplers
> > >>>>>>>>  * Name generator -- data set and samplers for generating 
> > >>>>>>>> names
> > >>>>>>>>  * Location data set -- data set and classes for US zip 
> > >>>>>>>> codes,
> > >> their
> > >>>>>>>> GPS coordinates, median house hold incomes, and population 
> > >>>>>>>> sizes
> > >>>>>>>>  * Product generator -- library for enumerating products 
> > >>>>>>>> from a specification file.  Comes with default 
> > >>>>>>>> specifications for
> > >> BigPetStore
> > >>>>>>>>
> > >>>>>>>> I also expect that I'll add libraries for:
> > >>>>>>>>
> > >>>>>>>>   * Particle simulation -- customer movement in a room
> > >>>>>>>>   * Latent factor model generation -- generate latent 
> > >>>>>>>> factors and customer weights to create something like MovieLens data.
> > >>>>>>>> Used in
> > >> Bazaar
> > >>>>>>>> for booth preferences and potentially in BigPetStore for 
> > >>>>>>>> customer
> > >> item
> > >>>>>>>> preferences
> > >>>>>>>>
> > >>>>>>>> Most of these libraries came out of the BigPetStore data 
> > >>>>>>>> generator
> > >> but the
> > >>>>>>>> other generators have been refactored to be based off the 
> > >>>>>>>> standard
> > >> set of
> > >>>>>>>> libraries.
> > >>
> > >>
> > >>
> >
>
>


--
jay vyas


Re: Proposal for "BigTop Data Generators"

Posted by jay vyas <ja...@gmail.com>.
Hola nate.  Well, here are the Use cases I know of that I have used the
data generators for.

Dockerfile:

(1) for testing kubernetes.  For this, I just use transaction-queue docker
file.
(2) for testing GlusterFS small file workloads, maybe with other analytics
tools...

Maven repo

(3) Java maprduce/ignite/spark applications, which can just add a mvn repo
when compiling.  Java developers never add jars through RPM repos.

RPM/DEB packages:

I could see people using an RPM/DEB data generator, and I'm not against
it.  But I simply don't know of any real world projects which *currently*
need RPM/Deb packages, which is why I haven't bothered to propose it as a
requirement.  Nevertheless linux packages are always a welcome addition if
 someone wants to create em !




On Sun, Aug 30, 2015 at 4:34 PM, <na...@reactor8.com> wrote:

> Would container be in addition to deb/rpm, or instead of?  If latter can
> we do deb/rpm as base then have container either created from them or
> directly from artifacts?
>
> On test usage side, seems could probably break up tests into base/required
> and then optional/add-on tests/test-suites.  Think remember seeing mention
> of certain tests that are failing at times on certain component(s) anyways
> in the core builds but don’t mean that the build is broken, so would make
> sense to have some clean up around those anyways.
>
> -----Original Message-----
> From: RJ Nowling [mailto:rnowling@gmail.com]
> Sent: Sunday, August 30, 2015 1:11 PM
> To: dev@bigtop.apache.org
> Subject: Re: Proposal for "BigTop Data Generators"
>
> I agree with the above. :)
>
> On Sun, Aug 30, 2015 at 11:19 AM, Jay Vyas <ja...@gmail.com>
> wrote:
>
> > Hi RJ.
> >
> > Maven repositories and docker containers for the transaction queue are
> > good enough IMO.  That will give people a way to compose them in
> > different idioms (one for Java folks, another for broader Linux audience
> ).
> >
> > I think the lib designs are fairly intuitive.  I would say that we
> > should constrain them all to being written in Java or Groovy to keep
> > the bigtop theme of "JVM for everything" :).
> >
> > Any particular questions you have around technical design can be
> > followed in a JIRA or else maybe a Readme spec that goes in a  top
> > level of the data-generators dir...
> >
> > > On Aug 30, 2015, at 1:51 AM, RJ Nowling <rn...@gmail.com> wrote:
> > >
> > > I'd like to keep this conversation going.
> > >
> > > So here are a few discussion points:
> > >
> > > 1. How do we want to make the data generators available?  Maven?
> > > RPMs
> > and
> > > Debs?
> > >
> > > For now, I'm using a gradle multi-project build to easily build and
> > install
> > > the BPS data generators and its libraries into a local maven repo.
> > > This makes development easy.  Eventually, I would like to post
> > > binaries
> > through
> > > Maven for easy integration by users.  RPMs / Debs could be
> > > interesting since I use a pattern where the data generators are
> > > libraries (to support application integration / parallelization by
> > > the host framework) but also provide CLI drivers for local testing.
> > >
> > > 2.  The idea of using the data generators as part of the smoke tests
> > > came up.  Since there is concern about making the data generators
> > > required, we could offer the blueprints (BigPetStore) as optional
> > > smoke tests.  Would that be a good compromise?
> > >
> > > 3.  How will they be maintained?
> > >
> > > I'll certainly add myself to the maintainers list and will be taking
> > > responsibility.  I'm happy to have others help as well if anyone
> > > wants to
> > > -- if not, that's cool, too.
> > >
> > > 4. Is anyone interested at all in discussing library APIs and designs?
> > > What about internal interfaces and such?
> > >
> > >
> > > My plan was to add at least one more data generator (weather
> > > simulator)
> > to
> > > bigtop-data-generators in the short term.  However, given the
> > > concerns raised by Cos (more discussion needed) and Olaf (don't want
> > > to force data generators on unsuspecting users ;) ), I would like to
> > > reach some
> > consensus
> > > on what people are concerned about and solutions.
> > >
> > > On Thu, Aug 27, 2015 at 12:38 PM, Konstantin Boudnik
> > > <co...@apache.org>
> > wrote:
> > >
> > >> Fine by me. I have linked this thread to the JIRA ticket that RJ
> > created,
> > >> so
> > >> we have a way to connect one to another ;)
> > >>
> > >>> On Thu, Aug 27, 2015 at 01:02PM, Olaf Flebbe wrote:
> > >>> Hi,
> > >>>
> > >>> I am not confident that moving important design discussions with
> > >>> impact
> > >> to
> > >>> the whole project to jira is a good idea.
> > >>>
> > >>> In the current JIRA Traffic storm it is not easy to identify and
> > >>> follow
> > >> important tickets.
> > >>>
> > >>> Please keep discussions on the list or at least, please state on
> > >>> this
> > >> list which Ticket to follow ...
> > >>>
> > >>> Olaf
> > >>>
> > >>>
> > >>>
> > >>>> Am 26.08.2015 um 22:56 schrieb Konstantin Boudnik <co...@apache.org>:
> > >>>>
> > >>>> On Wed, Aug 26, 2015 at 10:38PM, Olaf Flebbe wrote:
> > >>>>> Hi,
> > >>>>>
> > >>>>> Nive to have data generators in Bigtop.
> > >>>>>
> > >>>>> But please do not include it in bigtop_utils, since this package
> > >>>>> is mandatory. Not everyone needs a data generator .
> > >>>>
> > >>>> Yup. And let's move further design discussion to the JIRA!
> > >>>>
> > >>>>> Olaf
> > >>>>>
> > >>>>>
> > >>>>>> Am 26.08.2015 um 11:25 schrieb Jay Vyas <
> > jayunit100.apache@gmail.com
> > >>> :
> > >>>>>>
> > >>>>>> Publishing the jar to bigtops maven is probably a good first
> > >>>>>> step
> > >> ,Then apps can just include it as needed...?.
> > >>>>>>
> > >>>>>> I'm not against packaging if someone wants packages for this.
> > >>>>>> Maybe
> > >> even include it in bigtop util ?
> > >>>>>>
> > >>>>>> Let's move to jira,
> > >>>>>>
> > >>>>>>> On Aug 25, 2015, at 9:41 PM, Konstantin Boudnik
> > >>>>>>> <co...@apache.org>
> > >> wrote:
> > >>>>>>>
> > >>>>>>> It is pretty cool indeed!
> > >>>>>>>
> > >>>>>>> I wonder how it needs to be structured to be:
> > >>>>>>> - easy to access/use from other components wherever it is
> > >>>>>>> needed
> > >>>>>>> - doesn't interfere with the rest of the stack
> > >>>>>>>
> > >>>>>>> I guess one possible way would be to implement the generator
> > >>>>>>> as a
> > >> set of maven
> > >>>>>>> artifacts, that could be installed/consumed transparently by
> > >>>>>>> just
> > >> declaring a
> > >>>>>>> dependency e.g as proposed via top-level component.
> > >>>>>>>
> > >>>>>>> Another way is to have a new package like we do for
> > >>>>>>> bigtop-utils
> > >> and such.
> > >>>>>>>
> > >>>>>>> Perhaps this discussion should be moved to JIRA or shall we
> > >> continue on the
> > >>>>>>> dev@ ??
> > >>>>>>>
> > >>>>>>> Cos
> > >>>>>>>
> > >>>>>>>> On Sun, Aug 23, 2015 at 11:53AM, RJ Nowling wrote:
> > >>>>>>>> Hi BigTop,
> > >>>>>>>>
> > >>>>>>>> I had a discussion with Jay yesterday, we'd like to propose a
> > >>>>>>>> new
> > >> component
> > >>>>>>>> for BigTop: BigTop Data Generators.
> > >>>>>>>>
> > >>>>>>>> BigTop Data Generators would consist of a common set of
> > >>>>>>>> libraries
> > >> for
> > >>>>>>>> building data generators and three example data generators:
> > >>>>>>>>
> > >>>>>>>> * BigPetStore transaction generator (moved from BigPetStore)
> > >>>>>>>> * BigTop Bazaar -- attendee movement and interactions with
> > >>>>>>>> booths
> > >> on a
> > >>>>>>>> showroom floor, at a conference, or at a mall
> > >>>>>>>> * BigTop Weatherman -- stochastic weather simulation
> > >> (temperature, wind
> > >>>>>>>> speed, wind chill, rainfall, etc.) per zip code.  (From a
> > >>>>>>>> model
> > >> trained on
> > >>>>>>>> NOAA historical weather data)
> > >>>>>>>>
> > >>>>>>>> We believe that creating a common set of libraries will have
> > >> several
> > >>>>>>>> benefits including:
> > >>>>>>>>
> > >>>>>>>>  * Easier for others to build their own data generators
> > >>>>>>>>  * Make data generators smaller and easier to maintain
> > >>>>>>>>  * Share improvements across the data generators
> > >>>>>>>>
> > >>>>>>>> More details on the libraries are below.
> > >>>>>>>>
> > >>>>>>>> BigPetStore will be continue to focus on building  and
> > >>>>>>>> maintaining blueprints, powered by the BigTop Data Generators.
> > >>>>>>>>
> > >>>>>>>> Our vision is that we get all of Apache coming to BigTop for
> > >>>>>>>> tools
> > >> for
> > >>>>>>>> building better, more comprehensive blueprints.  We want to
> > >> support these
> > >>>>>>>> efforts through data generators and the initial set of
> > >>>>>>>> blueprint
> > >> we've been
> > >>>>>>>> building.
> > >>>>>>>>
> > >>>>>>>> If the community is generally in support of this, I can
> > >>>>>>>> create a
> > >> top-level
> > >>>>>>>> "bigtop-data-generators" directory and put the data
> > >>>>>>>> generators and libraries in there.
> > >>>>>>>>
> > >>>>>>>> Thanks!
> > >>>>>>>>
> > >>>>>>>> RJ
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> -------
> > >>>>>>>> Library details:
> > >>>>>>>>
> > >>>>>>>> So far, I've extracted the following common libraries:
> > >>>>>>>>
> > >>>>>>>>  * Samplers -- provides classes for PDFs and various samplers
> > >>>>>>>>  * Name generator -- data set and samplers for generating
> > >>>>>>>> names
> > >>>>>>>>  * Location data set -- data set and classes for US zip
> > >>>>>>>> codes,
> > >> their
> > >>>>>>>> GPS coordinates, median house hold incomes, and population
> > >>>>>>>> sizes
> > >>>>>>>>  * Product generator -- library for enumerating products from
> > >>>>>>>> a specification file.  Comes with default specifications for
> > >> BigPetStore
> > >>>>>>>>
> > >>>>>>>> I also expect that I'll add libraries for:
> > >>>>>>>>
> > >>>>>>>>   * Particle simulation -- customer movement in a room
> > >>>>>>>>   * Latent factor model generation -- generate latent factors
> > >>>>>>>> and customer weights to create something like MovieLens data.
> > >>>>>>>> Used in
> > >> Bazaar
> > >>>>>>>> for booth preferences and potentially in BigPetStore for
> > >>>>>>>> customer
> > >> item
> > >>>>>>>> preferences
> > >>>>>>>>
> > >>>>>>>> Most of these libraries came out of the BigPetStore data
> > >>>>>>>> generator
> > >> but the
> > >>>>>>>> other generators have been refactored to be based off the
> > >>>>>>>> standard
> > >> set of
> > >>>>>>>> libraries.
> > >>
> > >>
> > >>
> >
>
>


-- 
jay vyas

RE: Proposal for "BigTop Data Generators"

Posted by na...@reactor8.com.
Would container be in addition to deb/rpm, or instead of?  If latter can we do deb/rpm as base then have container either created from them or directly from artifacts?

On test usage side, seems could probably break up tests into base/required and then optional/add-on tests/test-suites.  Think remember seeing mention of certain tests that are failing at times on certain component(s) anyways in the core builds but don’t mean that the build is broken, so would make sense to have some clean up around those anyways.

-----Original Message-----
From: RJ Nowling [mailto:rnowling@gmail.com] 
Sent: Sunday, August 30, 2015 1:11 PM
To: dev@bigtop.apache.org
Subject: Re: Proposal for "BigTop Data Generators"

I agree with the above. :)

On Sun, Aug 30, 2015 at 11:19 AM, Jay Vyas <ja...@gmail.com>
wrote:

> Hi RJ.
>
> Maven repositories and docker containers for the transaction queue are 
> good enough IMO.  That will give people a way to compose them in 
> different idioms (one for Java folks, another for broader Linux audience ).
>
> I think the lib designs are fairly intuitive.  I would say that we 
> should constrain them all to being written in Java or Groovy to keep 
> the bigtop theme of "JVM for everything" :).
>
> Any particular questions you have around technical design can be 
> followed in a JIRA or else maybe a Readme spec that goes in a  top 
> level of the data-generators dir...
>
> > On Aug 30, 2015, at 1:51 AM, RJ Nowling <rn...@gmail.com> wrote:
> >
> > I'd like to keep this conversation going.
> >
> > So here are a few discussion points:
> >
> > 1. How do we want to make the data generators available?  Maven?  
> > RPMs
> and
> > Debs?
> >
> > For now, I'm using a gradle multi-project build to easily build and
> install
> > the BPS data generators and its libraries into a local maven repo.  
> > This makes development easy.  Eventually, I would like to post 
> > binaries
> through
> > Maven for easy integration by users.  RPMs / Debs could be 
> > interesting since I use a pattern where the data generators are 
> > libraries (to support application integration / parallelization by 
> > the host framework) but also provide CLI drivers for local testing.
> >
> > 2.  The idea of using the data generators as part of the smoke tests 
> > came up.  Since there is concern about making the data generators 
> > required, we could offer the blueprints (BigPetStore) as optional 
> > smoke tests.  Would that be a good compromise?
> >
> > 3.  How will they be maintained?
> >
> > I'll certainly add myself to the maintainers list and will be taking 
> > responsibility.  I'm happy to have others help as well if anyone 
> > wants to
> > -- if not, that's cool, too.
> >
> > 4. Is anyone interested at all in discussing library APIs and designs?
> > What about internal interfaces and such?
> >
> >
> > My plan was to add at least one more data generator (weather 
> > simulator)
> to
> > bigtop-data-generators in the short term.  However, given the 
> > concerns raised by Cos (more discussion needed) and Olaf (don't want 
> > to force data generators on unsuspecting users ;) ), I would like to 
> > reach some
> consensus
> > on what people are concerned about and solutions.
> >
> > On Thu, Aug 27, 2015 at 12:38 PM, Konstantin Boudnik 
> > <co...@apache.org>
> wrote:
> >
> >> Fine by me. I have linked this thread to the JIRA ticket that RJ
> created,
> >> so
> >> we have a way to connect one to another ;)
> >>
> >>> On Thu, Aug 27, 2015 at 01:02PM, Olaf Flebbe wrote:
> >>> Hi,
> >>>
> >>> I am not confident that moving important design discussions with 
> >>> impact
> >> to
> >>> the whole project to jira is a good idea.
> >>>
> >>> In the current JIRA Traffic storm it is not easy to identify and 
> >>> follow
> >> important tickets.
> >>>
> >>> Please keep discussions on the list or at least, please state on 
> >>> this
> >> list which Ticket to follow ...
> >>>
> >>> Olaf
> >>>
> >>>
> >>>
> >>>> Am 26.08.2015 um 22:56 schrieb Konstantin Boudnik <co...@apache.org>:
> >>>>
> >>>> On Wed, Aug 26, 2015 at 10:38PM, Olaf Flebbe wrote:
> >>>>> Hi,
> >>>>>
> >>>>> Nive to have data generators in Bigtop.
> >>>>>
> >>>>> But please do not include it in bigtop_utils, since this package 
> >>>>> is mandatory. Not everyone needs a data generator .
> >>>>
> >>>> Yup. And let's move further design discussion to the JIRA!
> >>>>
> >>>>> Olaf
> >>>>>
> >>>>>
> >>>>>> Am 26.08.2015 um 11:25 schrieb Jay Vyas <
> jayunit100.apache@gmail.com
> >>> :
> >>>>>>
> >>>>>> Publishing the jar to bigtops maven is probably a good first 
> >>>>>> step
> >> ,Then apps can just include it as needed...?.
> >>>>>>
> >>>>>> I'm not against packaging if someone wants packages for this.  
> >>>>>> Maybe
> >> even include it in bigtop util ?
> >>>>>>
> >>>>>> Let's move to jira,
> >>>>>>
> >>>>>>> On Aug 25, 2015, at 9:41 PM, Konstantin Boudnik 
> >>>>>>> <co...@apache.org>
> >> wrote:
> >>>>>>>
> >>>>>>> It is pretty cool indeed!
> >>>>>>>
> >>>>>>> I wonder how it needs to be structured to be:
> >>>>>>> - easy to access/use from other components wherever it is 
> >>>>>>> needed
> >>>>>>> - doesn't interfere with the rest of the stack
> >>>>>>>
> >>>>>>> I guess one possible way would be to implement the generator 
> >>>>>>> as a
> >> set of maven
> >>>>>>> artifacts, that could be installed/consumed transparently by 
> >>>>>>> just
> >> declaring a
> >>>>>>> dependency e.g as proposed via top-level component.
> >>>>>>>
> >>>>>>> Another way is to have a new package like we do for 
> >>>>>>> bigtop-utils
> >> and such.
> >>>>>>>
> >>>>>>> Perhaps this discussion should be moved to JIRA or shall we
> >> continue on the
> >>>>>>> dev@ ??
> >>>>>>>
> >>>>>>> Cos
> >>>>>>>
> >>>>>>>> On Sun, Aug 23, 2015 at 11:53AM, RJ Nowling wrote:
> >>>>>>>> Hi BigTop,
> >>>>>>>>
> >>>>>>>> I had a discussion with Jay yesterday, we'd like to propose a 
> >>>>>>>> new
> >> component
> >>>>>>>> for BigTop: BigTop Data Generators.
> >>>>>>>>
> >>>>>>>> BigTop Data Generators would consist of a common set of 
> >>>>>>>> libraries
> >> for
> >>>>>>>> building data generators and three example data generators:
> >>>>>>>>
> >>>>>>>> * BigPetStore transaction generator (moved from BigPetStore)
> >>>>>>>> * BigTop Bazaar -- attendee movement and interactions with 
> >>>>>>>> booths
> >> on a
> >>>>>>>> showroom floor, at a conference, or at a mall
> >>>>>>>> * BigTop Weatherman -- stochastic weather simulation
> >> (temperature, wind
> >>>>>>>> speed, wind chill, rainfall, etc.) per zip code.  (From a 
> >>>>>>>> model
> >> trained on
> >>>>>>>> NOAA historical weather data)
> >>>>>>>>
> >>>>>>>> We believe that creating a common set of libraries will have
> >> several
> >>>>>>>> benefits including:
> >>>>>>>>
> >>>>>>>>  * Easier for others to build their own data generators
> >>>>>>>>  * Make data generators smaller and easier to maintain
> >>>>>>>>  * Share improvements across the data generators
> >>>>>>>>
> >>>>>>>> More details on the libraries are below.
> >>>>>>>>
> >>>>>>>> BigPetStore will be continue to focus on building  and 
> >>>>>>>> maintaining blueprints, powered by the BigTop Data Generators.
> >>>>>>>>
> >>>>>>>> Our vision is that we get all of Apache coming to BigTop for 
> >>>>>>>> tools
> >> for
> >>>>>>>> building better, more comprehensive blueprints.  We want to
> >> support these
> >>>>>>>> efforts through data generators and the initial set of 
> >>>>>>>> blueprint
> >> we've been
> >>>>>>>> building.
> >>>>>>>>
> >>>>>>>> If the community is generally in support of this, I can 
> >>>>>>>> create a
> >> top-level
> >>>>>>>> "bigtop-data-generators" directory and put the data 
> >>>>>>>> generators and libraries in there.
> >>>>>>>>
> >>>>>>>> Thanks!
> >>>>>>>>
> >>>>>>>> RJ
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> -------
> >>>>>>>> Library details:
> >>>>>>>>
> >>>>>>>> So far, I've extracted the following common libraries:
> >>>>>>>>
> >>>>>>>>  * Samplers -- provides classes for PDFs and various samplers
> >>>>>>>>  * Name generator -- data set and samplers for generating 
> >>>>>>>> names
> >>>>>>>>  * Location data set -- data set and classes for US zip 
> >>>>>>>> codes,
> >> their
> >>>>>>>> GPS coordinates, median house hold incomes, and population 
> >>>>>>>> sizes
> >>>>>>>>  * Product generator -- library for enumerating products from 
> >>>>>>>> a specification file.  Comes with default specifications for
> >> BigPetStore
> >>>>>>>>
> >>>>>>>> I also expect that I'll add libraries for:
> >>>>>>>>
> >>>>>>>>   * Particle simulation -- customer movement in a room
> >>>>>>>>   * Latent factor model generation -- generate latent factors 
> >>>>>>>> and customer weights to create something like MovieLens data.  
> >>>>>>>> Used in
> >> Bazaar
> >>>>>>>> for booth preferences and potentially in BigPetStore for 
> >>>>>>>> customer
> >> item
> >>>>>>>> preferences
> >>>>>>>>
> >>>>>>>> Most of these libraries came out of the BigPetStore data 
> >>>>>>>> generator
> >> but the
> >>>>>>>> other generators have been refactored to be based off the 
> >>>>>>>> standard
> >> set of
> >>>>>>>> libraries.
> >>
> >>
> >>
>


Re: Proposal for "BigTop Data Generators"

Posted by RJ Nowling <rn...@gmail.com>.
I agree with the above. :)

On Sun, Aug 30, 2015 at 11:19 AM, Jay Vyas <ja...@gmail.com>
wrote:

> Hi RJ.
>
> Maven repositories and docker containers for the transaction queue are
> good enough IMO.  That will give people a way to compose them in different
> idioms (one for Java folks, another for broader Linux audience ).
>
> I think the lib designs are fairly intuitive.  I would say that we should
> constrain them all to being written in Java or Groovy to keep the bigtop
> theme of "JVM for everything" :).
>
> Any particular questions you have around technical design can be followed
> in a JIRA or else maybe a Readme spec that goes in a  top level of the
> data-generators dir...
>
> > On Aug 30, 2015, at 1:51 AM, RJ Nowling <rn...@gmail.com> wrote:
> >
> > I'd like to keep this conversation going.
> >
> > So here are a few discussion points:
> >
> > 1. How do we want to make the data generators available?  Maven?  RPMs
> and
> > Debs?
> >
> > For now, I'm using a gradle multi-project build to easily build and
> install
> > the BPS data generators and its libraries into a local maven repo.  This
> > makes development easy.  Eventually, I would like to post binaries
> through
> > Maven for easy integration by users.  RPMs / Debs could be interesting
> > since I use a pattern where the data generators are libraries (to support
> > application integration / parallelization by the host framework) but also
> > provide CLI drivers for local testing.
> >
> > 2.  The idea of using the data generators as part of the smoke tests came
> > up.  Since there is concern about making the data generators required, we
> > could offer the blueprints (BigPetStore) as optional smoke tests.  Would
> > that be a good compromise?
> >
> > 3.  How will they be maintained?
> >
> > I'll certainly add myself to the maintainers list and will be taking
> > responsibility.  I'm happy to have others help as well if anyone wants to
> > -- if not, that's cool, too.
> >
> > 4. Is anyone interested at all in discussing library APIs and designs?
> > What about internal interfaces and such?
> >
> >
> > My plan was to add at least one more data generator (weather simulator)
> to
> > bigtop-data-generators in the short term.  However, given the concerns
> > raised by Cos (more discussion needed) and Olaf (don't want to force data
> > generators on unsuspecting users ;) ), I would like to reach some
> consensus
> > on what people are concerned about and solutions.
> >
> > On Thu, Aug 27, 2015 at 12:38 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> >
> >> Fine by me. I have linked this thread to the JIRA ticket that RJ
> created,
> >> so
> >> we have a way to connect one to another ;)
> >>
> >>> On Thu, Aug 27, 2015 at 01:02PM, Olaf Flebbe wrote:
> >>> Hi,
> >>>
> >>> I am not confident that moving important design discussions with impact
> >> to
> >>> the whole project to jira is a good idea.
> >>>
> >>> In the current JIRA Traffic storm it is not easy to identify and follow
> >> important tickets.
> >>>
> >>> Please keep discussions on the list or at least, please state on this
> >> list which Ticket to follow ...
> >>>
> >>> Olaf
> >>>
> >>>
> >>>
> >>>> Am 26.08.2015 um 22:56 schrieb Konstantin Boudnik <co...@apache.org>:
> >>>>
> >>>> On Wed, Aug 26, 2015 at 10:38PM, Olaf Flebbe wrote:
> >>>>> Hi,
> >>>>>
> >>>>> Nive to have data generators in Bigtop.
> >>>>>
> >>>>> But please do not include it in bigtop_utils, since this package is
> >>>>> mandatory. Not everyone needs a data generator .
> >>>>
> >>>> Yup. And let's move further design discussion to the JIRA!
> >>>>
> >>>>> Olaf
> >>>>>
> >>>>>
> >>>>>> Am 26.08.2015 um 11:25 schrieb Jay Vyas <
> jayunit100.apache@gmail.com
> >>> :
> >>>>>>
> >>>>>> Publishing the jar to bigtops maven is probably a good first step
> >> ,Then apps can just include it as needed...?.
> >>>>>>
> >>>>>> I'm not against packaging if someone wants packages for this.  Maybe
> >> even include it in bigtop util ?
> >>>>>>
> >>>>>> Let's move to jira,
> >>>>>>
> >>>>>>> On Aug 25, 2015, at 9:41 PM, Konstantin Boudnik <co...@apache.org>
> >> wrote:
> >>>>>>>
> >>>>>>> It is pretty cool indeed!
> >>>>>>>
> >>>>>>> I wonder how it needs to be structured to be:
> >>>>>>> - easy to access/use from other components wherever it is needed
> >>>>>>> - doesn't interfere with the rest of the stack
> >>>>>>>
> >>>>>>> I guess one possible way would be to implement the generator as a
> >> set of maven
> >>>>>>> artifacts, that could be installed/consumed transparently by just
> >> declaring a
> >>>>>>> dependency e.g as proposed via top-level component.
> >>>>>>>
> >>>>>>> Another way is to have a new package like we do for bigtop-utils
> >> and such.
> >>>>>>>
> >>>>>>> Perhaps this discussion should be moved to JIRA or shall we
> >> continue on the
> >>>>>>> dev@ ??
> >>>>>>>
> >>>>>>> Cos
> >>>>>>>
> >>>>>>>> On Sun, Aug 23, 2015 at 11:53AM, RJ Nowling wrote:
> >>>>>>>> Hi BigTop,
> >>>>>>>>
> >>>>>>>> I had a discussion with Jay yesterday, we'd like to propose a new
> >> component
> >>>>>>>> for BigTop: BigTop Data Generators.
> >>>>>>>>
> >>>>>>>> BigTop Data Generators would consist of a common set of libraries
> >> for
> >>>>>>>> building data generators and three example data generators:
> >>>>>>>>
> >>>>>>>> * BigPetStore transaction generator (moved from BigPetStore)
> >>>>>>>> * BigTop Bazaar -- attendee movement and interactions with booths
> >> on a
> >>>>>>>> showroom floor, at a conference, or at a mall
> >>>>>>>> * BigTop Weatherman -- stochastic weather simulation
> >> (temperature, wind
> >>>>>>>> speed, wind chill, rainfall, etc.) per zip code.  (From a model
> >> trained on
> >>>>>>>> NOAA historical weather data)
> >>>>>>>>
> >>>>>>>> We believe that creating a common set of libraries will have
> >> several
> >>>>>>>> benefits including:
> >>>>>>>>
> >>>>>>>>  * Easier for others to build their own data generators
> >>>>>>>>  * Make data generators smaller and easier to maintain
> >>>>>>>>  * Share improvements across the data generators
> >>>>>>>>
> >>>>>>>> More details on the libraries are below.
> >>>>>>>>
> >>>>>>>> BigPetStore will be continue to focus on building  and maintaining
> >>>>>>>> blueprints, powered by the BigTop Data Generators.
> >>>>>>>>
> >>>>>>>> Our vision is that we get all of Apache coming to BigTop for tools
> >> for
> >>>>>>>> building better, more comprehensive blueprints.  We want to
> >> support these
> >>>>>>>> efforts through data generators and the initial set of blueprint
> >> we've been
> >>>>>>>> building.
> >>>>>>>>
> >>>>>>>> If the community is generally in support of this, I can create a
> >> top-level
> >>>>>>>> "bigtop-data-generators" directory and put the data generators and
> >>>>>>>> libraries in there.
> >>>>>>>>
> >>>>>>>> Thanks!
> >>>>>>>>
> >>>>>>>> RJ
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> -------
> >>>>>>>> Library details:
> >>>>>>>>
> >>>>>>>> So far, I've extracted the following common libraries:
> >>>>>>>>
> >>>>>>>>  * Samplers -- provides classes for PDFs and various samplers
> >>>>>>>>  * Name generator -- data set and samplers for generating names
> >>>>>>>>  * Location data set -- data set and classes for US zip codes,
> >> their
> >>>>>>>> GPS coordinates, median house hold incomes, and population sizes
> >>>>>>>>  * Product generator -- library for enumerating products from a
> >>>>>>>> specification file.  Comes with default specifications for
> >> BigPetStore
> >>>>>>>>
> >>>>>>>> I also expect that I'll add libraries for:
> >>>>>>>>
> >>>>>>>>   * Particle simulation -- customer movement in a room
> >>>>>>>>   * Latent factor model generation -- generate latent factors and
> >>>>>>>> customer weights to create something like MovieLens data.  Used in
> >> Bazaar
> >>>>>>>> for booth preferences and potentially in BigPetStore for customer
> >> item
> >>>>>>>> preferences
> >>>>>>>>
> >>>>>>>> Most of these libraries came out of the BigPetStore data generator
> >> but the
> >>>>>>>> other generators have been refactored to be based off the standard
> >> set of
> >>>>>>>> libraries.
> >>
> >>
> >>
>

Re: Proposal for "BigTop Data Generators"

Posted by Jay Vyas <ja...@gmail.com>.
Hi RJ. 

Maven repositories and docker containers for the transaction queue are good enough IMO.  That will give people a way to compose them in different idioms (one for Java folks, another for broader Linux audience ). 

I think the lib designs are fairly intuitive.  I would say that we should constrain them all to being written in Java or Groovy to keep the bigtop theme of "JVM for everything" :).

Any particular questions you have around technical design can be followed in a JIRA or else maybe a Readme spec that goes in a  top level of the data-generators dir...

> On Aug 30, 2015, at 1:51 AM, RJ Nowling <rn...@gmail.com> wrote:
> 
> I'd like to keep this conversation going.
> 
> So here are a few discussion points:
> 
> 1. How do we want to make the data generators available?  Maven?  RPMs and
> Debs?
> 
> For now, I'm using a gradle multi-project build to easily build and install
> the BPS data generators and its libraries into a local maven repo.  This
> makes development easy.  Eventually, I would like to post binaries through
> Maven for easy integration by users.  RPMs / Debs could be interesting
> since I use a pattern where the data generators are libraries (to support
> application integration / parallelization by the host framework) but also
> provide CLI drivers for local testing.
> 
> 2.  The idea of using the data generators as part of the smoke tests came
> up.  Since there is concern about making the data generators required, we
> could offer the blueprints (BigPetStore) as optional smoke tests.  Would
> that be a good compromise?
> 
> 3.  How will they be maintained?
> 
> I'll certainly add myself to the maintainers list and will be taking
> responsibility.  I'm happy to have others help as well if anyone wants to
> -- if not, that's cool, too.
> 
> 4. Is anyone interested at all in discussing library APIs and designs?
> What about internal interfaces and such?
> 
> 
> My plan was to add at least one more data generator (weather simulator) to
> bigtop-data-generators in the short term.  However, given the concerns
> raised by Cos (more discussion needed) and Olaf (don't want to force data
> generators on unsuspecting users ;) ), I would like to reach some consensus
> on what people are concerned about and solutions.
> 
> On Thu, Aug 27, 2015 at 12:38 PM, Konstantin Boudnik <co...@apache.org> wrote:
> 
>> Fine by me. I have linked this thread to the JIRA ticket that RJ created,
>> so
>> we have a way to connect one to another ;)
>> 
>>> On Thu, Aug 27, 2015 at 01:02PM, Olaf Flebbe wrote:
>>> Hi,
>>> 
>>> I am not confident that moving important design discussions with impact
>> to
>>> the whole project to jira is a good idea.
>>> 
>>> In the current JIRA Traffic storm it is not easy to identify and follow
>> important tickets.
>>> 
>>> Please keep discussions on the list or at least, please state on this
>> list which Ticket to follow ...
>>> 
>>> Olaf
>>> 
>>> 
>>> 
>>>> Am 26.08.2015 um 22:56 schrieb Konstantin Boudnik <co...@apache.org>:
>>>> 
>>>> On Wed, Aug 26, 2015 at 10:38PM, Olaf Flebbe wrote:
>>>>> Hi,
>>>>> 
>>>>> Nive to have data generators in Bigtop.
>>>>> 
>>>>> But please do not include it in bigtop_utils, since this package is
>>>>> mandatory. Not everyone needs a data generator .
>>>> 
>>>> Yup. And let's move further design discussion to the JIRA!
>>>> 
>>>>> Olaf
>>>>> 
>>>>> 
>>>>>> Am 26.08.2015 um 11:25 schrieb Jay Vyas <jayunit100.apache@gmail.com
>>> :
>>>>>> 
>>>>>> Publishing the jar to bigtops maven is probably a good first step
>> ,Then apps can just include it as needed...?.
>>>>>> 
>>>>>> I'm not against packaging if someone wants packages for this.  Maybe
>> even include it in bigtop util ?
>>>>>> 
>>>>>> Let's move to jira,
>>>>>> 
>>>>>>> On Aug 25, 2015, at 9:41 PM, Konstantin Boudnik <co...@apache.org>
>> wrote:
>>>>>>> 
>>>>>>> It is pretty cool indeed!
>>>>>>> 
>>>>>>> I wonder how it needs to be structured to be:
>>>>>>> - easy to access/use from other components wherever it is needed
>>>>>>> - doesn't interfere with the rest of the stack
>>>>>>> 
>>>>>>> I guess one possible way would be to implement the generator as a
>> set of maven
>>>>>>> artifacts, that could be installed/consumed transparently by just
>> declaring a
>>>>>>> dependency e.g as proposed via top-level component.
>>>>>>> 
>>>>>>> Another way is to have a new package like we do for bigtop-utils
>> and such.
>>>>>>> 
>>>>>>> Perhaps this discussion should be moved to JIRA or shall we
>> continue on the
>>>>>>> dev@ ??
>>>>>>> 
>>>>>>> Cos
>>>>>>> 
>>>>>>>> On Sun, Aug 23, 2015 at 11:53AM, RJ Nowling wrote:
>>>>>>>> Hi BigTop,
>>>>>>>> 
>>>>>>>> I had a discussion with Jay yesterday, we'd like to propose a new
>> component
>>>>>>>> for BigTop: BigTop Data Generators.
>>>>>>>> 
>>>>>>>> BigTop Data Generators would consist of a common set of libraries
>> for
>>>>>>>> building data generators and three example data generators:
>>>>>>>> 
>>>>>>>> * BigPetStore transaction generator (moved from BigPetStore)
>>>>>>>> * BigTop Bazaar -- attendee movement and interactions with booths
>> on a
>>>>>>>> showroom floor, at a conference, or at a mall
>>>>>>>> * BigTop Weatherman -- stochastic weather simulation
>> (temperature, wind
>>>>>>>> speed, wind chill, rainfall, etc.) per zip code.  (From a model
>> trained on
>>>>>>>> NOAA historical weather data)
>>>>>>>> 
>>>>>>>> We believe that creating a common set of libraries will have
>> several
>>>>>>>> benefits including:
>>>>>>>> 
>>>>>>>>  * Easier for others to build their own data generators
>>>>>>>>  * Make data generators smaller and easier to maintain
>>>>>>>>  * Share improvements across the data generators
>>>>>>>> 
>>>>>>>> More details on the libraries are below.
>>>>>>>> 
>>>>>>>> BigPetStore will be continue to focus on building  and maintaining
>>>>>>>> blueprints, powered by the BigTop Data Generators.
>>>>>>>> 
>>>>>>>> Our vision is that we get all of Apache coming to BigTop for tools
>> for
>>>>>>>> building better, more comprehensive blueprints.  We want to
>> support these
>>>>>>>> efforts through data generators and the initial set of blueprint
>> we've been
>>>>>>>> building.
>>>>>>>> 
>>>>>>>> If the community is generally in support of this, I can create a
>> top-level
>>>>>>>> "bigtop-data-generators" directory and put the data generators and
>>>>>>>> libraries in there.
>>>>>>>> 
>>>>>>>> Thanks!
>>>>>>>> 
>>>>>>>> RJ
>>>>>>>> 
>>>>>>>> 
>>>>>>>> -------
>>>>>>>> Library details:
>>>>>>>> 
>>>>>>>> So far, I've extracted the following common libraries:
>>>>>>>> 
>>>>>>>>  * Samplers -- provides classes for PDFs and various samplers
>>>>>>>>  * Name generator -- data set and samplers for generating names
>>>>>>>>  * Location data set -- data set and classes for US zip codes,
>> their
>>>>>>>> GPS coordinates, median house hold incomes, and population sizes
>>>>>>>>  * Product generator -- library for enumerating products from a
>>>>>>>> specification file.  Comes with default specifications for
>> BigPetStore
>>>>>>>> 
>>>>>>>> I also expect that I'll add libraries for:
>>>>>>>> 
>>>>>>>>   * Particle simulation -- customer movement in a room
>>>>>>>>   * Latent factor model generation -- generate latent factors and
>>>>>>>> customer weights to create something like MovieLens data.  Used in
>> Bazaar
>>>>>>>> for booth preferences and potentially in BigPetStore for customer
>> item
>>>>>>>> preferences
>>>>>>>> 
>>>>>>>> Most of these libraries came out of the BigPetStore data generator
>> but the
>>>>>>>> other generators have been refactored to be based off the standard
>> set of
>>>>>>>> libraries.
>> 
>> 
>> 

Re: Proposal for "BigTop Data Generators"

Posted by RJ Nowling <rn...@gmail.com>.
I'd like to keep this conversation going.

So here are a few discussion points:

1. How do we want to make the data generators available?  Maven?  RPMs and
Debs?

For now, I'm using a gradle multi-project build to easily build and install
the BPS data generators and its libraries into a local maven repo.  This
makes development easy.  Eventually, I would like to post binaries through
Maven for easy integration by users.  RPMs / Debs could be interesting
since I use a pattern where the data generators are libraries (to support
application integration / parallelization by the host framework) but also
provide CLI drivers for local testing.

2.  The idea of using the data generators as part of the smoke tests came
up.  Since there is concern about making the data generators required, we
could offer the blueprints (BigPetStore) as optional smoke tests.  Would
that be a good compromise?

3.  How will they be maintained?

I'll certainly add myself to the maintainers list and will be taking
responsibility.  I'm happy to have others help as well if anyone wants to
-- if not, that's cool, too.

4. Is anyone interested at all in discussing library APIs and designs?
What about internal interfaces and such?


My plan was to add at least one more data generator (weather simulator) to
bigtop-data-generators in the short term.  However, given the concerns
raised by Cos (more discussion needed) and Olaf (don't want to force data
generators on unsuspecting users ;) ), I would like to reach some consensus
on what people are concerned about and solutions.

On Thu, Aug 27, 2015 at 12:38 PM, Konstantin Boudnik <co...@apache.org> wrote:

> Fine by me. I have linked this thread to the JIRA ticket that RJ created,
> so
> we have a way to connect one to another ;)
>
> On Thu, Aug 27, 2015 at 01:02PM, Olaf Flebbe wrote:
> > Hi,
> >
> > I am not confident that moving important design discussions with impact
> to
> > the whole project to jira is a good idea.
> >
> > In the current JIRA Traffic storm it is not easy to identify and follow
> important tickets.
> >
> > Please keep discussions on the list or at least, please state on this
> list which Ticket to follow ...
> >
> > Olaf
> >
> >
> >
> > > Am 26.08.2015 um 22:56 schrieb Konstantin Boudnik <co...@apache.org>:
> > >
> > > On Wed, Aug 26, 2015 at 10:38PM, Olaf Flebbe wrote:
> > >> Hi,
> > >>
> > >> Nive to have data generators in Bigtop.
> > >>
> > >> But please do not include it in bigtop_utils, since this package is
> > >> mandatory. Not everyone needs a data generator .
> > >
> > > Yup. And let's move further design discussion to the JIRA!
> > >
> > >> Olaf
> > >>
> > >>
> > >>> Am 26.08.2015 um 11:25 schrieb Jay Vyas <jayunit100.apache@gmail.com
> >:
> > >>>
> > >>> Publishing the jar to bigtops maven is probably a good first step
> ,Then apps can just include it as needed...?.
> > >>>
> > >>> I'm not against packaging if someone wants packages for this.  Maybe
> even include it in bigtop util ?
> > >>>
> > >>> Let's move to jira,
> > >>>
> > >>>> On Aug 25, 2015, at 9:41 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> > >>>>
> > >>>> It is pretty cool indeed!
> > >>>>
> > >>>> I wonder how it needs to be structured to be:
> > >>>> - easy to access/use from other components wherever it is needed
> > >>>> - doesn't interfere with the rest of the stack
> > >>>>
> > >>>> I guess one possible way would be to implement the generator as a
> set of maven
> > >>>> artifacts, that could be installed/consumed transparently by just
> declaring a
> > >>>> dependency e.g as proposed via top-level component.
> > >>>>
> > >>>> Another way is to have a new package like we do for bigtop-utils
> and such.
> > >>>>
> > >>>> Perhaps this discussion should be moved to JIRA or shall we
> continue on the
> > >>>> dev@ ??
> > >>>>
> > >>>> Cos
> > >>>>
> > >>>>> On Sun, Aug 23, 2015 at 11:53AM, RJ Nowling wrote:
> > >>>>> Hi BigTop,
> > >>>>>
> > >>>>> I had a discussion with Jay yesterday, we'd like to propose a new
> component
> > >>>>> for BigTop: BigTop Data Generators.
> > >>>>>
> > >>>>> BigTop Data Generators would consist of a common set of libraries
> for
> > >>>>> building data generators and three example data generators:
> > >>>>>
> > >>>>>  * BigPetStore transaction generator (moved from BigPetStore)
> > >>>>>  * BigTop Bazaar -- attendee movement and interactions with booths
> on a
> > >>>>> showroom floor, at a conference, or at a mall
> > >>>>>  * BigTop Weatherman -- stochastic weather simulation
> (temperature, wind
> > >>>>> speed, wind chill, rainfall, etc.) per zip code.  (From a model
> trained on
> > >>>>> NOAA historical weather data)
> > >>>>>
> > >>>>> We believe that creating a common set of libraries will have
> several
> > >>>>> benefits including:
> > >>>>>
> > >>>>>   * Easier for others to build their own data generators
> > >>>>>   * Make data generators smaller and easier to maintain
> > >>>>>   * Share improvements across the data generators
> > >>>>>
> > >>>>> More details on the libraries are below.
> > >>>>>
> > >>>>> BigPetStore will be continue to focus on building  and maintaining
> > >>>>> blueprints, powered by the BigTop Data Generators.
> > >>>>>
> > >>>>> Our vision is that we get all of Apache coming to BigTop for tools
> for
> > >>>>> building better, more comprehensive blueprints.  We want to
> support these
> > >>>>> efforts through data generators and the initial set of blueprint
> we've been
> > >>>>> building.
> > >>>>>
> > >>>>> If the community is generally in support of this, I can create a
> top-level
> > >>>>> "bigtop-data-generators" directory and put the data generators and
> > >>>>> libraries in there.
> > >>>>>
> > >>>>> Thanks!
> > >>>>>
> > >>>>> RJ
> > >>>>>
> > >>>>>
> > >>>>> -------
> > >>>>> Library details:
> > >>>>>
> > >>>>> So far, I've extracted the following common libraries:
> > >>>>>
> > >>>>>   * Samplers -- provides classes for PDFs and various samplers
> > >>>>>   * Name generator -- data set and samplers for generating names
> > >>>>>   * Location data set -- data set and classes for US zip codes,
> their
> > >>>>> GPS coordinates, median house hold incomes, and population sizes
> > >>>>>   * Product generator -- library for enumerating products from a
> > >>>>> specification file.  Comes with default specifications for
> BigPetStore
> > >>>>>
> > >>>>> I also expect that I'll add libraries for:
> > >>>>>
> > >>>>>    * Particle simulation -- customer movement in a room
> > >>>>>    * Latent factor model generation -- generate latent factors and
> > >>>>> customer weights to create something like MovieLens data.  Used in
> Bazaar
> > >>>>> for booth preferences and potentially in BigPetStore for customer
> item
> > >>>>> preferences
> > >>>>>
> > >>>>> Most of these libraries came out of the BigPetStore data generator
> but the
> > >>>>> other generators have been refactored to be based off the standard
> set of
> > >>>>> libraries.
> > >>
> > >
> > >
> >
>
>
>

Re: Proposal for "BigTop Data Generators"

Posted by Konstantin Boudnik <co...@apache.org>.
Fine by me. I have linked this thread to the JIRA ticket that RJ created, so
we have a way to connect one to another ;)

On Thu, Aug 27, 2015 at 01:02PM, Olaf Flebbe wrote:
> Hi,
> 
> I am not confident that moving important design discussions with impact to
> the whole project to jira is a good idea.
> 
> In the current JIRA Traffic storm it is not easy to identify and follow important tickets.
> 
> Please keep discussions on the list or at least, please state on this list which Ticket to follow ...
> 
> Olaf
> 
> 
> 
> > Am 26.08.2015 um 22:56 schrieb Konstantin Boudnik <co...@apache.org>:
> > 
> > On Wed, Aug 26, 2015 at 10:38PM, Olaf Flebbe wrote:
> >> Hi,
> >> 
> >> Nive to have data generators in Bigtop.
> >> 
> >> But please do not include it in bigtop_utils, since this package is
> >> mandatory. Not everyone needs a data generator .
> > 
> > Yup. And let's move further design discussion to the JIRA!
> > 
> >> Olaf
> >> 
> >> 
> >>> Am 26.08.2015 um 11:25 schrieb Jay Vyas <ja...@gmail.com>:
> >>> 
> >>> Publishing the jar to bigtops maven is probably a good first step ,Then apps can just include it as needed...?.
> >>> 
> >>> I'm not against packaging if someone wants packages for this.  Maybe even include it in bigtop util ?
> >>> 
> >>> Let's move to jira,
> >>> 
> >>>> On Aug 25, 2015, at 9:41 PM, Konstantin Boudnik <co...@apache.org> wrote:
> >>>> 
> >>>> It is pretty cool indeed!
> >>>> 
> >>>> I wonder how it needs to be structured to be:
> >>>> - easy to access/use from other components wherever it is needed
> >>>> - doesn't interfere with the rest of the stack
> >>>> 
> >>>> I guess one possible way would be to implement the generator as a set of maven
> >>>> artifacts, that could be installed/consumed transparently by just declaring a
> >>>> dependency e.g as proposed via top-level component.
> >>>> 
> >>>> Another way is to have a new package like we do for bigtop-utils and such.
> >>>> 
> >>>> Perhaps this discussion should be moved to JIRA or shall we continue on the
> >>>> dev@ ??
> >>>> 
> >>>> Cos
> >>>> 
> >>>>> On Sun, Aug 23, 2015 at 11:53AM, RJ Nowling wrote:
> >>>>> Hi BigTop,
> >>>>> 
> >>>>> I had a discussion with Jay yesterday, we'd like to propose a new component
> >>>>> for BigTop: BigTop Data Generators.
> >>>>> 
> >>>>> BigTop Data Generators would consist of a common set of libraries for
> >>>>> building data generators and three example data generators:
> >>>>> 
> >>>>>  * BigPetStore transaction generator (moved from BigPetStore)
> >>>>>  * BigTop Bazaar -- attendee movement and interactions with booths on a
> >>>>> showroom floor, at a conference, or at a mall
> >>>>>  * BigTop Weatherman -- stochastic weather simulation (temperature, wind
> >>>>> speed, wind chill, rainfall, etc.) per zip code.  (From a model trained on
> >>>>> NOAA historical weather data)
> >>>>> 
> >>>>> We believe that creating a common set of libraries will have several
> >>>>> benefits including:
> >>>>> 
> >>>>>   * Easier for others to build their own data generators
> >>>>>   * Make data generators smaller and easier to maintain
> >>>>>   * Share improvements across the data generators
> >>>>> 
> >>>>> More details on the libraries are below.
> >>>>> 
> >>>>> BigPetStore will be continue to focus on building  and maintaining
> >>>>> blueprints, powered by the BigTop Data Generators.
> >>>>> 
> >>>>> Our vision is that we get all of Apache coming to BigTop for tools for
> >>>>> building better, more comprehensive blueprints.  We want to support these
> >>>>> efforts through data generators and the initial set of blueprint we've been
> >>>>> building.
> >>>>> 
> >>>>> If the community is generally in support of this, I can create a top-level
> >>>>> "bigtop-data-generators" directory and put the data generators and
> >>>>> libraries in there.
> >>>>> 
> >>>>> Thanks!
> >>>>> 
> >>>>> RJ
> >>>>> 
> >>>>> 
> >>>>> -------
> >>>>> Library details:
> >>>>> 
> >>>>> So far, I've extracted the following common libraries:
> >>>>> 
> >>>>>   * Samplers -- provides classes for PDFs and various samplers
> >>>>>   * Name generator -- data set and samplers for generating names
> >>>>>   * Location data set -- data set and classes for US zip codes, their
> >>>>> GPS coordinates, median house hold incomes, and population sizes
> >>>>>   * Product generator -- library for enumerating products from a
> >>>>> specification file.  Comes with default specifications for BigPetStore
> >>>>> 
> >>>>> I also expect that I'll add libraries for:
> >>>>> 
> >>>>>    * Particle simulation -- customer movement in a room
> >>>>>    * Latent factor model generation -- generate latent factors and
> >>>>> customer weights to create something like MovieLens data.  Used in Bazaar
> >>>>> for booth preferences and potentially in BigPetStore for customer item
> >>>>> preferences
> >>>>> 
> >>>>> Most of these libraries came out of the BigPetStore data generator but the
> >>>>> other generators have been refactored to be based off the standard set of
> >>>>> libraries.
> >> 
> > 
> > 
> 



Re: Proposal for "BigTop Data Generators"

Posted by RJ Nowling <rn...@gmail.com>.
Let's kick off the design discussion here.  I'll provide some background on
the current state of affairs.

The BigPetStore data generator is a library with 5-6 top-level classes
which generate different types (stores, customers, products, purchasing
profiles, and transactions) of data.  I designed it this way so that each
part could be parallelized / distributed using a framework of the user's
choice (Java threads, Spark, Hadoop, etc.).  The data generator takes a
handful of parameters: number of stores, number of customers, number of
purchasing profiles, and the number of days to simulate.  (Everything else
is defined as constants in a Constants class.)

The BPS Spark driver uses this library to generate data.  Jay has also
created a "transaction queue" which uses Java threads and pushes the data
to a rotating file queue or via HTTP.  The BPS data generator does have a
non-threaded, serial CLI driver built-in that I've used for testing and
which generates CSV files:

java -jar bigpetstore-data-generator-1.1.0-SNAPSHOT.jar output/ ...

My anticipation is to do the same for the others.  Implement them as
libraries with a small set of parameters, if necessary.  The libraries can
be composed in that they can depend on one another.  I'll provide CLI
interfaces for simple use cases.  All data are packaged with the libraries
to prevent having to track a bunch of data files.

These libraries should be easily callable from any Groovy or Java tests.
Likewise, we can write Hadoop / Spark / etc. wrappers for them to be copied
onto the cluster for data generation there.

Please let me know what you guys think.



On Thu, Aug 27, 2015 at 6:02 AM, Olaf Flebbe <of...@oflebbe.de> wrote:

> Hi,
>
> I am not confident that moving important design discussions with impact to
> the whole project to jira is a good idea.
>
> In the current JIRA Traffic storm it is not easy to identify and follow
> important tickets.
>
> Please keep discussions on the list or at least, please state on this list
> which Ticket to follow ...
>
> Olaf
>
>
>
> > Am 26.08.2015 um 22:56 schrieb Konstantin Boudnik <co...@apache.org>:
> >
> > On Wed, Aug 26, 2015 at 10:38PM, Olaf Flebbe wrote:
> >> Hi,
> >>
> >> Nive to have data generators in Bigtop.
> >>
> >> But please do not include it in bigtop_utils, since this package is
> >> mandatory. Not everyone needs a data generator .
> >
> > Yup. And let's move further design discussion to the JIRA!
> >
> >> Olaf
> >>
> >>
> >>> Am 26.08.2015 um 11:25 schrieb Jay Vyas <ja...@gmail.com>:
> >>>
> >>> Publishing the jar to bigtops maven is probably a good first step
> ,Then apps can just include it as needed...?.
> >>>
> >>> I'm not against packaging if someone wants packages for this.  Maybe
> even include it in bigtop util ?
> >>>
> >>> Let's move to jira,
> >>>
> >>>> On Aug 25, 2015, at 9:41 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> >>>>
> >>>> It is pretty cool indeed!
> >>>>
> >>>> I wonder how it needs to be structured to be:
> >>>> - easy to access/use from other components wherever it is needed
> >>>> - doesn't interfere with the rest of the stack
> >>>>
> >>>> I guess one possible way would be to implement the generator as a set
> of maven
> >>>> artifacts, that could be installed/consumed transparently by just
> declaring a
> >>>> dependency e.g as proposed via top-level component.
> >>>>
> >>>> Another way is to have a new package like we do for bigtop-utils and
> such.
> >>>>
> >>>> Perhaps this discussion should be moved to JIRA or shall we continue
> on the
> >>>> dev@ ??
> >>>>
> >>>> Cos
> >>>>
> >>>>> On Sun, Aug 23, 2015 at 11:53AM, RJ Nowling wrote:
> >>>>> Hi BigTop,
> >>>>>
> >>>>> I had a discussion with Jay yesterday, we'd like to propose a new
> component
> >>>>> for BigTop: BigTop Data Generators.
> >>>>>
> >>>>> BigTop Data Generators would consist of a common set of libraries for
> >>>>> building data generators and three example data generators:
> >>>>>
> >>>>>  * BigPetStore transaction generator (moved from BigPetStore)
> >>>>>  * BigTop Bazaar -- attendee movement and interactions with booths
> on a
> >>>>> showroom floor, at a conference, or at a mall
> >>>>>  * BigTop Weatherman -- stochastic weather simulation (temperature,
> wind
> >>>>> speed, wind chill, rainfall, etc.) per zip code.  (From a model
> trained on
> >>>>> NOAA historical weather data)
> >>>>>
> >>>>> We believe that creating a common set of libraries will have several
> >>>>> benefits including:
> >>>>>
> >>>>>   * Easier for others to build their own data generators
> >>>>>   * Make data generators smaller and easier to maintain
> >>>>>   * Share improvements across the data generators
> >>>>>
> >>>>> More details on the libraries are below.
> >>>>>
> >>>>> BigPetStore will be continue to focus on building  and maintaining
> >>>>> blueprints, powered by the BigTop Data Generators.
> >>>>>
> >>>>> Our vision is that we get all of Apache coming to BigTop for tools
> for
> >>>>> building better, more comprehensive blueprints.  We want to support
> these
> >>>>> efforts through data generators and the initial set of blueprint
> we've been
> >>>>> building.
> >>>>>
> >>>>> If the community is generally in support of this, I can create a
> top-level
> >>>>> "bigtop-data-generators" directory and put the data generators and
> >>>>> libraries in there.
> >>>>>
> >>>>> Thanks!
> >>>>>
> >>>>> RJ
> >>>>>
> >>>>>
> >>>>> -------
> >>>>> Library details:
> >>>>>
> >>>>> So far, I've extracted the following common libraries:
> >>>>>
> >>>>>   * Samplers -- provides classes for PDFs and various samplers
> >>>>>   * Name generator -- data set and samplers for generating names
> >>>>>   * Location data set -- data set and classes for US zip codes, their
> >>>>> GPS coordinates, median house hold incomes, and population sizes
> >>>>>   * Product generator -- library for enumerating products from a
> >>>>> specification file.  Comes with default specifications for
> BigPetStore
> >>>>>
> >>>>> I also expect that I'll add libraries for:
> >>>>>
> >>>>>    * Particle simulation -- customer movement in a room
> >>>>>    * Latent factor model generation -- generate latent factors and
> >>>>> customer weights to create something like MovieLens data.  Used in
> Bazaar
> >>>>> for booth preferences and potentially in BigPetStore for customer
> item
> >>>>> preferences
> >>>>>
> >>>>> Most of these libraries came out of the BigPetStore data generator
> but the
> >>>>> other generators have been refactored to be based off the standard
> set of
> >>>>> libraries.
> >>
> >
> >
>
>

Re: Proposal for "BigTop Data Generators"

Posted by Olaf Flebbe <of...@oflebbe.de>.
Hi,

I am not confident that moving important design discussions with impact to the whole project to jira is a good idea.

In the current JIRA Traffic storm it is not easy to identify and follow important tickets.

Please keep discussions on the list or at least, please state on this list which Ticket to follow ...

Olaf



> Am 26.08.2015 um 22:56 schrieb Konstantin Boudnik <co...@apache.org>:
> 
> On Wed, Aug 26, 2015 at 10:38PM, Olaf Flebbe wrote:
>> Hi,
>> 
>> Nive to have data generators in Bigtop.
>> 
>> But please do not include it in bigtop_utils, since this package is
>> mandatory. Not everyone needs a data generator .
> 
> Yup. And let's move further design discussion to the JIRA!
> 
>> Olaf
>> 
>> 
>>> Am 26.08.2015 um 11:25 schrieb Jay Vyas <ja...@gmail.com>:
>>> 
>>> Publishing the jar to bigtops maven is probably a good first step ,Then apps can just include it as needed...?.
>>> 
>>> I'm not against packaging if someone wants packages for this.  Maybe even include it in bigtop util ?
>>> 
>>> Let's move to jira,
>>> 
>>>> On Aug 25, 2015, at 9:41 PM, Konstantin Boudnik <co...@apache.org> wrote:
>>>> 
>>>> It is pretty cool indeed!
>>>> 
>>>> I wonder how it needs to be structured to be:
>>>> - easy to access/use from other components wherever it is needed
>>>> - doesn't interfere with the rest of the stack
>>>> 
>>>> I guess one possible way would be to implement the generator as a set of maven
>>>> artifacts, that could be installed/consumed transparently by just declaring a
>>>> dependency e.g as proposed via top-level component.
>>>> 
>>>> Another way is to have a new package like we do for bigtop-utils and such.
>>>> 
>>>> Perhaps this discussion should be moved to JIRA or shall we continue on the
>>>> dev@ ??
>>>> 
>>>> Cos
>>>> 
>>>>> On Sun, Aug 23, 2015 at 11:53AM, RJ Nowling wrote:
>>>>> Hi BigTop,
>>>>> 
>>>>> I had a discussion with Jay yesterday, we'd like to propose a new component
>>>>> for BigTop: BigTop Data Generators.
>>>>> 
>>>>> BigTop Data Generators would consist of a common set of libraries for
>>>>> building data generators and three example data generators:
>>>>> 
>>>>>  * BigPetStore transaction generator (moved from BigPetStore)
>>>>>  * BigTop Bazaar -- attendee movement and interactions with booths on a
>>>>> showroom floor, at a conference, or at a mall
>>>>>  * BigTop Weatherman -- stochastic weather simulation (temperature, wind
>>>>> speed, wind chill, rainfall, etc.) per zip code.  (From a model trained on
>>>>> NOAA historical weather data)
>>>>> 
>>>>> We believe that creating a common set of libraries will have several
>>>>> benefits including:
>>>>> 
>>>>>   * Easier for others to build their own data generators
>>>>>   * Make data generators smaller and easier to maintain
>>>>>   * Share improvements across the data generators
>>>>> 
>>>>> More details on the libraries are below.
>>>>> 
>>>>> BigPetStore will be continue to focus on building  and maintaining
>>>>> blueprints, powered by the BigTop Data Generators.
>>>>> 
>>>>> Our vision is that we get all of Apache coming to BigTop for tools for
>>>>> building better, more comprehensive blueprints.  We want to support these
>>>>> efforts through data generators and the initial set of blueprint we've been
>>>>> building.
>>>>> 
>>>>> If the community is generally in support of this, I can create a top-level
>>>>> "bigtop-data-generators" directory and put the data generators and
>>>>> libraries in there.
>>>>> 
>>>>> Thanks!
>>>>> 
>>>>> RJ
>>>>> 
>>>>> 
>>>>> -------
>>>>> Library details:
>>>>> 
>>>>> So far, I've extracted the following common libraries:
>>>>> 
>>>>>   * Samplers -- provides classes for PDFs and various samplers
>>>>>   * Name generator -- data set and samplers for generating names
>>>>>   * Location data set -- data set and classes for US zip codes, their
>>>>> GPS coordinates, median house hold incomes, and population sizes
>>>>>   * Product generator -- library for enumerating products from a
>>>>> specification file.  Comes with default specifications for BigPetStore
>>>>> 
>>>>> I also expect that I'll add libraries for:
>>>>> 
>>>>>    * Particle simulation -- customer movement in a room
>>>>>    * Latent factor model generation -- generate latent factors and
>>>>> customer weights to create something like MovieLens data.  Used in Bazaar
>>>>> for booth preferences and potentially in BigPetStore for customer item
>>>>> preferences
>>>>> 
>>>>> Most of these libraries came out of the BigPetStore data generator but the
>>>>> other generators have been refactored to be based off the standard set of
>>>>> libraries.
>> 
> 
> 


Re: Proposal for "BigTop Data Generators"

Posted by Konstantin Boudnik <co...@apache.org>.
On Wed, Aug 26, 2015 at 10:38PM, Olaf Flebbe wrote:
> Hi,
> 
> Nive to have data generators in Bigtop.
> 
> But please do not include it in bigtop_utils, since this package is
> mandatory. Not everyone needs a data generator .

Yup. And let's move further design discussion to the JIRA!

> Olaf
> 
> 
> > Am 26.08.2015 um 11:25 schrieb Jay Vyas <ja...@gmail.com>:
> > 
> > Publishing the jar to bigtops maven is probably a good first step ,Then apps can just include it as needed...?.
> > 
> > I'm not against packaging if someone wants packages for this.  Maybe even include it in bigtop util ?
> > 
> > Let's move to jira,
> > 
> >> On Aug 25, 2015, at 9:41 PM, Konstantin Boudnik <co...@apache.org> wrote:
> >> 
> >> It is pretty cool indeed!
> >> 
> >> I wonder how it needs to be structured to be:
> >> - easy to access/use from other components wherever it is needed
> >> - doesn't interfere with the rest of the stack
> >> 
> >> I guess one possible way would be to implement the generator as a set of maven
> >> artifacts, that could be installed/consumed transparently by just declaring a
> >> dependency e.g as proposed via top-level component.
> >> 
> >> Another way is to have a new package like we do for bigtop-utils and such.
> >> 
> >> Perhaps this discussion should be moved to JIRA or shall we continue on the
> >> dev@ ??
> >> 
> >> Cos
> >> 
> >>> On Sun, Aug 23, 2015 at 11:53AM, RJ Nowling wrote:
> >>> Hi BigTop,
> >>> 
> >>> I had a discussion with Jay yesterday, we'd like to propose a new component
> >>> for BigTop: BigTop Data Generators.
> >>> 
> >>> BigTop Data Generators would consist of a common set of libraries for
> >>> building data generators and three example data generators:
> >>> 
> >>>   * BigPetStore transaction generator (moved from BigPetStore)
> >>>   * BigTop Bazaar -- attendee movement and interactions with booths on a
> >>> showroom floor, at a conference, or at a mall
> >>>   * BigTop Weatherman -- stochastic weather simulation (temperature, wind
> >>> speed, wind chill, rainfall, etc.) per zip code.  (From a model trained on
> >>> NOAA historical weather data)
> >>> 
> >>> We believe that creating a common set of libraries will have several
> >>> benefits including:
> >>> 
> >>>    * Easier for others to build their own data generators
> >>>    * Make data generators smaller and easier to maintain
> >>>    * Share improvements across the data generators
> >>> 
> >>> More details on the libraries are below.
> >>> 
> >>> BigPetStore will be continue to focus on building  and maintaining
> >>> blueprints, powered by the BigTop Data Generators.
> >>> 
> >>> Our vision is that we get all of Apache coming to BigTop for tools for
> >>> building better, more comprehensive blueprints.  We want to support these
> >>> efforts through data generators and the initial set of blueprint we've been
> >>> building.
> >>> 
> >>> If the community is generally in support of this, I can create a top-level
> >>> "bigtop-data-generators" directory and put the data generators and
> >>> libraries in there.
> >>> 
> >>> Thanks!
> >>> 
> >>> RJ
> >>> 
> >>> 
> >>> -------
> >>> Library details:
> >>> 
> >>> So far, I've extracted the following common libraries:
> >>> 
> >>>    * Samplers -- provides classes for PDFs and various samplers
> >>>    * Name generator -- data set and samplers for generating names
> >>>    * Location data set -- data set and classes for US zip codes, their
> >>> GPS coordinates, median house hold incomes, and population sizes
> >>>    * Product generator -- library for enumerating products from a
> >>> specification file.  Comes with default specifications for BigPetStore
> >>> 
> >>> I also expect that I'll add libraries for:
> >>> 
> >>>     * Particle simulation -- customer movement in a room
> >>>     * Latent factor model generation -- generate latent factors and
> >>> customer weights to create something like MovieLens data.  Used in Bazaar
> >>> for booth preferences and potentially in BigPetStore for customer item
> >>> preferences
> >>> 
> >>> Most of these libraries came out of the BigPetStore data generator but the
> >>> other generators have been refactored to be based off the standard set of
> >>> libraries.
> 



Re: Proposal for "BigTop Data Generators"

Posted by Olaf Flebbe <of...@oflebbe.de>.
Hi,

Nive to have data generators in Bigtop.

But please do not include it in bigtop_utils, since this package is mandatory. Not everyone needs a data generator .

Olaf


> Am 26.08.2015 um 11:25 schrieb Jay Vyas <ja...@gmail.com>:
> 
> Publishing the jar to bigtops maven is probably a good first step ,Then apps can just include it as needed...?.
> 
> I'm not against packaging if someone wants packages for this.  Maybe even include it in bigtop util ?
> 
> Let's move to jira,
> 
>> On Aug 25, 2015, at 9:41 PM, Konstantin Boudnik <co...@apache.org> wrote:
>> 
>> It is pretty cool indeed!
>> 
>> I wonder how it needs to be structured to be:
>> - easy to access/use from other components wherever it is needed
>> - doesn't interfere with the rest of the stack
>> 
>> I guess one possible way would be to implement the generator as a set of maven
>> artifacts, that could be installed/consumed transparently by just declaring a
>> dependency e.g as proposed via top-level component.
>> 
>> Another way is to have a new package like we do for bigtop-utils and such.
>> 
>> Perhaps this discussion should be moved to JIRA or shall we continue on the
>> dev@ ??
>> 
>> Cos
>> 
>>> On Sun, Aug 23, 2015 at 11:53AM, RJ Nowling wrote:
>>> Hi BigTop,
>>> 
>>> I had a discussion with Jay yesterday, we'd like to propose a new component
>>> for BigTop: BigTop Data Generators.
>>> 
>>> BigTop Data Generators would consist of a common set of libraries for
>>> building data generators and three example data generators:
>>> 
>>>   * BigPetStore transaction generator (moved from BigPetStore)
>>>   * BigTop Bazaar -- attendee movement and interactions with booths on a
>>> showroom floor, at a conference, or at a mall
>>>   * BigTop Weatherman -- stochastic weather simulation (temperature, wind
>>> speed, wind chill, rainfall, etc.) per zip code.  (From a model trained on
>>> NOAA historical weather data)
>>> 
>>> We believe that creating a common set of libraries will have several
>>> benefits including:
>>> 
>>>    * Easier for others to build their own data generators
>>>    * Make data generators smaller and easier to maintain
>>>    * Share improvements across the data generators
>>> 
>>> More details on the libraries are below.
>>> 
>>> BigPetStore will be continue to focus on building  and maintaining
>>> blueprints, powered by the BigTop Data Generators.
>>> 
>>> Our vision is that we get all of Apache coming to BigTop for tools for
>>> building better, more comprehensive blueprints.  We want to support these
>>> efforts through data generators and the initial set of blueprint we've been
>>> building.
>>> 
>>> If the community is generally in support of this, I can create a top-level
>>> "bigtop-data-generators" directory and put the data generators and
>>> libraries in there.
>>> 
>>> Thanks!
>>> 
>>> RJ
>>> 
>>> 
>>> -------
>>> Library details:
>>> 
>>> So far, I've extracted the following common libraries:
>>> 
>>>    * Samplers -- provides classes for PDFs and various samplers
>>>    * Name generator -- data set and samplers for generating names
>>>    * Location data set -- data set and classes for US zip codes, their
>>> GPS coordinates, median house hold incomes, and population sizes
>>>    * Product generator -- library for enumerating products from a
>>> specification file.  Comes with default specifications for BigPetStore
>>> 
>>> I also expect that I'll add libraries for:
>>> 
>>>     * Particle simulation -- customer movement in a room
>>>     * Latent factor model generation -- generate latent factors and
>>> customer weights to create something like MovieLens data.  Used in Bazaar
>>> for booth preferences and potentially in BigPetStore for customer item
>>> preferences
>>> 
>>> Most of these libraries came out of the BigPetStore data generator but the
>>> other generators have been refactored to be based off the standard set of
>>> libraries.


Re: Proposal for "BigTop Data Generators"

Posted by Jay Vyas <ja...@gmail.com>.
Publishing the jar to bigtops maven is probably a good first step ,Then apps can just include it as needed...?.

I'm not against packaging if someone wants packages for this.  Maybe even include it in bigtop util ?

Let's move to jira, 

> On Aug 25, 2015, at 9:41 PM, Konstantin Boudnik <co...@apache.org> wrote:
> 
> It is pretty cool indeed!
> 
> I wonder how it needs to be structured to be:
> - easy to access/use from other components wherever it is needed
> - doesn't interfere with the rest of the stack
> 
> I guess one possible way would be to implement the generator as a set of maven
> artifacts, that could be installed/consumed transparently by just declaring a
> dependency e.g as proposed via top-level component.
> 
> Another way is to have a new package like we do for bigtop-utils and such.
> 
> Perhaps this discussion should be moved to JIRA or shall we continue on the
> dev@ ??
> 
> Cos
> 
>> On Sun, Aug 23, 2015 at 11:53AM, RJ Nowling wrote:
>> Hi BigTop,
>> 
>> I had a discussion with Jay yesterday, we'd like to propose a new component
>> for BigTop: BigTop Data Generators.
>> 
>> BigTop Data Generators would consist of a common set of libraries for
>> building data generators and three example data generators:
>> 
>>    * BigPetStore transaction generator (moved from BigPetStore)
>>    * BigTop Bazaar -- attendee movement and interactions with booths on a
>> showroom floor, at a conference, or at a mall
>>    * BigTop Weatherman -- stochastic weather simulation (temperature, wind
>> speed, wind chill, rainfall, etc.) per zip code.  (From a model trained on
>> NOAA historical weather data)
>> 
>> We believe that creating a common set of libraries will have several
>> benefits including:
>> 
>>     * Easier for others to build their own data generators
>>     * Make data generators smaller and easier to maintain
>>     * Share improvements across the data generators
>> 
>> More details on the libraries are below.
>> 
>> BigPetStore will be continue to focus on building  and maintaining
>> blueprints, powered by the BigTop Data Generators.
>> 
>> Our vision is that we get all of Apache coming to BigTop for tools for
>> building better, more comprehensive blueprints.  We want to support these
>> efforts through data generators and the initial set of blueprint we've been
>> building.
>> 
>> If the community is generally in support of this, I can create a top-level
>> "bigtop-data-generators" directory and put the data generators and
>> libraries in there.
>> 
>> Thanks!
>> 
>> RJ
>> 
>> 
>> -------
>> Library details:
>> 
>> So far, I've extracted the following common libraries:
>> 
>>     * Samplers -- provides classes for PDFs and various samplers
>>     * Name generator -- data set and samplers for generating names
>>     * Location data set -- data set and classes for US zip codes, their
>> GPS coordinates, median house hold incomes, and population sizes
>>     * Product generator -- library for enumerating products from a
>> specification file.  Comes with default specifications for BigPetStore
>> 
>> I also expect that I'll add libraries for:
>> 
>>      * Particle simulation -- customer movement in a room
>>      * Latent factor model generation -- generate latent factors and
>> customer weights to create something like MovieLens data.  Used in Bazaar
>> for booth preferences and potentially in BigPetStore for customer item
>> preferences
>> 
>> Most of these libraries came out of the BigPetStore data generator but the
>> other generators have been refactored to be based off the standard set of
>> libraries.

Re: Proposal for "BigTop Data Generators"

Posted by Konstantin Boudnik <co...@apache.org>.
It is pretty cool indeed!

I wonder how it needs to be structured to be:
 - easy to access/use from other components wherever it is needed
 - doesn't interfere with the rest of the stack

I guess one possible way would be to implement the generator as a set of maven
artifacts, that could be installed/consumed transparently by just declaring a
dependency e.g as proposed via top-level component.

Another way is to have a new package like we do for bigtop-utils and such.

Perhaps this discussion should be moved to JIRA or shall we continue on the
dev@ ??

Cos

On Sun, Aug 23, 2015 at 11:53AM, RJ Nowling wrote:
> Hi BigTop,
> 
> I had a discussion with Jay yesterday, we'd like to propose a new component
> for BigTop: BigTop Data Generators.
> 
> BigTop Data Generators would consist of a common set of libraries for
> building data generators and three example data generators:
> 
>     * BigPetStore transaction generator (moved from BigPetStore)
>     * BigTop Bazaar -- attendee movement and interactions with booths on a
> showroom floor, at a conference, or at a mall
>     * BigTop Weatherman -- stochastic weather simulation (temperature, wind
> speed, wind chill, rainfall, etc.) per zip code.  (From a model trained on
> NOAA historical weather data)
> 
> We believe that creating a common set of libraries will have several
> benefits including:
> 
>      * Easier for others to build their own data generators
>      * Make data generators smaller and easier to maintain
>      * Share improvements across the data generators
> 
> More details on the libraries are below.
> 
> BigPetStore will be continue to focus on building  and maintaining
> blueprints, powered by the BigTop Data Generators.
> 
> Our vision is that we get all of Apache coming to BigTop for tools for
> building better, more comprehensive blueprints.  We want to support these
> efforts through data generators and the initial set of blueprint we've been
> building.
> 
> If the community is generally in support of this, I can create a top-level
> "bigtop-data-generators" directory and put the data generators and
> libraries in there.
> 
> Thanks!
> 
> RJ
> 
> 
> -------
> Library details:
> 
> So far, I've extracted the following common libraries:
> 
>      * Samplers -- provides classes for PDFs and various samplers
>      * Name generator -- data set and samplers for generating names
>      * Location data set -- data set and classes for US zip codes, their
> GPS coordinates, median house hold incomes, and population sizes
>      * Product generator -- library for enumerating products from a
> specification file.  Comes with default specifications for BigPetStore
> 
> I also expect that I'll add libraries for:
> 
>       * Particle simulation -- customer movement in a room
>       * Latent factor model generation -- generate latent factors and
> customer weights to create something like MovieLens data.  Used in Bazaar
> for booth preferences and potentially in BigPetStore for customer item
> preferences
> 
> Most of these libraries came out of the BigPetStore data generator but the
> other generators have been refactored to be based off the standard set of
> libraries.

Re: Proposal for "BigTop Data Generators"

Posted by RJ Nowling <rn...@gmail.com>.
I created a JIRA (BIGTOP-1982) with various subtasks to track the overall
effort.  If someone would be willing to start reviewing my changes, please
start with the pull request for BIGTOP-1983:

https://github.com/apache/bigtop/pull/33

Thanks!

On Mon, Aug 24, 2015 at 8:27 AM, RJ Nowling <rn...@gmail.com> wrote:

> Agreed -- would be good to use data generators in the smoke tests.
>
> With Youngwoo's work on adding Spark smoke tests, I don't think add BPS
> Spark will be too hard.  We just need to figure out to make the BPS Spark
> jar available for the smoke tests.
>
>
>
> On Mon, Aug 24, 2015 at 6:33 AM, Evans Ye <ev...@apache.org> wrote:
>
>> +1.
>> And definitly good to have at least one demo case in our smoke test for
>> each
>> so that people can fully understand what data/format it's generating and
>> how to process.
>>
>> 2015-08-24 12:38 GMT+08:00 김영우 (Youngwoo Kim) <wa...@gmail.com>:
>>
>> > +1
>> >
>> > I hope Bigtop DG would help all over the Bigtop infra -- blueprints,
>> > smokes, benchmarks and etc.
>> >
>> > Thanks,
>> > Youngwoo
>> >
>> > On Mon, Aug 24, 2015 at 1:53 AM, RJ Nowling <rn...@gmail.com> wrote:
>> >
>> > > Hi BigTop,
>> > >
>> > > I had a discussion with Jay yesterday, we'd like to propose a new
>> > component
>> > > for BigTop: BigTop Data Generators.
>> > >
>> > > BigTop Data Generators would consist of a common set of libraries for
>> > > building data generators and three example data generators:
>> > >
>> > >     * BigPetStore transaction generator (moved from BigPetStore)
>> > >     * BigTop Bazaar -- attendee movement and interactions with booths
>> on
>> > a
>> > > showroom floor, at a conference, or at a mall
>> > >     * BigTop Weatherman -- stochastic weather simulation (temperature,
>> > wind
>> > > speed, wind chill, rainfall, etc.) per zip code.  (From a model
>> trained
>> > on
>> > > NOAA historical weather data)
>> > >
>> > > We believe that creating a common set of libraries will have several
>> > > benefits including:
>> > >
>> > >      * Easier for others to build their own data generators
>> > >      * Make data generators smaller and easier to maintain
>> > >      * Share improvements across the data generators
>> > >
>> > > More details on the libraries are below.
>> > >
>> > > BigPetStore will be continue to focus on building  and maintaining
>> > > blueprints, powered by the BigTop Data Generators.
>> > >
>> > > Our vision is that we get all of Apache coming to BigTop for tools for
>> > > building better, more comprehensive blueprints.  We want to support
>> these
>> > > efforts through data generators and the initial set of blueprint we've
>> > been
>> > > building.
>> > >
>> > > If the community is generally in support of this, I can create a
>> > top-level
>> > > "bigtop-data-generators" directory and put the data generators and
>> > > libraries in there.
>> > >
>> > > Thanks!
>> > >
>> > > RJ
>> > >
>> > >
>> > > -------
>> > > Library details:
>> > >
>> > > So far, I've extracted the following common libraries:
>> > >
>> > >      * Samplers -- provides classes for PDFs and various samplers
>> > >      * Name generator -- data set and samplers for generating names
>> > >      * Location data set -- data set and classes for US zip codes,
>> their
>> > > GPS coordinates, median house hold incomes, and population sizes
>> > >      * Product generator -- library for enumerating products from a
>> > > specification file.  Comes with default specifications for BigPetStore
>> > >
>> > > I also expect that I'll add libraries for:
>> > >
>> > >       * Particle simulation -- customer movement in a room
>> > >       * Latent factor model generation -- generate latent factors and
>> > > customer weights to create something like MovieLens data.  Used in
>> Bazaar
>> > > for booth preferences and potentially in BigPetStore for customer item
>> > > preferences
>> > >
>> > > Most of these libraries came out of the BigPetStore data generator but
>> > the
>> > > other generators have been refactored to be based off the standard
>> set of
>> > > libraries.
>> > >
>> >
>>
>
>

Re: Proposal for "BigTop Data Generators"

Posted by RJ Nowling <rn...@gmail.com>.
Agreed -- would be good to use data generators in the smoke tests.

With Youngwoo's work on adding Spark smoke tests, I don't think add BPS
Spark will be too hard.  We just need to figure out to make the BPS Spark
jar available for the smoke tests.



On Mon, Aug 24, 2015 at 6:33 AM, Evans Ye <ev...@apache.org> wrote:

> +1.
> And definitly good to have at least one demo case in our smoke test for
> each
> so that people can fully understand what data/format it's generating and
> how to process.
>
> 2015-08-24 12:38 GMT+08:00 김영우 (Youngwoo Kim) <wa...@gmail.com>:
>
> > +1
> >
> > I hope Bigtop DG would help all over the Bigtop infra -- blueprints,
> > smokes, benchmarks and etc.
> >
> > Thanks,
> > Youngwoo
> >
> > On Mon, Aug 24, 2015 at 1:53 AM, RJ Nowling <rn...@gmail.com> wrote:
> >
> > > Hi BigTop,
> > >
> > > I had a discussion with Jay yesterday, we'd like to propose a new
> > component
> > > for BigTop: BigTop Data Generators.
> > >
> > > BigTop Data Generators would consist of a common set of libraries for
> > > building data generators and three example data generators:
> > >
> > >     * BigPetStore transaction generator (moved from BigPetStore)
> > >     * BigTop Bazaar -- attendee movement and interactions with booths
> on
> > a
> > > showroom floor, at a conference, or at a mall
> > >     * BigTop Weatherman -- stochastic weather simulation (temperature,
> > wind
> > > speed, wind chill, rainfall, etc.) per zip code.  (From a model trained
> > on
> > > NOAA historical weather data)
> > >
> > > We believe that creating a common set of libraries will have several
> > > benefits including:
> > >
> > >      * Easier for others to build their own data generators
> > >      * Make data generators smaller and easier to maintain
> > >      * Share improvements across the data generators
> > >
> > > More details on the libraries are below.
> > >
> > > BigPetStore will be continue to focus on building  and maintaining
> > > blueprints, powered by the BigTop Data Generators.
> > >
> > > Our vision is that we get all of Apache coming to BigTop for tools for
> > > building better, more comprehensive blueprints.  We want to support
> these
> > > efforts through data generators and the initial set of blueprint we've
> > been
> > > building.
> > >
> > > If the community is generally in support of this, I can create a
> > top-level
> > > "bigtop-data-generators" directory and put the data generators and
> > > libraries in there.
> > >
> > > Thanks!
> > >
> > > RJ
> > >
> > >
> > > -------
> > > Library details:
> > >
> > > So far, I've extracted the following common libraries:
> > >
> > >      * Samplers -- provides classes for PDFs and various samplers
> > >      * Name generator -- data set and samplers for generating names
> > >      * Location data set -- data set and classes for US zip codes,
> their
> > > GPS coordinates, median house hold incomes, and population sizes
> > >      * Product generator -- library for enumerating products from a
> > > specification file.  Comes with default specifications for BigPetStore
> > >
> > > I also expect that I'll add libraries for:
> > >
> > >       * Particle simulation -- customer movement in a room
> > >       * Latent factor model generation -- generate latent factors and
> > > customer weights to create something like MovieLens data.  Used in
> Bazaar
> > > for booth preferences and potentially in BigPetStore for customer item
> > > preferences
> > >
> > > Most of these libraries came out of the BigPetStore data generator but
> > the
> > > other generators have been refactored to be based off the standard set
> of
> > > libraries.
> > >
> >
>

Re: Proposal for "BigTop Data Generators"

Posted by Evans Ye <ev...@apache.org>.
+1.
And definitly good to have at least one demo case in our smoke test for
each
so that people can fully understand what data/format it's generating and
how to process.

2015-08-24 12:38 GMT+08:00 김영우 (Youngwoo Kim) <wa...@gmail.com>:

> +1
>
> I hope Bigtop DG would help all over the Bigtop infra -- blueprints,
> smokes, benchmarks and etc.
>
> Thanks,
> Youngwoo
>
> On Mon, Aug 24, 2015 at 1:53 AM, RJ Nowling <rn...@gmail.com> wrote:
>
> > Hi BigTop,
> >
> > I had a discussion with Jay yesterday, we'd like to propose a new
> component
> > for BigTop: BigTop Data Generators.
> >
> > BigTop Data Generators would consist of a common set of libraries for
> > building data generators and three example data generators:
> >
> >     * BigPetStore transaction generator (moved from BigPetStore)
> >     * BigTop Bazaar -- attendee movement and interactions with booths on
> a
> > showroom floor, at a conference, or at a mall
> >     * BigTop Weatherman -- stochastic weather simulation (temperature,
> wind
> > speed, wind chill, rainfall, etc.) per zip code.  (From a model trained
> on
> > NOAA historical weather data)
> >
> > We believe that creating a common set of libraries will have several
> > benefits including:
> >
> >      * Easier for others to build their own data generators
> >      * Make data generators smaller and easier to maintain
> >      * Share improvements across the data generators
> >
> > More details on the libraries are below.
> >
> > BigPetStore will be continue to focus on building  and maintaining
> > blueprints, powered by the BigTop Data Generators.
> >
> > Our vision is that we get all of Apache coming to BigTop for tools for
> > building better, more comprehensive blueprints.  We want to support these
> > efforts through data generators and the initial set of blueprint we've
> been
> > building.
> >
> > If the community is generally in support of this, I can create a
> top-level
> > "bigtop-data-generators" directory and put the data generators and
> > libraries in there.
> >
> > Thanks!
> >
> > RJ
> >
> >
> > -------
> > Library details:
> >
> > So far, I've extracted the following common libraries:
> >
> >      * Samplers -- provides classes for PDFs and various samplers
> >      * Name generator -- data set and samplers for generating names
> >      * Location data set -- data set and classes for US zip codes, their
> > GPS coordinates, median house hold incomes, and population sizes
> >      * Product generator -- library for enumerating products from a
> > specification file.  Comes with default specifications for BigPetStore
> >
> > I also expect that I'll add libraries for:
> >
> >       * Particle simulation -- customer movement in a room
> >       * Latent factor model generation -- generate latent factors and
> > customer weights to create something like MovieLens data.  Used in Bazaar
> > for booth preferences and potentially in BigPetStore for customer item
> > preferences
> >
> > Most of these libraries came out of the BigPetStore data generator but
> the
> > other generators have been refactored to be based off the standard set of
> > libraries.
> >
>

Re: Proposal for "BigTop Data Generators"

Posted by "김영우 (Youngwoo Kim)" <wa...@gmail.com>.
+1

I hope Bigtop DG would help all over the Bigtop infra -- blueprints,
smokes, benchmarks and etc.

Thanks,
Youngwoo

On Mon, Aug 24, 2015 at 1:53 AM, RJ Nowling <rn...@gmail.com> wrote:

> Hi BigTop,
>
> I had a discussion with Jay yesterday, we'd like to propose a new component
> for BigTop: BigTop Data Generators.
>
> BigTop Data Generators would consist of a common set of libraries for
> building data generators and three example data generators:
>
>     * BigPetStore transaction generator (moved from BigPetStore)
>     * BigTop Bazaar -- attendee movement and interactions with booths on a
> showroom floor, at a conference, or at a mall
>     * BigTop Weatherman -- stochastic weather simulation (temperature, wind
> speed, wind chill, rainfall, etc.) per zip code.  (From a model trained on
> NOAA historical weather data)
>
> We believe that creating a common set of libraries will have several
> benefits including:
>
>      * Easier for others to build their own data generators
>      * Make data generators smaller and easier to maintain
>      * Share improvements across the data generators
>
> More details on the libraries are below.
>
> BigPetStore will be continue to focus on building  and maintaining
> blueprints, powered by the BigTop Data Generators.
>
> Our vision is that we get all of Apache coming to BigTop for tools for
> building better, more comprehensive blueprints.  We want to support these
> efforts through data generators and the initial set of blueprint we've been
> building.
>
> If the community is generally in support of this, I can create a top-level
> "bigtop-data-generators" directory and put the data generators and
> libraries in there.
>
> Thanks!
>
> RJ
>
>
> -------
> Library details:
>
> So far, I've extracted the following common libraries:
>
>      * Samplers -- provides classes for PDFs and various samplers
>      * Name generator -- data set and samplers for generating names
>      * Location data set -- data set and classes for US zip codes, their
> GPS coordinates, median house hold incomes, and population sizes
>      * Product generator -- library for enumerating products from a
> specification file.  Comes with default specifications for BigPetStore
>
> I also expect that I'll add libraries for:
>
>       * Particle simulation -- customer movement in a room
>       * Latent factor model generation -- generate latent factors and
> customer weights to create something like MovieLens data.  Used in Bazaar
> for booth preferences and potentially in BigPetStore for customer item
> preferences
>
> Most of these libraries came out of the BigPetStore data generator but the
> other generators have been refactored to be based off the standard set of
> libraries.
>

Re: Proposal for "BigTop Data Generators"

Posted by Andrew Musselman <an...@gmail.com>.
+1 (non-binding :))

On Sunday, August 23, 2015, Suneel Marthi <su...@gmail.com> wrote:

> +1 from me too, unlike Jay I am unbiased. great idea RJ.
>
> On Sun, Aug 23, 2015 at 1:42 PM, jay vyas <jayunit100.apache@gmail.com
> <javascript:;>>
> wrote:
>
> > +1 from me ; but of course im a little biased.  Thanks for summing  up
> this
> > idea RJ!
> >
> >
> >
> > On Sun, Aug 23, 2015 at 12:53 PM, RJ Nowling <rnowling@gmail.com
> <javascript:;>> wrote:
> >
> > > Hi BigTop,
> > >
> > > I had a discussion with Jay yesterday, we'd like to propose a new
> > component
> > > for BigTop: BigTop Data Generators.
> > >
> > > BigTop Data Generators would consist of a common set of libraries for
> > > building data generators and three example data generators:
> > >
> > >     * BigPetStore transaction generator (moved from BigPetStore)
> > >     * BigTop Bazaar -- attendee movement and interactions with booths
> on
> > a
> > > showroom floor, at a conference, or at a mall
> > >     * BigTop Weatherman -- stochastic weather simulation (temperature,
> > wind
> > > speed, wind chill, rainfall, etc.) per zip code.  (From a model trained
> > on
> > > NOAA historical weather data)
> > >
> > > We believe that creating a common set of libraries will have several
> > > benefits including:
> > >
> > >      * Easier for others to build their own data generators
> > >      * Make data generators smaller and easier to maintain
> > >      * Share improvements across the data generators
> > >
> > > More details on the libraries are below.
> > >
> > > BigPetStore will be continue to focus on building  and maintaining
> > > blueprints, powered by the BigTop Data Generators.
> > >
> > > Our vision is that we get all of Apache coming to BigTop for tools for
> > > building better, more comprehensive blueprints.  We want to support
> these
> > > efforts through data generators and the initial set of blueprint we've
> > been
> > > building.
> > >
> > > If the community is generally in support of this, I can create a
> > top-level
> > > "bigtop-data-generators" directory and put the data generators and
> > > libraries in there.
> > >
> > > Thanks!
> > >
> > > RJ
> > >
> > >
> > > -------
> > > Library details:
> > >
> > > So far, I've extracted the following common libraries:
> > >
> > >      * Samplers -- provides classes for PDFs and various samplers
> > >      * Name generator -- data set and samplers for generating names
> > >      * Location data set -- data set and classes for US zip codes,
> their
> > > GPS coordinates, median house hold incomes, and population sizes
> > >      * Product generator -- library for enumerating products from a
> > > specification file.  Comes with default specifications for BigPetStore
> > >
> > > I also expect that I'll add libraries for:
> > >
> > >       * Particle simulation -- customer movement in a room
> > >       * Latent factor model generation -- generate latent factors and
> > > customer weights to create something like MovieLens data.  Used in
> Bazaar
> > > for booth preferences and potentially in BigPetStore for customer item
> > > preferences
> > >
> > > Most of these libraries came out of the BigPetStore data generator but
> > the
> > > other generators have been refactored to be based off the standard set
> of
> > > libraries.
> > >
> >
> >
> >
> > --
> > jay vyas
> >
>

Re: Proposal for "BigTop Data Generators"

Posted by Suneel Marthi <su...@gmail.com>.
+1 from me too, unlike Jay I am unbiased. great idea RJ.

On Sun, Aug 23, 2015 at 1:42 PM, jay vyas <ja...@gmail.com>
wrote:

> +1 from me ; but of course im a little biased.  Thanks for summing  up this
> idea RJ!
>
>
>
> On Sun, Aug 23, 2015 at 12:53 PM, RJ Nowling <rn...@gmail.com> wrote:
>
> > Hi BigTop,
> >
> > I had a discussion with Jay yesterday, we'd like to propose a new
> component
> > for BigTop: BigTop Data Generators.
> >
> > BigTop Data Generators would consist of a common set of libraries for
> > building data generators and three example data generators:
> >
> >     * BigPetStore transaction generator (moved from BigPetStore)
> >     * BigTop Bazaar -- attendee movement and interactions with booths on
> a
> > showroom floor, at a conference, or at a mall
> >     * BigTop Weatherman -- stochastic weather simulation (temperature,
> wind
> > speed, wind chill, rainfall, etc.) per zip code.  (From a model trained
> on
> > NOAA historical weather data)
> >
> > We believe that creating a common set of libraries will have several
> > benefits including:
> >
> >      * Easier for others to build their own data generators
> >      * Make data generators smaller and easier to maintain
> >      * Share improvements across the data generators
> >
> > More details on the libraries are below.
> >
> > BigPetStore will be continue to focus on building  and maintaining
> > blueprints, powered by the BigTop Data Generators.
> >
> > Our vision is that we get all of Apache coming to BigTop for tools for
> > building better, more comprehensive blueprints.  We want to support these
> > efforts through data generators and the initial set of blueprint we've
> been
> > building.
> >
> > If the community is generally in support of this, I can create a
> top-level
> > "bigtop-data-generators" directory and put the data generators and
> > libraries in there.
> >
> > Thanks!
> >
> > RJ
> >
> >
> > -------
> > Library details:
> >
> > So far, I've extracted the following common libraries:
> >
> >      * Samplers -- provides classes for PDFs and various samplers
> >      * Name generator -- data set and samplers for generating names
> >      * Location data set -- data set and classes for US zip codes, their
> > GPS coordinates, median house hold incomes, and population sizes
> >      * Product generator -- library for enumerating products from a
> > specification file.  Comes with default specifications for BigPetStore
> >
> > I also expect that I'll add libraries for:
> >
> >       * Particle simulation -- customer movement in a room
> >       * Latent factor model generation -- generate latent factors and
> > customer weights to create something like MovieLens data.  Used in Bazaar
> > for booth preferences and potentially in BigPetStore for customer item
> > preferences
> >
> > Most of these libraries came out of the BigPetStore data generator but
> the
> > other generators have been refactored to be based off the standard set of
> > libraries.
> >
>
>
>
> --
> jay vyas
>

Re: Proposal for "BigTop Data Generators"

Posted by jay vyas <ja...@gmail.com>.
+1 from me ; but of course im a little biased.  Thanks for summing  up this
idea RJ!



On Sun, Aug 23, 2015 at 12:53 PM, RJ Nowling <rn...@gmail.com> wrote:

> Hi BigTop,
>
> I had a discussion with Jay yesterday, we'd like to propose a new component
> for BigTop: BigTop Data Generators.
>
> BigTop Data Generators would consist of a common set of libraries for
> building data generators and three example data generators:
>
>     * BigPetStore transaction generator (moved from BigPetStore)
>     * BigTop Bazaar -- attendee movement and interactions with booths on a
> showroom floor, at a conference, or at a mall
>     * BigTop Weatherman -- stochastic weather simulation (temperature, wind
> speed, wind chill, rainfall, etc.) per zip code.  (From a model trained on
> NOAA historical weather data)
>
> We believe that creating a common set of libraries will have several
> benefits including:
>
>      * Easier for others to build their own data generators
>      * Make data generators smaller and easier to maintain
>      * Share improvements across the data generators
>
> More details on the libraries are below.
>
> BigPetStore will be continue to focus on building  and maintaining
> blueprints, powered by the BigTop Data Generators.
>
> Our vision is that we get all of Apache coming to BigTop for tools for
> building better, more comprehensive blueprints.  We want to support these
> efforts through data generators and the initial set of blueprint we've been
> building.
>
> If the community is generally in support of this, I can create a top-level
> "bigtop-data-generators" directory and put the data generators and
> libraries in there.
>
> Thanks!
>
> RJ
>
>
> -------
> Library details:
>
> So far, I've extracted the following common libraries:
>
>      * Samplers -- provides classes for PDFs and various samplers
>      * Name generator -- data set and samplers for generating names
>      * Location data set -- data set and classes for US zip codes, their
> GPS coordinates, median house hold incomes, and population sizes
>      * Product generator -- library for enumerating products from a
> specification file.  Comes with default specifications for BigPetStore
>
> I also expect that I'll add libraries for:
>
>       * Particle simulation -- customer movement in a room
>       * Latent factor model generation -- generate latent factors and
> customer weights to create something like MovieLens data.  Used in Bazaar
> for booth preferences and potentially in BigPetStore for customer item
> preferences
>
> Most of these libraries came out of the BigPetStore data generator but the
> other generators have been refactored to be based off the standard set of
> libraries.
>



-- 
jay vyas