Re: Bigtop: generating fake data

[Bruno Mahé <bm...@apache.org>]

On 02/15/2014 06:19 AM, Jay Vyas wrote:
> Hi bigtop.  Are we interested in maintaining our own infra for generating fake data , rather than relying on and downloading external data sources for smokes?  Fake data is great for testing I think...
>
> In bigpetstore I'm generating fake data , written a lot of code to do this in the custom input formats.... but I just found :
>
> http://codearte.github.io/jfairy/
>
> Which is a groovy tool for doing the same....
>
>    I wonder wether generating fake data for testing big data should be a first-class part of bigtop ?  Would others use a utility or just me ?
>
> It might be another useful artifact for the community especially for bigpetstore but also for testing a variety of other machine learning related projects....
>
> I think it's bad to rely on external websites for our tests, maybe in time we could move over to our in internally curated/generated data sets , and a data generation tool like the above moves us in that direction.
>

Hi Jay,

Generating fake data is an interesting idea and I don't see any reason 
to not use that when appropriate.

Regarding having our own framework vs re-using a library, it depends.
Writing our own framework is an option if there is no existing APLv2 
(-compatible?) library we can use or extend for our needs.
But writing code to facilitate such task would be welcome in any case. 
Ex: map/reduce jobs that use jfairy to generate TBs of data.


Thanks,
Bruno