New jiras (thanks) :)

[Jay Vyas <ja...@gmail.com>]

Thanks for all these new jiras focused on end users roman / cons !!(documentation, simplifying hive, jenkins)... 

The hive one is sooo true: it's annoying that I need a hive server running to do smoke tests. Would like to just smoke test hive ETL without the extra stuff for simpler clusters.

I think in general there should be different types of smokes so that people don't require a whole installed infrastructure to run smokes.  Although the vendor distros expect all services running, individual users or small startups don't necessarily run all the ecosystem services , they just run the bare Apis, and smoke testing those is still of value on a cluster. 

In our cluster we have a simple hive ETL that just runs a .q file which imports some tables. We package it by including shell scripts directly in the test-artifacts/pom file, and trigger those scripts from itest.

We do that instead of running the complex hive ETL. For pig we do something's similar , since we want to test basic pig API and not only the new .11 Apis, which are the only one supported currently in bigtop .

Anyways.. Thanks for keeping an eye out in this! Jiras like this will make bigtop more accessible to the masses !:)



Sent from my iPhone

Re: New jiras (thanks) :)

[Roman Shaposhnik <ro...@shaposhnik.org>]

Hi Jay,

thanks for the kind words. Here's one thing that I wanted to comment on:

On Wed, Sep 25, 2013 at 6:47 AM, Jay Vyas <ja...@gmail.com> wrote:
> Thanks for all these new jiras focused on end users roman / cons !!(documentation, simplifying hive, jenkins)...
>
> The hive one is sooo true: it's annoying that I need a hive server running to do smoke tests.
> Would like to just smoke test hive ETL without the extra stuff for simpler clusters.
>
> I think in general there should be different types of smokes so that people don't require a whole
> installed infrastructure to run smokes.  Although the vendor distros expect all services running,
> individual users or small startups don't necessarily run all the ecosystem services ,
> they just run the bare Apis, and smoke testing those is still of value on a cluster.

I think this boils down to a few fundamental improvements that we'd have to have
in our test framework to make it flexible:
    * let tests declare (via annotations or somehow else) what
environment they expect
    * have more topologies defined via puppet or similar provisioning frameworks
    * embrace containers Docker.io or similar to not require a full fledged VMs

The above 3 would get us a lot of mileage.

> We do that instead of running the complex hive ETL. For pig we do something's similar ,
> since we want to test basic pig API and not only the new .11 Apis, which are the only one
> supported currently in bigtop .
>
> Anyways.. Thanks for keeping an eye out in this! Jiras like this will make bigtop more
> accessible to the masses !:)

I think its all about more and more folks feeling compelled to contribute small
incremental improvements. Lets hope this will continue!

Thanks,
Roman.