Big Top and Ecosystem coverage goals...

[Jay Vyas <ja...@gmail.com>]

Hi ! New to the big top project but super excited as we have alot of work
going on with HCFS file systems, and scaling out big top to run on broader
deployment targets is one of my goals...

Im noticing that the hive/hcatalog ecosystem are not explicitly listed in
the "how to install..." section, which mentions

hadoop\*
flume\*
mahout\*
oozie\*
whirr\*
hbase\*

(from
https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+Hadoop+distribution+from+Bigtop+0.5.0
).

1) I'm assuming im missing something... But is it the case that Big Top
doesnt test Hive/Hcatalog?
2) Is there a single web page that lists all of the HCatalog ecosystem test
coverage?  If not maybe I can dig through the source and compile one for
the site.

-- 
Jay Vyas
http://jayunit100.blogspot.com

Re: Big Top and Ecosystem coverage goals...

[Mark Grover <gr...@gmail.com>]

Hi Jay,
Welcome!

Bigtop 0.5.0 definitely supports hive, I have corrected the wiki page to
reflect that. Thanks for pointing that out.

The mandate is to have some integration tests for every component in the
Bigtop stack. You can find the Hive tests at
https://github.com/apache/bigtop/tree/master/bigtop-tests/test-artifacts/hive/src/main/groovy/org/apache/bigtop/itest/hivesmoke

The soon to be released Bigtop 0.6 release will have hcatalog. We will be
cutting out an RC real soon but you can grab the artifacts from
http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-0.6.0/? in the
meanwhile.
HCatalog's integration tests can be found at
https://github.com/apache/bigtop/tree/master/bigtop-tests/test-artifacts/hcatalog/src/main/groovy/org/apache/bigtop/itest/hcatalogsmoke

Please feel free to reach out if you have any further questions!

Mark

On Mon, Jun 10, 2013 at 11:02 AM, Jay Vyas <ja...@gmail.com> wrote:

> Hi ! New to the big top project but super excited as we have alot of work
> going on with HCFS file systems, and scaling out big top to run on broader
> deployment targets is one of my goals...
>
> Im noticing that the hive/hcatalog ecosystem are not explicitly listed in
> the "how to install..." section, which mentions
>
> hadoop\*
> flume\*
> mahout\*
> oozie\*
> whirr\*
> hbase\*
>
> (from
> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+Hadoop+distribution+from+Bigtop+0.5.0
> ).
>
> 1) I'm assuming im missing something... But is it the case that Big Top
> doesnt test Hive/Hcatalog?
> 2) Is there a single web page that lists all of the HCatalog ecosystem
> test coverage?  If not maybe I can dig through the source and compile one
> for the site.
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>

Re: Big Top and Ecosystem coverage goals...

[Roman Shaposhnik <rv...@apache.org>]

On Mon, Jun 10, 2013 at 11:17 AM, Roman Shaposhnik <ro...@shaposhnik.org> wrote:
> On Mon, Jun 10, 2013 at 11:02 AM, Jay Vyas <ja...@gmail.com> wrote:
>> (from
>> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+Hadoop+distribution+from+Bigtop+0.5.0).

Btw, we're about to start a release vote on Bigtop 0.6.0. It would
be really appreciated if you could install it:
   http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-0.6.0/

and let us know what you think. The installation procedure
is about the same -- you need the repo files for the OS of
your liking. E.g Fedora18:
   http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-0.6.0/label=fedora18/lastSuccessfulBuild/artifact/output/bigtop.repo

Thanks,
Roman.

Re: Big Top and Ecosystem coverage goals...

[Roman Shaposhnik <ro...@shaposhnik.org>]

On Mon, Jun 10, 2013 at 11:02 AM, Jay Vyas <ja...@gmail.com> wrote:
>
> Hi ! New to the big top project but super excited as we have alot of work
> going on with HCFS file systems, and scaling out big top to run on broader
> deployment targets is one of my goals...
>
> Im noticing that the hive/hcatalog ecosystem are not explicitly listed in
> the "how to install..." section, which mentions
>
> hadoop\*
> flume\*
> mahout\*
> oozie\*
> whirr\*
> hbase\*
>
> (from
> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+Hadoop+distribution+from+Bigtop+0.5.0).

This wiki is meant to be an example (although keeping our docs
up to date is something that we would really appreciate help
on). You can get a complete list of components currently packaged
by Bigtop by running a yum/apt-cache querry command.

> 1) I'm assuming im missing something... But is it the case that Big Top
> doesnt test Hive/Hcatalog?

Yes we do:
    http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-smokecluster/lastCompletedBuild/testReport/org.apache.bigtop.itest.hivesmoke/

> 2) Is there a single web page that lists all of the HCatalog ecosystem
> test coverage?  If not maybe I can dig through the source and compile one
> for the site.

We've got the following two jobs that run nightly:
   http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-smokecluster/
   http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-smokecluster-secure/

There's an additional chunk of integration tests that
we run on-demand. Let me know if you have interest
in any specific one.

Thanks,
Roman.