[jira] [Comment Edited] (BIGTOP-1222) Simplify and gradleize a subset of the bigtop smokes

["jay vyas (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BIGTOP-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14040283#comment-14040283 ] 

jay vyas edited comment on BIGTOP-1222 at 6/23/14 2:10 AM:
-----------------------------------------------------------

Okay ! I'm (finally) done testing the first iteration of these functional smokes 

0) They use gradle for compiling stnadalone classes and adding some light dependencies - but REQUIRE NO JAR files - so they are totally hackable and extensible by the community.
1) They work for *mahout, sqoop, flume, hive, pig, mapreduce*  
2) And they are 100% *HCFS compliant*. 
3) Did I say They require *NO JAR FILES* :) - 
4) They create an *EMBEDDED database for Sqoop ETL* (no more MySQL dependency), by leveraging the HSQL libs.
5) They *generate data* for the pig phase - and are a proper integration test (existing pig integration from pig.apache.org tests dont seem to be cluster scale smokes). 
6) The check for all environmental variables upfront in the gradle script.

Anyone wants to play around with this patch and leave feedback let me know.  Some cleanup is required... thanks.


was (Author: jayunit100):
Okay ! I'm done testing the first iteration of these functional smokes ! 

0) They use gradle for compiling stnadalone classes and adding some light dependencies - but REQUIRE NO JAR files - so they are totally hackable and extensible by the community.
1) They work for mahout sqoop flume hive pig mapreduce 
2) And they are 100% HCFS compliant. 
3) Did I see They require *NO JAR FILES* :) - 
4) They create an EMBEDDED database for Sqoop ETL (no more MySQL dependency), by leveraging the java de
5) They generate data for the pig phase - and are a proper integration test (existing pig integration from pig.apache.org tests dont seem to be cluster scale smokes). 
6) The check for all environmental variables upfront in the gradle script.

Anyone wants to play around with this patch and leave feedback let me know.  Some cleanup is required... thanks 

> Simplify and gradleize a subset of the bigtop smokes
> ----------------------------------------------------
>
>                 Key: BIGTOP-1222
>                 URL: https://issues.apache.org/jira/browse/BIGTOP-1222
>             Project: Bigtop
>          Issue Type: Improvement
>          Components: Build, Tests
>    Affects Versions: 0.7.0
>            Reporter: jay vyas
>            Assignee: jay vyas
>             Fix For: backlog
>
>         Attachments: BIGTOP-1222-2.patch, BIGTOP-1222.patch, BIGTOP-1222.patch, BIGTOP-1222.patch, BIGTOP-1222.patch, BIGTOP-1222.patch
>
>
> (Rewritten the description for clarity)
> We need an easier way to run bigtop smoke tests, and gradle provides this:
> 1) Easy to script/modify
> 2) Human readable
> 3) equally oriented towards both groovy and plain old java
> The advantage of this method to running smokes : 
> 1) No need to compile a jar : this is a costly step and not much value added, also creates indirection which can make debugging a broken test very hard.
> 2) Simple: A smoke test doesnt need to make low level API calls or be compiled against the right APIs - rather, it should test the end user interface ("hive -q  ....", "pig -x ....", "hadoop jar ....", and so on).  
> 3) Customizable:  The smoke tests shouldnt require users to have to write XML and debug environmental variables / grep around for System properties etc.  Rather, a high level controller should do all that checking for you.  
> The initial idea was to write a python/bash implementation wrapper of scripts, but that was replaced by the idea of using gradle.  The advantage of gradle is that we don't need to manually set the classpath and run groovy commands: Gradle wraps groovy scripts in their native java context quite nicely - but it doesnt add any other unnecessary overhead (xml, jar files, no need for complex xml tag wrappers for simple tasks - just plain groovy code).
> So, here the goal is just to create a nice, clean, extensible non-jar, non-API dependent gradle runner for the smoke tests which exersizes the hadoop cluster the same way a typical end-user would.



--
This message was sent by Atlassian JIRA
(v6.2#6252)