[jira] [Comment Edited] (BIGTOP-1222) Simplify and gradleize a subset of the bigtop smokes

["jay vyas (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BIGTOP-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993364#comment-13993364 ] 

jay vyas edited comment on BIGTOP-1222 at 5/10/14 3:44 AM:
-----------------------------------------------------------

Made some progress with a raw gradle test runner which runs pure groovy scripts, no jar required (couple of hacks required to comment out the jar copy resource directive and replace with a manual copy of the input tests files for TestHadoopExamples.groovy, but those will be easy to solve with an try/catch block update to TestHadoopExamples.groovy which i can put in the final patch. 

Very raw, just a prototype of how it might work. 

This snippet runs the hadoop examples tests, but you have to manually copy over data from the resources first and "comment out" the jar oriented build utilities hooks which extract the files from the jar. 

{noformat}
apply plugin: 'groovy'

repositories {
    mavenCentral()
}



dependencies {
    
    //needed to avoid groovy not on classpath error. 
    testCompile module('org.codehaus.groovy:groovy:1.8.0')
    testCompile group:'org.apache.bigtop.itest', name:'itest-common', version:'0.7.0',transitive:'true'
    testCompile group:'org.apache.hadoop',name:'hadoop-common', version:'2.0.6-alpha',transitive:'false'
    testCompile group:'org.apache.hadoop',name:'hadoop-hdfs', version:'2.1.0-beta',transitive:'false'
}

sourceSets {
 test {
        /**
        * This will put the input files into the jar. 
        * TODO: We should refactor tests, over time, not to require running from jar.
        */
  
        resources { 
              srcDirs = 
              ['/opt/bigtop/bigtop-tests/test-artifacts/hadoop/src/main/resources/',
               'conf/']
        }
        
        groovy {       
          srcDirs = ['/opt/bigtop/bigtop-tests/test-artifacts/hadoop/']
          exclude 'src/main/groovy/org/apache/bigtop/itest/hadoop/hdfs/**'
          exclude 'src/main/groovy/org/apache/bigtop/itest/hadoop/hdfs/**'
          exclude 'src/main/groovy/org/apache/bigtop/itest/hadoop/hcfs/**'
          exclude 'src/main/groovy/org/apache/bigtop/itest/hadoop/yarn/**'
        }
 }
}


test.doFirst {

    def vars = ["HADOOP_MAPRED_HOME"]

    vars.each() {
        def value = System.getenv("${it}")
        if(value == null || value == "null")
            throw new GradleException("undeclared env variable: ${it}")
        else 
            println("${it} = $value");
    }
}

{noformat} 


was (Author: jayunit100):
Made some progress with a raw gradle test runner which runs pure groovy scripts, no jar required (couple of hacks required to comment out the jar copy resource directive and replace with a manual copy of the input tests files for TestHadoopExamples.groovy, but those will be easy to solve with an try/catch block update to TestHadoopExamples.groovy which i can put in the final patch. 

Very raw, just a prototype of how it might work. 

This snippet runs the hadoop examples tests, but you have to manually copy over data from the resources first and "comment out" the jar oriented build utilities hooks which extract the files from the jar. 

{noformat}
apply plugin: 'groovy'

repositories {
    mavenCentral()
}

dependencies {
    
    //needed to avoid groovy not on classpath error. 
    testCompile module('org.codehaus.groovy:groovy:1.8.0')
    testCompile 'org.apache.bigtop.itest:itest-common:0.7.0'
    testCompile 'org.apache.hadoop:hadoop-common:2.0.6-alpha' 
    testCompile 'org.apache.hadoop:hadoop-hdfs:2.1.0-beta'
}

sourceSets {
 test {
        /**
        * This will put the input files into the jar. 
        * TODO: We should refactor tests, over time, not to require running from jar.
	*/
        resources { 
           srcDirs = ['/opt/bigtop/bigtop-tests/test-artifacts/hadoop/src/main/resources/']
        }   
        groovy {
           srcDirs = ['/opt/bigtop/bigtop-tests/test-artifacts/hadoop/']
	   exclude 'src/main/groovy/org/apache/bigtop/itest/hadoop/hdfs/**'
           exclude 'src/main/groovy/org/apache/bigtop/itest/hadoop/hcfs/**'
           exclude 'src/main/groovy/org/apache/bigtop/itest/hadoop/yarn/**'
        }
 }
}


test.doFirst {  

    def vars = ["HADOOP_MAPRED_HOME"]

    vars.each() {
        def value = System.getenv("${it}")
        if(value == null || value == "null")
            throw new GradleException("undeclared env variable: ${it}")
        else 
            println("${it} = $value");
    }
}


{noformat} 

> Simplify and gradleize a subset of the bigtop smokes
> ----------------------------------------------------
>
>                 Key: BIGTOP-1222
>                 URL: https://issues.apache.org/jira/browse/BIGTOP-1222
>             Project: Bigtop
>          Issue Type: Improvement
>          Components: Build
>    Affects Versions: 0.7.0
>            Reporter: jay vyas
>             Fix For: backlog
>
>
> Currently, there is a JIRA underway to make running the maven based smoke tests easier:  BIGTOP-1195.  
> Eventually, however, maybe we could run these smokes from gradle.    I think that will obviate BIGTOP-1195 (Although i still assert a bash driver is a big win/gain for bigtop's goals : which are to unify the hadoop packaging and deployment paradigm).
> - run the smokes using a simple gradle goal
> - smokes should be easily runnable as scripts, with no need for jar file intermediates.
> - The bash driver for BIGTOP-1195 (if accepted, still under debate) should be upgraded to use the new gradle smokes
> - Delete old maven smokes. 
> This might be a little ambitious, if so others chime in.  I'm not a gradle/groovy expert but getting more well versed.  



--
This message was sent by Atlassian JIRA
(v6.2#6252)