Building Spark with Custom Hadoop Version

[Charles Wright <ch...@live.ca>]

Hello,

I have made some modifications to the YARN source code that I want to 
test with Spark, how do I do this? I know that I need to include my 
custom hadoop jar as a dependency but I don't know how to do this as I 
am not very familiar with maven.

Any help is appreciated.

Thanks,
Charles.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org

Re: Building Spark with Custom Hadoop Version

[Steve Loughran <st...@hortonworks.com>]

> On 4 Feb 2016, at 23:11, Ted Yu <yu...@gmail.com> wrote:
> 
> Assuming your change is based on hadoop-2 branch, you can use 'mvn install' command which would put artifacts under 2.8.0-SNAPSHOT subdir in your local maven repo.
> 
> Here is an example:
> ~/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.8.0-SNAPSHOT
> 
> Then you can use the following command to build Spark:
> 
> -Pyarn -Phadoop-2.4 -Dhadoop.version=2.8.0-SNAPSHOT
> 

Better to choose the hadoop-2.6 profile, e.g.

mvn test -Pyarn,hadoop-2.6 -Dhadoop.version=2.7.1  -pl yarn -Dtest=m  -DwildcardSuites=org.apache.spark.deploy.yarn.YarnClusterSuite

(the -Dtest= assignment skips all java tests)

if you are playing with -SNAPSHOT sourcess

(a) rebuild them every morning
(b) never do a test run that spans midnight

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org

Re: Building Spark with Custom Hadoop Version

[Steve Loughran <st...@hortonworks.com>]

> On 4 Feb 2016, at 23:11, Ted Yu <yu...@gmail.com> wrote:
> 
> Assuming your change is based on hadoop-2 branch, you can use 'mvn install' command which would put artifacts under 2.8.0-SNAPSHOT subdir in your local maven repo.
> 


+ generally, unless you want to run all the hadoop tests, set the  -DskipTests on the mvn commands. The HDFS ones take a while and can use up all your file handles.

mvn install -DskipTests

here's the aliases I use


export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m -Xms256m -Djava.awt.headless=true"
alias mi="mvn install -DskipTests"
alias mci="mvn clean install -DskipTests"
alias mvt="mvn test"
alias mvct="mvn clean test"
alias mvp="mvn package -DskipTests"
alias mvcp="mvn clean package -DskipTests"
alias mvnsite="mvn site:site -Dmaven.javadoc.skip=true -DskipTests"
alias mvndep="mvn dependency:tree -Dverbose"


mvndep > target/dependencies.txt is my command of choice to start working out where some random dependency is coming in from

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org

Re: Building Spark with Custom Hadoop Version

[Ted Yu <yu...@gmail.com>]

Assuming your change is based on hadoop-2 branch, you can use 'mvn install'
command which would put artifacts under 2.8.0-SNAPSHOT subdir in your local
maven repo.

Here is an example:
~/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.8.0-SNAPSHOT

Then you can use the following command to build Spark:

-Pyarn -Phadoop-2.4 -Dhadoop.version=2.8.0-SNAPSHOT

FYI

On Thu, Feb 4, 2016 at 3:03 PM, Charles Wright <ch...@live.ca>
wrote:

> Hello,
>
> I have made some modifications to the YARN source code that I want to test
> with Spark, how do I do this? I know that I need to include my custom
> hadoop jar as a dependency but I don't know how to do this as I am not very
> familiar with maven.
>
> Any help is appreciated.
>
> Thanks,
> Charles.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>