[GitHub] [bigtop] guyuqi commented on a change in pull request #697: BIGTOP-3443: Building Spark fails on Ubuntu 16.04

[GitBox <gi...@apache.org>]

guyuqi commented on a change in pull request #697:
URL: https://github.com/apache/bigtop/pull/697#discussion_r521846933



##########
File path: bigtop-packages/src/common/spark/do-component-build
##########
@@ -23,7 +23,7 @@ BUILD_OPTS="-Divy.home=${HOME}/.ivy2 -Dsbt.ivy.home=${HOME}/.ivy2 -Duser.home=${
             -Dreactor.repo=file://${HOME}/.m2/repository \
             -Dhadoop.version=$HADOOP_VERSION \
             -Dyarn.version=$HADOOP_VERSION \
-            -Pyarn -Phadoop-2.7 \
+            -Pyarn -Phadoop-$HADOOP_VERSION \

Review comment:
       
   I met build issue with Hadoop-2.10.0:
   
   ```
   [ERROR] Failed to execute goal org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (process-resource-bundles) on project spark-network-yarn_2.11: Failed to resolve dependencies for one or more projects in the reactor. Reason: No versions are present in the repository for the artifact with a range [1.3.1,2.3]
   [ERROR]   net.minidev:json-smart:jar:null
   [ERROR]
   [ERROR] from the specified remote repositories:
   [ERROR]   gcs-maven-central-mirror (https://maven-central.storage-download.googleapis.com/repos/central/data/, releases=true, snapshots=false),
   [ERROR]   central (https://repo.maven.apache.org/maven2, releases=true, snapshots=false),
   [ERROR]   apache.snapshots (https://repository.apache.org/snapshots, releases=false, snapshots=true),
   [ERROR]   dynamodb-local-oregon (https://s3-us-west-2.amazonaws.com/dynamodb-local/release, releases=true, snapshots=true),
   [ERROR]   apache.snapshots.https (https://repository.apache.org/content/repositories/snapshots, releases=true, snapshots=true),
   [ERROR]   repository.jboss.org (https://repository.jboss.org/nexus/content/groups/public/, releases=true, snapshots=false)
   [ERROR] Path to dependency:
   [ERROR]         1) org.apache.spark:spark-network-yarn_2.11:jar:2.4.5
   [ERROR]         2) org.apache.hadoop:hadoop-client:jar:2.10.0
   [ERROR]         3) org.apache.hadoop:hadoop-common:jar:2.10.0
   [ERROR]         4) org.apache.hadoop:hadoop-auth:jar:2.10.0
   [ERROR]         5) com.nimbusds:nimbus-jose-jwt:jar:4.41.1
   [ERROR]
   [ERROR]
   [ERROR] -> [Help 1]
   [ERROR]
   [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
   [ERROR] Re-run Maven using the -X switch to enable full debug logging.
   [ERROR]
   [ERROR] For more information about the errors and possible solutions, please read the following articles:
   [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
   [ERROR]
   [ERROR] After correcting the problems, you can resume the build with the command
   [ERROR]   mvn <goals> -rf :spark-network-yarn_2.11
   debuild: fatal error at line 1376:
   
   ```
   Overriding the properties in the hadoop-2.7 profile doesn't work.(`-Phadoop-2.7`)
   `-Phadoop-$HADOOP_VERSION ` makes spark build ok.
   
   I didn't notice that the pom does not have profile ( hadoop-2.10.0 / hadoop-2.10.1) in https://github.com/apache/spark/blob/v2.4.5/pom.xml#L2740-L2775. 
   It's informative, thanks @iwasakims .
   
   I rebased the code (upgrading Hadoop-2.10.1) just now. And it seems `-Phadoop-2.7` works again. Thanks for your comments.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org