[jira] [Commented] (CARBONDATA-1919) sbt project with carbondata dependencies got error when compiling

["CHENYANG XU (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CARBONDATA-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16379887#comment-16379887 ] 

CHENYANG XU commented on CARBONDATA-1919:
-----------------------------------------

It seems that this is a problem related to maven, and can be found here:  [MNG-2971|https://issues.apache.org/jira/browse/MNG-2971]
Pom files of subproject which contains variables like ${scala.binary.version} will only be copied but not newly generated when executing maven install, and sbt doesn't know how to deal with them. Even maven may not correctly resolve these variables. 
Spark met the same problem once, and it leveraged maven-shade-plugin to force maven to build effective pom files when installing. Please see [SPARK-3812|https://issues.apache.org/jira/browse/SPARK-3812] .

{panel:title=Copied from pom.xml of spark-parent_2.11}
    <!--
      This is a dummy dependency that is used to trigger the maven-shade plugin so that Spark's
      published POMs are flattened and do not contain variables. Without this dependency, some
      subprojects' published POMs would contain variables like ${scala.binary.version} that will
      be substituted according to the default properties instead of the ones determined by the
      profiles that were active during publishing, causing the Scala 2.10 build's POMs to have 2.11
      dependencies due to the incorrect substitutions. By ensuring that maven-shade runs for all
      subprojects, we eliminate this problem because the substitutions are baked into the final POM.

      For more details, see SPARK-3812 and MNG-2971.
    -->
    <dependency>
      <groupId>org.spark-project.spark</groupId>
      <artifactId>unused</artifactId>
      <version>1.0.0</version>
    </dependency>
{panel}


> sbt project with carbondata dependencies got error when compiling
> -----------------------------------------------------------------
>
>                 Key: CARBONDATA-1919
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-1919
>             Project: CarbonData
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 1.2.0
>         Environment: sbt 0.13.15
>            Reporter: CHENYANG XU
>            Priority: Major
>
> In my sbt project, I added carbondata-spark2 dependency like below:
> libraryDependencies += "org.apache.carbondata" % "carbondata-spark2" % "1.2.0"
> Unfortunately error occured during compilation:
> {noformat}
> [warn] 	Note: Unresolved dependencies path:
> [warn] 		org.apache.spark:spark-sql_${scala.binary.version}:${spark.version}
> [warn] 		  +- org.apache.carbondata:carbondata-core:1.2.0
> [warn] 		  +- org.apache.carbondata:carbondata-processing:1.2.0
> [warn] 		  +- org.apache.carbondata:carbondata-hadoop:1.2.0
> [warn] 		  +- org.apache.carbondata:carbondata-spark-common:1.2.0
> [warn] 		  +- org.apache.carbondata:carbondata-spark2:1.2.0
> [warn] 		org.apache.spark:spark-hive-thriftserver_${scala.binary.version}:${spark.version}
> [warn] 		  +- org.apache.carbondata:carbondata-spark-common:1.2.0
> [warn] 		  +- org.apache.carbondata:carbondata-spark2:1.2.0
> [warn] 		org.apache.spark:spark-repl_${scala.binary.version}:${spark.version}
> [warn] 		  +- org.apache.carbondata:carbondata-spark2:1.2.0 
> [error] sbt.ResolveException: unresolved dependency: org.apache.spark#spark-sql_${scala.binary.version};${spark.version}: not found
> [error] unresolved dependency: org.apache.spark#spark-hive-thriftserver_${scala.binary.version};${spark.version}: not found
> [error] unresolved dependency: org.apache.spark#spark-repl_${scala.binary.version};${spark.version}: not found
> {noformat}
> The project has a lot of spark dependencies, and all versions of spark-sql can be found in my .ivy2 folder and .m2 folder. Also when I remove the carbondata dependency, everything works fine. An empty sbt project with only carbondata dependency has the same problem, but maven project can successfully resolve the dependency.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)