[jira] [Created] (KUDU-1817) Failed to execute spark-shell with kudu-spark2 package

["Jun He (JIRA)" <ji...@apache.org>]

Jun He created KUDU-1817:
----------------------------

             Summary: Failed to execute spark-shell with kudu-spark2 package
                 Key: KUDU-1817
                 URL: https://issues.apache.org/jira/browse/KUDU-1817
             Project: Kudu
          Issue Type: Bug
          Components: client, spark
    Affects Versions: 1.1.0
            Reporter: Jun He
            Priority: Blocker


Tried to run 
{code}spark-shell --packages org.apache.kudu:kudu-spark2_2.11:1.1.0{code} and failed with the below error message:

{quote}
:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS

Exception in thread "main" java.lang.RuntimeException: [unresolved dependency: org.apache.kudu#kudu-spark2_2.11;1.1.0: java.text.ParseException: inconsistent module descriptor file found in 'https://repo1.maven.org/maven2/org/apache/kudu/kudu-spark2_2.11/1.1.0/kudu-spark2_2.11-1.1.0.pom': bad module name: expected='kudu-spark2_2.11' found='kudu-spark_2.10'; ]

at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1076)

at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:294)

at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:158)

at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)

at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
{quote}

I checked http://repo1.maven.org/maven2/org/apache/kudu/kudu-spark2_2.11/1.1.0/kudu-spark2_2.11-1.1.0.pom.

The artifactId in pom.xml is 
{code}kudu-${spark.version.label}_${scala.binary.version}{code}

As the properties are 
{code}
<properties>
<scala.version>2.10.4</scala.version>
<compat.src>src/main/spark1</compat.src>
<scala.binary.version>2.10</scala.binary.version>
<spark.version.label>spark</spark.version.label>
<top.dir>${project.basedir}/..</top.dir>
<spark.version>1.6.1</spark.version>
</properties>
{code}
So it will be translated to `kafka-spark_2.10`.

By checking the code, I think this is caused by how maven shade plugin generates pom file.

During running {code}mvn clean package -P spark2_2.11{code}

maven shade plugin will generate a dependency-reduced-pom.xml, which will be used later in the release pom file.

In dependency-reduced-pom.xml, the shade plugin will only explicitly parse properties for all dependencies. However, it won't parse artifact or plugin configurations. So we see {code}kudu-${spark.version.label}_${scala.binary.version}{code} in the release.

It will cause the problem when other applications try to load package because the properties values are for spark1 and profile values for spark2 won't be used when loading the package.

As pom files are supposed to be static in maven, a quick fix will be creating two new modules (spark1 and spark2) to build them separately.

Please let me know your comments. Thanks.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)