You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Mich Talebzadeh <mi...@gmail.com> on 2023/05/28 19:01:17 UTC
maven with Spark 3.4.0 fails compilation
This compilation works fine with SBT but fails with maven!
Spark version 3.4.0
Apache Maven 3.9.2 (c9616018c7a021c1c39be70fb2843d6f5f9b8a1c)
Java version: 11.0.1, vendor: Oracle Corporation, runtime: /opt/jdk-11.0.1
This from the pom.xml file
<project xmlns="https://maven.apache.org/POM/4.0.0" xmlns:xsi="
https://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="https://maven.apache.org/POM/4.0.0
https://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>spark</groupId>
<version>3.0</version>
<name>${project.artifactId}</name>
<properties>
<maven.compiler.source>1.7</maven.compiler.source>
<maven.compiler.target>1.7</maven.compiler.target>
<encoding>UTF-8</encoding>
<scala.version>2.12.17</scala.version>
<maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
</properties>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.13.6</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.13</artifactId>
<version>3.4.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.13</artifactId>
<version>3.4.0</version>
<scope>provided</scope>
The pom file is attached. These are the errors I am getting
[ERROR] error: error while loading package, class file
'/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/reflect/package.class)'
is broken
[ERROR] (class java.lang.RuntimeException/error reading Scala signature of
package.class: Scala signature package has wrong version
[ERROR] error: error while loading package, class file
'/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/package.class)'
is broken
[ERROR] (class java.lang.RuntimeException/error reading Scala signature of
package.class: Scala signature package has wrong version
[ERROR] error: error while loading package, class file
'/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/collection/package.class)'
is broken
[ERROR] (class java.lang.RuntimeException/error reading Scala signature of
package.class: Scala signature package has wrong version
[ERROR] three errors found
[ERROR] Failed to execute goal
org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project
scala: wrap: org.apache.commons.exec.ExecuteException: Process exited with
an error: 1(Exit value: 1) -> [Help 1]
Any ideas will be appreciated.
Mich Talebzadeh,
Lead Solutions Architect/Engineering Lead
Palantir Technologies Limited
London
United Kingdom
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
Re: Re: maven with Spark 3.4.0 fails compilation
Posted by Mich Talebzadeh <mi...@gmail.com>.
A good way to check the required Scala version for the Spark release is
ls $SPARK_HOME/jars | grep '^scala'
scala-collection-compat_2.12-2.7.0.jar
*scala-compiler-2.12.17.jar*
scala-library-2.12.17.jar
scala-parser-combinators_2.12-2.1.1.jar
scala-reflect-2.12.17.jar
scala-xml_2.12-2.1.0.jar
and scala-compiler-2.12.17.jar should tell you which version is required
HTH
Mich Talebzadeh,
Lead Solutions Architect/Engineering Lead
Palantir Technologies Limited
London
United Kingdom
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Tue, 30 May 2023 at 09:22, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Many thanks Lingzhe.
>
> I found the problem. The code was compiled against Scala 2.13 but it was
> running under Scala 2.12. So I changed it to compile under Scala 2.12.17
> and it worked fine.
>
> Thanks again to both Bjorn and Lingzhe.
>
> Regards
>
> Mich Talebzadeh,
> Lead Solutions Architect/Engineering Lead
> Palantir Technologies Limited
> London
> United Kingdom
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
> https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Tue, 30 May 2023 at 03:57, Lingzhe Sun <li...@hirain.com> wrote:
>
>> Hi Mich,
>> Your part of pom looks good. Another hint would be the exception message
>> itself:
>> Exception in thread "main" java.lang.NoSuchMethodError:
>> scala.package$.Seq()Lscala/collection/immutable/Seq$;
>> It basically suggest that your class could not be found in your compiled
>> code. It says that the Seq() from the scala.package whose return type is
>> scala.collection.immutable.Seq. The reason could be that you have mutiple
>> versions of same classes in your classpath, which typically dues to using
>> imcompatible library versions, like here:
>> [WARNING] Expected all dependencies to require Scala version: 2.13.8
>> [WARNING] spark:ReduceByKey:3.0 requires scala version: 2.13.8
>> [WARNING] org.scala-lang.modules:scala-parallel-collections_2.13:1.0.4
>> requires scala version: 2.13.6
>> [WARNING] Multiple versions of scala libraries detected!
>> You choose to use one version of the scala library(2.13.8) while one
>> dependency use another library(2.13.6). I'd suggest looking in the
>> following order:
>>
>> 1. Remove the library scala-parallel-collections_2.13:1.0.4 and see
>> if it works.
>> 2. Check the line (ReduceByKey.scala:23) to check what's the package
>> that the function belongs to. If it's using incompatible library, try
>> remove it and see if the problem persist.
>> 3. Build the artifact and check its output using IDEA build artifact
>> and option--copy to the output dir, which also could be a fast way to see
>> if you have included imcompatible library accidentally.
>> 4. Set your maven-scala-plugin to a newer version
>> <https://mvnrepository.com/artifact/net.alchim31.maven/scala-maven-plugin>
>> to avoid potential issues.
>> 5. Check the warnings below and set the maven.plugin.validation
>> property to VERBOSE to see if there're potential problems: [WARNING]
>> * org.scala-tools:maven-scala-plugin:2.15.2
>> [WARNING] * org.apache.maven.plugins:maven-compiler-plugin:3.10.1
>> [WARNING] * org.apache.maven.plugins:maven-shade-plugin:1.6
>> [WARNING] * org.apache.maven.plugins:maven-resources-plugin:3.3.0
>> [WARNING]
>> [WARNING] For more or less details, use 'maven.plugin.validation'
>> property with one of the values (case insensitive): [BRIEF, DEFAULT,
>> VERBOSE]
>> 6. Some code sharing would help, if it's possible.
>>
>> And TBH, I have no idea how could Seq() from a scala.package be used
>> cause there's no such package. There could be some underlying mechanism
>> that I do not know. Hope someone could kindly explain this.
>> ------------------------------
>> Best Regards!
>>
>> ...........................................................................
>> Lingzhe Sun
>> Hirain Technology
>>
>>
>> *From:* Bjørn Jørgensen <bj...@gmail.com>
>> *Date:* 2023-05-30 00:22
>> *To:* Mich Talebzadeh <mi...@gmail.com>
>> *CC:* Lingzhe Sun <li...@hirain.com>; User <us...@spark.apache.org>
>> *Subject:* Re: Re: maven with Spark 3.4.0 fails compilation
>> <scala.version>2.13.8</scala.version>
>> <https://github.com/apache/spark/blob/87a5442f7ed96b11051d8a9333476d080054e5a0/pom.xml#LL3606C46-L3606C46>
>>
>> you must change 2.13.6 to 2.13.8
>>
>>
>> man. 29. mai 2023 kl. 18:02 skrev Mich Talebzadeh <
>> mich.talebzadeh@gmail.com>:
>>
>>> Thanks everyone. Still not much progress :(. It is becoming a bit
>>> confusing as I am getting this error
>>>
>>> Compiling ReduceByKey
>>> [INFO] Scanning for projects...
>>> [INFO]
>>> [INFO] -------------------------< spark:ReduceByKey
>>> >--------------------------
>>> [INFO] Building ReduceByKey 3.0
>>> [INFO] from pom.xml
>>> [INFO] --------------------------------[ jar
>>> ]---------------------------------
>>> [INFO]
>>> [INFO] --- resources:3.3.0:resources (default-resources) @ ReduceByKey
>>> ---
>>> [WARNING] Using platform encoding (ANSI_X3.4-1968 actually) to copy
>>> filtered resources, i.e. build is platform dependent!
>>> [INFO] skip non existing resourceDirectory
>>> /data6/hduser/scala/ReduceByKey/src/main/resources
>>> [INFO]
>>> [INFO] --- compiler:3.10.1:compile (default-compile) @ ReduceByKey ---
>>> [INFO] Nothing to compile - all classes are up to date
>>> [INFO]
>>> [INFO] --- scala:2.15.2:compile (default) @ ReduceByKey ---
>>> [INFO] Checking for multiple versions of scala
>>> [WARNING] Expected all dependencies to require Scala version: 2.13.8
>>> [WARNING] spark:ReduceByKey:3.0 requires scala version: 2.13.8
>>> [WARNING] org.scala-lang.modules:scala-parallel-collections_2.13:1.0.4
>>> requires scala version: 2.13.6
>>> [WARNING] Multiple versions of scala libraries detected!
>>> [INFO] includes = [**/*.java,**/*.scala,]
>>> [INFO] excludes = []
>>> [INFO] Nothing to compile - all classes are up to date
>>> [INFO]
>>> [INFO] --- resources:3.3.0:testResources (default-testResources) @
>>> ReduceByKey ---
>>> [WARNING] Using platform encoding (ANSI_X3.4-1968 actually) to copy
>>> filtered resources, i.e. build is platform dependent!
>>> [INFO] skip non existing resourceDirectory
>>> /data6/hduser/scala/ReduceByKey/src/test/resources
>>> [INFO]
>>> [INFO] --- compiler:3.10.1:testCompile (default-testCompile) @
>>> ReduceByKey ---
>>> [INFO] No sources to compile
>>> [INFO]
>>> [INFO] --- surefire:3.0.0:test (default-test) @ ReduceByKey ---
>>> [INFO] No tests to run.
>>> [INFO]
>>> [INFO] --- jar:3.3.0:jar (default-jar) @ ReduceByKey ---
>>> [INFO] Building jar:
>>> /data6/hduser/scala/ReduceByKey/target/ReduceByKey.jar
>>> [INFO]
>>> [INFO] --- shade:1.6:shade (default) @ ReduceByKey ---
>>> [INFO] Including org.scala-lang:scala-library:jar:2.13.8 in the shaded
>>> jar.
>>> [INFO] Replacing original artifact with shaded artifact.
>>> [INFO] Replacing /data6/hduser/scala/ReduceByKey/target/ReduceByKey.jar
>>> with /data6/hduser/scala/ReduceByKey/target/ReduceByKey-3.0-shaded.jar
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] BUILD SUCCESS
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Total time: 3.321 s
>>> [INFO] Finished at: 2023-05-29T16:54:47+01:00
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [WARNING]
>>> [WARNING] Plugin validation issues were detected in 4 plugin(s)
>>> [WARNING]
>>> [WARNING] * org.scala-tools:maven-scala-plugin:2.15.2
>>> [WARNING] * org.apache.maven.plugins:maven-compiler-plugin:3.10.1
>>> [WARNING] * org.apache.maven.plugins:maven-shade-plugin:1.6
>>> [WARNING] * org.apache.maven.plugins:maven-resources-plugin:3.3.0
>>> [WARNING]
>>> [WARNING] For more or less details, use 'maven.plugin.validation'
>>> property with one of the values (case insensitive): [BRIEF, DEFAULT,
>>> VERBOSE]
>>> [WARNING]
>>> Completed compiling
>>> Mon May 29 16:54:47 BST 2023 , Running in **local mode**
>>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>>> SLF4J: Defaulting to no-operation (NOP) logger implementation
>>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
>>> further details.
>>>
>>> Application is ReduceByKey
>>>
>>> Exception in thread "main" java.lang.NoSuchMethodError:
>>> scala.package$.Seq()Lscala/collection/immutable/Seq$;
>>> at ReduceByKey$.main(ReduceByKey.scala:23)
>>> at ReduceByKey.main(ReduceByKey.scala)
>>> at
>>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>>> Method)
>>> at
>>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> at
>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>> at
>>> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
>>> at org.apache.spark.deploy.SparkSubmit.org
>>> $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
>>> at
>>> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
>>> at
>>> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
>>> at
>>> org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
>>> at
>>> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
>>> at
>>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>
>>>
>>>
>>> Part of pom.xml is here
>>>
>>>
>>> <project xmlns="https://maven.apache.org/POM/4.0.0" xmlns:xsi="
>>> https://www.w3.org/2001/XMLSchema-instance"
>>> xsi:schemaLocation="https://maven.apache.org/POM/4.0.0
>>> https://maven.apache.org/maven-v4_0_0.xsd">
>>> <modelVersion>4.0.0</modelVersion>
>>> <groupId>spark</groupId>
>>> <version>3.0</version>
>>> <artifactId>ReduceByKey</artifactId>
>>> <name>${project.artifactId}</name>
>>>
>>> <properties>
>>> <maven.compiler.source>11.0.1</maven.compiler.source>
>>> <maven.compiler.target>11.0.1</maven.compiler.target>
>>> <encoding>UTF-8</encoding>
>>> <scala.version>2.13.8</scala.version> <!-- Is this correct for spark
>>> 3.4? -->
>>> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
>>> </properties>
>>>
>>> <dependencies>
>>> <dependency>
>>> <groupId>org.scala-lang</groupId>
>>> <artifactId>scala-library</artifactId>
>>> <version>2.13.8</version> <!-- Is this correct for spark 3.4? -->
>>> </dependency>
>>> <dependency>
>>> <groupId>org.apache.spark</groupId>
>>> <artifactId>spark-core_2.13</artifactId> <!-- Is this correct for
>>> spark 3.4? -->
>>> <version>3.4.0</version>
>>> <scope>provided</scope>
>>> </dependency>
>>> <dependency>
>>> <groupId>org.apache.spark</groupId>
>>> <artifactId>spark-sql_2.13</artifactId>
>>> <version>3.4.0</version>
>>> <scope>provided</scope>
>>> </dependency>
>>>
>>>
>>> Thanks
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>>
>>> On Mon, 29 May 2023 at 13:44, Bjørn Jørgensen <bj...@gmail.com>
>>> wrote:
>>>
>>>> Change
>>>>
>>>> <dependencies>
>>>> <dependency>
>>>> <groupId>org.scala-lang</groupId>
>>>> <artifactId>scala-library</artifactId>
>>>> <version>2.13.11-M2</version>
>>>> </dependency>
>>>> <dependency>
>>>>
>>>> to
>>>>
>>>> <dependencies>
>>>> <dependency>
>>>> <groupId>org.scala-lang</groupId>
>>>> <artifactId>scala-library</artifactId>
>>>> <version>${scala.version}</version>
>>>> </dependency>
>>>>
>>>> man. 29. mai 2023 kl. 13:20 skrev Lingzhe Sun <li...@hirain.com>:
>>>>
>>>>> Hi Mich,
>>>>>
>>>>> Spark 3.4.0 prebuilt with scala 2.13 is built with version 2.13.8
>>>>> <https://github.com/apache/spark/blob/88f69d6f92860823b1a90bc162ebca2b7c8132fc/pom.xml#L170>.
>>>>> Since you are using spark-core_2.13 and spark-sql_2.13, you should stick to
>>>>> the major(13) and the minor version(8). Not using any of these may cause
>>>>> unexpected behaviour(though scala claims compatibility among minor version
>>>>> changes, I've encountered problem using the scala package with the same
>>>>> major version and different minor version. That may due to bug fixes and
>>>>> upgrade of scala itself.).
>>>>> And although I did not encountered such problem, this
>>>>> <https://stackoverflow.com/a/26411339/19476830>can be a a pitfall for
>>>>> you.
>>>>>
>>>>> ------------------------------
>>>>> Best Regards!
>>>>>
>>>>> ...........................................................................
>>>>> Lingzhe Sun
>>>>> Hirain Technology
>>>>>
>>>>>
>>>>> *From:* Mich Talebzadeh <mi...@gmail.com>
>>>>> *Date:* 2023-05-29 17:55
>>>>> *To:* Bjørn Jørgensen <bj...@gmail.com>
>>>>> *CC:* user @spark <us...@spark.apache.org>
>>>>> *Subject:* Re: maven with Spark 3.4.0 fails compilation
>>>>> Thanks for your helpful comments Bjorn.
>>>>>
>>>>> I managed to compile the code with maven but when it run it fails with
>>>>>
>>>>> Application is ReduceByKey
>>>>>
>>>>> Exception in thread "main" java.lang.NoSuchMethodError:
>>>>> scala.package$.Seq()Lscala/collection/immutable/Seq$;
>>>>> at ReduceByKey$.main(ReduceByKey.scala:23)
>>>>> at ReduceByKey.main(ReduceByKey.scala)
>>>>> at
>>>>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>> Method)
>>>>> at
>>>>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>> at
>>>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>>>> at
>>>>> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
>>>>> at org.apache.spark.deploy.SparkSubmit.org
>>>>> $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
>>>>> at
>>>>> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
>>>>> at
>>>>> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
>>>>> at
>>>>> org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
>>>>> at
>>>>> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
>>>>> at
>>>>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
>>>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>>>
>>>>> I attach the pom.xml and the sample scala code is self contained and
>>>>> basic. Again it runs with SBT with no issues.
>>>>>
>>>>> FYI, my scala version on host is
>>>>>
>>>>> scala -version
>>>>> Scala code runner version 2.13.6 -- Copyright 2002-2021, LAMP/EPFL and
>>>>> Lightbend, Inc.
>>>>>
>>>>> I think I have a scala incompatible somewhere again
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> Mich Talebzadeh,
>>>>> Lead Solutions Architect/Engineering Lead
>>>>> Palantir Technologies Limited
>>>>> London
>>>>> United Kingdom
>>>>>
>>>>>
>>>>> view my Linkedin profile
>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>
>>>>>
>>>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>>>
>>>>>
>>>>>
>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>>> any loss, damage or destruction of data or any other property which may
>>>>> arise from relying on this email's technical content is explicitly
>>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>>> arising from such loss, damage or destruction.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Sun, 28 May 2023 at 20:29, Bjørn Jørgensen <
>>>>> bjornjorgensen@gmail.com> wrote:
>>>>>
>>>>>> From chatgpt4
>>>>>>
>>>>>>
>>>>>> The problem appears to be that there is a mismatch between the
>>>>>> version of Scala used by the Scala Maven plugin and the version of the
>>>>>> Scala library defined as a dependency in your POM. You've defined your
>>>>>> Scala version in your properties as `2.12.17` but you're pulling in
>>>>>> `scala-library` version `2.13.6` as a dependency.
>>>>>>
>>>>>> The Scala Maven plugin will be using the Scala version defined in the
>>>>>> `scala.version` property for compilation, but then it tries to load classes
>>>>>> from a different Scala version, hence the error.
>>>>>>
>>>>>> To resolve this issue, make sure the `scala.version` property matches
>>>>>> the version of `scala-library` defined in your dependencies. In your case,
>>>>>> you may want to change `scala.version` to `2.13.6`.
>>>>>>
>>>>>> Here's the corrected part of your POM:
>>>>>>
>>>>>> ```xml
>>>>>> <properties>
>>>>>> <maven.compiler.source>1.7</maven.compiler.source>
>>>>>> <maven.compiler.target>1.7</maven.compiler.target>
>>>>>> <encoding>UTF-8</encoding>
>>>>>> <scala.version>2.13.6</scala.version> <!-- Here's the change -->
>>>>>> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
>>>>>> </properties>
>>>>>> ```
>>>>>>
>>>>>> Additionally, ensure that the Scala versions in the Spark
>>>>>> dependencies match the `scala.version` property as well. If you've updated
>>>>>> the Scala version to `2.13.6`, the artifactIds for Spark dependencies
>>>>>> should be `spark-core_2.13` and `spark-sql_2.13`.
>>>>>>
>>>>>> Another thing to consider: your Java version defined in
>>>>>> `maven.compiler.source` and `maven.compiler.target` is `1.7`, which is
>>>>>> quite outdated and might not be compatible with the latest versions of
>>>>>> these libraries. Consider updating to a more recent version of Java, such
>>>>>> as Java 8 or above, depending on the requirements of the libraries you're
>>>>>> using.
>>>>>>
>>>>>>
>>>>>>
>>>>>> The same problem persists in this updated POM file - there's a
>>>>>> mismatch in the Scala version declared in the properties and the version
>>>>>> used in your dependencies. Here's what you need to update:
>>>>>>
>>>>>> 1. Update the Scala version in your properties to match the Scala
>>>>>> library and your Spark dependencies:
>>>>>>
>>>>>> ```xml
>>>>>> <properties>
>>>>>> <maven.compiler.source>1.7</maven.compiler.source>
>>>>>> <maven.compiler.target>1.7</maven.compiler.target>
>>>>>> <encoding>UTF-8</encoding>
>>>>>> <scala.version>2.13.6</scala.version> <!-- updated Scala version
>>>>>> -->
>>>>>> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
>>>>>> </properties>
>>>>>> ```
>>>>>>
>>>>>> 2. Make sure all your Spark dependencies use the same Scala version.
>>>>>> In this case, I see `spark-streaming-kafka_2.11` which should be
>>>>>> `spark-streaming-kafka_2.13` if you're using Scala `2.13.6`.
>>>>>>
>>>>>> ```xml
>>>>>> <dependency>
>>>>>> <groupId>org.apache.spark</groupId>
>>>>>> <artifactId>spark-streaming-kafka_2.13</artifactId> <!-- updated
>>>>>> to 2.13 -->
>>>>>> <version>1.6.3</version> <!-- this might also need to be updated
>>>>>> as this is a very old version -->
>>>>>> <scope>provided</scope>
>>>>>> </dependency>
>>>>>> ```
>>>>>>
>>>>>> 3. As mentioned in the previous message, your Java version
>>>>>> (`maven.compiler.source` and `maven.compiler.target`) is also quite
>>>>>> outdated. Depending on the requirements of the libraries you're using, you
>>>>>> might want to update this to a newer version, such as Java 8 or above.
>>>>>>
>>>>>> Finally, ensure that the correct versions of these libraries exist in
>>>>>> your Maven repository or are available in the central Maven repository. If
>>>>>> the versions don't match, Maven will not be able to find and download the
>>>>>> correct dependencies, which can lead to problems.
>>>>>>
>>>>>> Please note that it's crucial to maintain consistency in your Scala
>>>>>> and Java versions across your project and its dependencies to avoid these
>>>>>> kinds of issues. Mixing versions can lead to binary incompatibility errors,
>>>>>> such as the one you're seeing.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> The differences in behavior between SBT and Maven when resolving
>>>>>> Scala dependencies can be attributed to how they each handle Scala binary
>>>>>> versions.
>>>>>>
>>>>>> SBT is specifically designed for Scala projects and has built-in
>>>>>> support for handling Scala's binary compatibility issues. When you define a
>>>>>> Scala library dependency in SBT, you can specify the Scala binary version
>>>>>> as "_2.12", "_2.13", etc. in the library artifact name. SBT will then
>>>>>> automatically replace this with the actual Scala binary version defined in
>>>>>> your SBT configuration. So, if you've defined different Scala versions for
>>>>>> your project and for a specific dependency, SBT will handle this gracefully.
>>>>>>
>>>>>> Maven, on the other hand, is more generic and does not have the same
>>>>>> level of built-in support for Scala's binary versions. In Maven, the Scala
>>>>>> version is typically hardcoded in the artifact name, and if this doesn't
>>>>>> match the actual Scala version used in your project, it can lead to binary
>>>>>> compatibility issues.
>>>>>>
>>>>>> This difference between SBT and Maven means that a project can work
>>>>>> fine when built with SBT but fail when built with Maven, due to these Scala
>>>>>> version discrepancies. To avoid these issues in Maven, you need to ensure
>>>>>> that the Scala version is consistent across your project configuration and
>>>>>> all your dependencies.
>>>>>>
>>>>>> Also, another reason for this can be because SBT often fetches
>>>>>> dependencies on the fly, while Maven requires them to be explicitly
>>>>>> declared. SBT can download and link the correct Scala library version even
>>>>>> if it's not explicitly declared, while Maven will only use the versions
>>>>>> that have been specified in the POM file.
>>>>>>
>>>>>> To summarize, SBT is more flexible and Scala-oriented than Maven,
>>>>>> which can lead to different behavior when handling Scala dependencies. When
>>>>>> using Maven for Scala projects, it's essential to ensure that all Scala
>>>>>> versions match across the project.
>>>>>>
>>>>>> søn. 28. mai 2023 kl. 21:03 skrev Mich Talebzadeh <
>>>>>> mich.talebzadeh@gmail.com>:
>>>>>>
>>>>>>> This compilation works fine with SBT but fails with maven!
>>>>>>>
>>>>>>> Spark version 3.4.0
>>>>>>> Apache Maven 3.9.2 (c9616018c7a021c1c39be70fb2843d6f5f9b8a1c)
>>>>>>> Java version: 11.0.1, vendor: Oracle Corporation, runtime:
>>>>>>> /opt/jdk-11.0.1
>>>>>>>
>>>>>>> This from the pom.xml file
>>>>>>>
>>>>>>> <project xmlns="https://maven.apache.org/POM/4.0.0" xmlns:xsi="
>>>>>>> https://www.w3.org/2001/XMLSchema-instance"
>>>>>>> xsi:schemaLocation="https://maven.apache.org/POM/4.0.0
>>>>>>> https://maven.apache.org/maven-v4_0_0.xsd">
>>>>>>> <modelVersion>4.0.0</modelVersion>
>>>>>>> <groupId>spark</groupId>
>>>>>>> <version>3.0</version>
>>>>>>> <name>${project.artifactId}</name>
>>>>>>>
>>>>>>> <properties>
>>>>>>> <maven.compiler.source>1.7</maven.compiler.source>
>>>>>>> <maven.compiler.target>1.7</maven.compiler.target>
>>>>>>> <encoding>UTF-8</encoding>
>>>>>>> <scala.version>2.12.17</scala.version>
>>>>>>> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
>>>>>>> </properties>
>>>>>>> <dependencies>
>>>>>>> <dependency>
>>>>>>> <groupId>org.scala-lang</groupId>
>>>>>>> <artifactId>scala-library</artifactId>
>>>>>>> <version>2.13.6</version>
>>>>>>> </dependency>
>>>>>>> <dependency>
>>>>>>> <groupId>org.apache.spark</groupId>
>>>>>>> <artifactId>spark-core_2.13</artifactId>
>>>>>>> <version>3.4.0</version>
>>>>>>> <scope>provided</scope>
>>>>>>> </dependency>
>>>>>>> <dependency>
>>>>>>> <groupId>org.apache.spark</groupId>
>>>>>>> <artifactId>spark-sql_2.13</artifactId>
>>>>>>> <version>3.4.0</version>
>>>>>>> <scope>provided</scope>
>>>>>>>
>>>>>>> The pom file is attached. These are the errors I am getting
>>>>>>>
>>>>>>> [ERROR] error: error while loading package, class file
>>>>>>> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/reflect/package.class)'
>>>>>>> is broken
>>>>>>> [ERROR] (class java.lang.RuntimeException/error reading Scala
>>>>>>> signature of package.class: Scala signature package has wrong version
>>>>>>> [ERROR] error: error while loading package, class file
>>>>>>> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/package.class)'
>>>>>>> is broken
>>>>>>> [ERROR] (class java.lang.RuntimeException/error reading Scala
>>>>>>> signature of package.class: Scala signature package has wrong version
>>>>>>> [ERROR] error: error while loading package, class file
>>>>>>> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/collection/package.class)'
>>>>>>> is broken
>>>>>>> [ERROR] (class java.lang.RuntimeException/error reading Scala
>>>>>>> signature of package.class: Scala signature package has wrong version
>>>>>>> [ERROR] three errors found
>>>>>>> [ERROR] Failed to execute goal
>>>>>>> org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project
>>>>>>> scala: wrap: org.apache.commons.exec.ExecuteException: Process exited with
>>>>>>> an error: 1(Exit value: 1) -> [Help 1]
>>>>>>>
>>>>>>> Any ideas will be appreciated.
>>>>>>>
>>>>>>> Mich Talebzadeh,
>>>>>>> Lead Solutions Architect/Engineering Lead
>>>>>>> Palantir Technologies Limited
>>>>>>> London
>>>>>>> United Kingdom
>>>>>>>
>>>>>>>
>>>>>>> view my Linkedin profile
>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>
>>>>>>>
>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility
>>>>>>> for any loss, damage or destruction of data or any other property which may
>>>>>>> arise from relying on this email's technical content is explicitly
>>>>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>>>>> arising from such loss, damage or destruction.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Bjørn Jørgensen
>>>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>>>> Norge
>>>>>>
>>>>>> +47 480 94 297
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Bjørn Jørgensen
>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>> Norge
>>>>
>>>> +47 480 94 297
>>>>
>>>
>>
>> --
>> Bjørn Jørgensen
>> Vestre Aspehaug 4, 6010 Ålesund
>> Norge
>>
>> +47 480 94 297
>>
>>
Re: Re: maven with Spark 3.4.0 fails compilation
Posted by Mich Talebzadeh <mi...@gmail.com>.
Many thanks Lingzhe.
I found the problem. The code was compiled against Scala 2.13 but it was
running under Scala 2.12. So I changed it to compile under Scala 2.12.17
and it worked fine.
Thanks again to both Bjorn and Lingzhe.
Regards
Mich Talebzadeh,
Lead Solutions Architect/Engineering Lead
Palantir Technologies Limited
London
United Kingdom
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Tue, 30 May 2023 at 03:57, Lingzhe Sun <li...@hirain.com> wrote:
> Hi Mich,
> Your part of pom looks good. Another hint would be the exception message
> itself:
> Exception in thread "main" java.lang.NoSuchMethodError:
> scala.package$.Seq()Lscala/collection/immutable/Seq$;
> It basically suggest that your class could not be found in your compiled
> code. It says that the Seq() from the scala.package whose return type is
> scala.collection.immutable.Seq. The reason could be that you have mutiple
> versions of same classes in your classpath, which typically dues to using
> imcompatible library versions, like here:
> [WARNING] Expected all dependencies to require Scala version: 2.13.8
> [WARNING] spark:ReduceByKey:3.0 requires scala version: 2.13.8
> [WARNING] org.scala-lang.modules:scala-parallel-collections_2.13:1.0.4
> requires scala version: 2.13.6
> [WARNING] Multiple versions of scala libraries detected!
> You choose to use one version of the scala library(2.13.8) while one
> dependency use another library(2.13.6). I'd suggest looking in the
> following order:
>
> 1. Remove the library scala-parallel-collections_2.13:1.0.4 and see if
> it works.
> 2. Check the line (ReduceByKey.scala:23) to check what's the package
> that the function belongs to. If it's using incompatible library, try
> remove it and see if the problem persist.
> 3. Build the artifact and check its output using IDEA build artifact
> and option--copy to the output dir, which also could be a fast way to see
> if you have included imcompatible library accidentally.
> 4. Set your maven-scala-plugin to a newer version
> <https://mvnrepository.com/artifact/net.alchim31.maven/scala-maven-plugin>
> to avoid potential issues.
> 5. Check the warnings below and set the maven.plugin.validation
> property to VERBOSE to see if there're potential problems: [WARNING]
> * org.scala-tools:maven-scala-plugin:2.15.2
> [WARNING] * org.apache.maven.plugins:maven-compiler-plugin:3.10.1
> [WARNING] * org.apache.maven.plugins:maven-shade-plugin:1.6
> [WARNING] * org.apache.maven.plugins:maven-resources-plugin:3.3.0
> [WARNING]
> [WARNING] For more or less details, use 'maven.plugin.validation'
> property with one of the values (case insensitive): [BRIEF, DEFAULT,
> VERBOSE]
> 6. Some code sharing would help, if it's possible.
>
> And TBH, I have no idea how could Seq() from a scala.package be used
> cause there's no such package. There could be some underlying mechanism
> that I do not know. Hope someone could kindly explain this.
> ------------------------------
> Best Regards!
> ...........................................................................
> Lingzhe Sun
> Hirain Technology
>
>
> *From:* Bjørn Jørgensen <bj...@gmail.com>
> *Date:* 2023-05-30 00:22
> *To:* Mich Talebzadeh <mi...@gmail.com>
> *CC:* Lingzhe Sun <li...@hirain.com>; User <us...@spark.apache.org>
> *Subject:* Re: Re: maven with Spark 3.4.0 fails compilation
> <scala.version>2.13.8</scala.version>
> <https://github.com/apache/spark/blob/87a5442f7ed96b11051d8a9333476d080054e5a0/pom.xml#LL3606C46-L3606C46>
>
> you must change 2.13.6 to 2.13.8
>
>
> man. 29. mai 2023 kl. 18:02 skrev Mich Talebzadeh <
> mich.talebzadeh@gmail.com>:
>
>> Thanks everyone. Still not much progress :(. It is becoming a bit
>> confusing as I am getting this error
>>
>> Compiling ReduceByKey
>> [INFO] Scanning for projects...
>> [INFO]
>> [INFO] -------------------------< spark:ReduceByKey
>> >--------------------------
>> [INFO] Building ReduceByKey 3.0
>> [INFO] from pom.xml
>> [INFO] --------------------------------[ jar
>> ]---------------------------------
>> [INFO]
>> [INFO] --- resources:3.3.0:resources (default-resources) @ ReduceByKey ---
>> [WARNING] Using platform encoding (ANSI_X3.4-1968 actually) to copy
>> filtered resources, i.e. build is platform dependent!
>> [INFO] skip non existing resourceDirectory
>> /data6/hduser/scala/ReduceByKey/src/main/resources
>> [INFO]
>> [INFO] --- compiler:3.10.1:compile (default-compile) @ ReduceByKey ---
>> [INFO] Nothing to compile - all classes are up to date
>> [INFO]
>> [INFO] --- scala:2.15.2:compile (default) @ ReduceByKey ---
>> [INFO] Checking for multiple versions of scala
>> [WARNING] Expected all dependencies to require Scala version: 2.13.8
>> [WARNING] spark:ReduceByKey:3.0 requires scala version: 2.13.8
>> [WARNING] org.scala-lang.modules:scala-parallel-collections_2.13:1.0.4
>> requires scala version: 2.13.6
>> [WARNING] Multiple versions of scala libraries detected!
>> [INFO] includes = [**/*.java,**/*.scala,]
>> [INFO] excludes = []
>> [INFO] Nothing to compile - all classes are up to date
>> [INFO]
>> [INFO] --- resources:3.3.0:testResources (default-testResources) @
>> ReduceByKey ---
>> [WARNING] Using platform encoding (ANSI_X3.4-1968 actually) to copy
>> filtered resources, i.e. build is platform dependent!
>> [INFO] skip non existing resourceDirectory
>> /data6/hduser/scala/ReduceByKey/src/test/resources
>> [INFO]
>> [INFO] --- compiler:3.10.1:testCompile (default-testCompile) @
>> ReduceByKey ---
>> [INFO] No sources to compile
>> [INFO]
>> [INFO] --- surefire:3.0.0:test (default-test) @ ReduceByKey ---
>> [INFO] No tests to run.
>> [INFO]
>> [INFO] --- jar:3.3.0:jar (default-jar) @ ReduceByKey ---
>> [INFO] Building jar:
>> /data6/hduser/scala/ReduceByKey/target/ReduceByKey.jar
>> [INFO]
>> [INFO] --- shade:1.6:shade (default) @ ReduceByKey ---
>> [INFO] Including org.scala-lang:scala-library:jar:2.13.8 in the shaded
>> jar.
>> [INFO] Replacing original artifact with shaded artifact.
>> [INFO] Replacing /data6/hduser/scala/ReduceByKey/target/ReduceByKey.jar
>> with /data6/hduser/scala/ReduceByKey/target/ReduceByKey-3.0-shaded.jar
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] BUILD SUCCESS
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Total time: 3.321 s
>> [INFO] Finished at: 2023-05-29T16:54:47+01:00
>> [INFO]
>> ------------------------------------------------------------------------
>> [WARNING]
>> [WARNING] Plugin validation issues were detected in 4 plugin(s)
>> [WARNING]
>> [WARNING] * org.scala-tools:maven-scala-plugin:2.15.2
>> [WARNING] * org.apache.maven.plugins:maven-compiler-plugin:3.10.1
>> [WARNING] * org.apache.maven.plugins:maven-shade-plugin:1.6
>> [WARNING] * org.apache.maven.plugins:maven-resources-plugin:3.3.0
>> [WARNING]
>> [WARNING] For more or less details, use 'maven.plugin.validation'
>> property with one of the values (case insensitive): [BRIEF, DEFAULT,
>> VERBOSE]
>> [WARNING]
>> Completed compiling
>> Mon May 29 16:54:47 BST 2023 , Running in **local mode**
>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>> SLF4J: Defaulting to no-operation (NOP) logger implementation
>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
>> further details.
>>
>> Application is ReduceByKey
>>
>> Exception in thread "main" java.lang.NoSuchMethodError:
>> scala.package$.Seq()Lscala/collection/immutable/Seq$;
>> at ReduceByKey$.main(ReduceByKey.scala:23)
>> at ReduceByKey.main(ReduceByKey.scala)
>> at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>> at
>> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
>> at org.apache.spark.deploy.SparkSubmit.org
>> $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
>> at
>> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
>> at
>> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
>> at
>> org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
>> at
>> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
>> at
>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
>>
>>
>> Part of pom.xml is here
>>
>>
>> <project xmlns="https://maven.apache.org/POM/4.0.0" xmlns:xsi="
>> https://www.w3.org/2001/XMLSchema-instance"
>> xsi:schemaLocation="https://maven.apache.org/POM/4.0.0
>> https://maven.apache.org/maven-v4_0_0.xsd">
>> <modelVersion>4.0.0</modelVersion>
>> <groupId>spark</groupId>
>> <version>3.0</version>
>> <artifactId>ReduceByKey</artifactId>
>> <name>${project.artifactId}</name>
>>
>> <properties>
>> <maven.compiler.source>11.0.1</maven.compiler.source>
>> <maven.compiler.target>11.0.1</maven.compiler.target>
>> <encoding>UTF-8</encoding>
>> <scala.version>2.13.8</scala.version> <!-- Is this correct for spark 3.4?
>> -->
>> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
>> </properties>
>>
>> <dependencies>
>> <dependency>
>> <groupId>org.scala-lang</groupId>
>> <artifactId>scala-library</artifactId>
>> <version>2.13.8</version> <!-- Is this correct for spark 3.4? -->
>> </dependency>
>> <dependency>
>> <groupId>org.apache.spark</groupId>
>> <artifactId>spark-core_2.13</artifactId> <!-- Is this correct for
>> spark 3.4? -->
>> <version>3.4.0</version>
>> <scope>provided</scope>
>> </dependency>
>> <dependency>
>> <groupId>org.apache.spark</groupId>
>> <artifactId>spark-sql_2.13</artifactId>
>> <version>3.4.0</version>
>> <scope>provided</scope>
>> </dependency>
>>
>>
>> Thanks
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Mon, 29 May 2023 at 13:44, Bjørn Jørgensen <bj...@gmail.com>
>> wrote:
>>
>>> Change
>>>
>>> <dependencies>
>>> <dependency>
>>> <groupId>org.scala-lang</groupId>
>>> <artifactId>scala-library</artifactId>
>>> <version>2.13.11-M2</version>
>>> </dependency>
>>> <dependency>
>>>
>>> to
>>>
>>> <dependencies>
>>> <dependency>
>>> <groupId>org.scala-lang</groupId>
>>> <artifactId>scala-library</artifactId>
>>> <version>${scala.version}</version>
>>> </dependency>
>>>
>>> man. 29. mai 2023 kl. 13:20 skrev Lingzhe Sun <li...@hirain.com>:
>>>
>>>> Hi Mich,
>>>>
>>>> Spark 3.4.0 prebuilt with scala 2.13 is built with version 2.13.8
>>>> <https://github.com/apache/spark/blob/88f69d6f92860823b1a90bc162ebca2b7c8132fc/pom.xml#L170>.
>>>> Since you are using spark-core_2.13 and spark-sql_2.13, you should stick to
>>>> the major(13) and the minor version(8). Not using any of these may cause
>>>> unexpected behaviour(though scala claims compatibility among minor version
>>>> changes, I've encountered problem using the scala package with the same
>>>> major version and different minor version. That may due to bug fixes and
>>>> upgrade of scala itself.).
>>>> And although I did not encountered such problem, this
>>>> <https://stackoverflow.com/a/26411339/19476830>can be a a pitfall for
>>>> you.
>>>>
>>>> ------------------------------
>>>> Best Regards!
>>>>
>>>> ...........................................................................
>>>> Lingzhe Sun
>>>> Hirain Technology
>>>>
>>>>
>>>> *From:* Mich Talebzadeh <mi...@gmail.com>
>>>> *Date:* 2023-05-29 17:55
>>>> *To:* Bjørn Jørgensen <bj...@gmail.com>
>>>> *CC:* user @spark <us...@spark.apache.org>
>>>> *Subject:* Re: maven with Spark 3.4.0 fails compilation
>>>> Thanks for your helpful comments Bjorn.
>>>>
>>>> I managed to compile the code with maven but when it run it fails with
>>>>
>>>> Application is ReduceByKey
>>>>
>>>> Exception in thread "main" java.lang.NoSuchMethodError:
>>>> scala.package$.Seq()Lscala/collection/immutable/Seq$;
>>>> at ReduceByKey$.main(ReduceByKey.scala:23)
>>>> at ReduceByKey.main(ReduceByKey.scala)
>>>> at
>>>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> at
>>>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>> at
>>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>>> at
>>>> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
>>>> at org.apache.spark.deploy.SparkSubmit.org
>>>> $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
>>>> at
>>>> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
>>>> at
>>>> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
>>>> at
>>>> org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
>>>> at
>>>> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
>>>> at
>>>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
>>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>>
>>>> I attach the pom.xml and the sample scala code is self contained and
>>>> basic. Again it runs with SBT with no issues.
>>>>
>>>> FYI, my scala version on host is
>>>>
>>>> scala -version
>>>> Scala code runner version 2.13.6 -- Copyright 2002-2021, LAMP/EPFL and
>>>> Lightbend, Inc.
>>>>
>>>> I think I have a scala incompatible somewhere again
>>>>
>>>> Cheers
>>>>
>>>>
>>>> Mich Talebzadeh,
>>>> Lead Solutions Architect/Engineering Lead
>>>> Palantir Technologies Limited
>>>> London
>>>> United Kingdom
>>>>
>>>>
>>>> view my Linkedin profile
>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>
>>>>
>>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>>
>>>>
>>>>
>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>> any loss, damage or destruction of data or any other property which may
>>>> arise from relying on this email's technical content is explicitly
>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>> arising from such loss, damage or destruction.
>>>>
>>>>
>>>>
>>>>
>>>> On Sun, 28 May 2023 at 20:29, Bjørn Jørgensen <bj...@gmail.com>
>>>> wrote:
>>>>
>>>>> From chatgpt4
>>>>>
>>>>>
>>>>> The problem appears to be that there is a mismatch between the version
>>>>> of Scala used by the Scala Maven plugin and the version of the Scala
>>>>> library defined as a dependency in your POM. You've defined your Scala
>>>>> version in your properties as `2.12.17` but you're pulling in
>>>>> `scala-library` version `2.13.6` as a dependency.
>>>>>
>>>>> The Scala Maven plugin will be using the Scala version defined in the
>>>>> `scala.version` property for compilation, but then it tries to load classes
>>>>> from a different Scala version, hence the error.
>>>>>
>>>>> To resolve this issue, make sure the `scala.version` property matches
>>>>> the version of `scala-library` defined in your dependencies. In your case,
>>>>> you may want to change `scala.version` to `2.13.6`.
>>>>>
>>>>> Here's the corrected part of your POM:
>>>>>
>>>>> ```xml
>>>>> <properties>
>>>>> <maven.compiler.source>1.7</maven.compiler.source>
>>>>> <maven.compiler.target>1.7</maven.compiler.target>
>>>>> <encoding>UTF-8</encoding>
>>>>> <scala.version>2.13.6</scala.version> <!-- Here's the change -->
>>>>> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
>>>>> </properties>
>>>>> ```
>>>>>
>>>>> Additionally, ensure that the Scala versions in the Spark dependencies
>>>>> match the `scala.version` property as well. If you've updated the Scala
>>>>> version to `2.13.6`, the artifactIds for Spark dependencies should be
>>>>> `spark-core_2.13` and `spark-sql_2.13`.
>>>>>
>>>>> Another thing to consider: your Java version defined in
>>>>> `maven.compiler.source` and `maven.compiler.target` is `1.7`, which is
>>>>> quite outdated and might not be compatible with the latest versions of
>>>>> these libraries. Consider updating to a more recent version of Java, such
>>>>> as Java 8 or above, depending on the requirements of the libraries you're
>>>>> using.
>>>>>
>>>>>
>>>>>
>>>>> The same problem persists in this updated POM file - there's a
>>>>> mismatch in the Scala version declared in the properties and the version
>>>>> used in your dependencies. Here's what you need to update:
>>>>>
>>>>> 1. Update the Scala version in your properties to match the Scala
>>>>> library and your Spark dependencies:
>>>>>
>>>>> ```xml
>>>>> <properties>
>>>>> <maven.compiler.source>1.7</maven.compiler.source>
>>>>> <maven.compiler.target>1.7</maven.compiler.target>
>>>>> <encoding>UTF-8</encoding>
>>>>> <scala.version>2.13.6</scala.version> <!-- updated Scala version
>>>>> -->
>>>>> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
>>>>> </properties>
>>>>> ```
>>>>>
>>>>> 2. Make sure all your Spark dependencies use the same Scala version.
>>>>> In this case, I see `spark-streaming-kafka_2.11` which should be
>>>>> `spark-streaming-kafka_2.13` if you're using Scala `2.13.6`.
>>>>>
>>>>> ```xml
>>>>> <dependency>
>>>>> <groupId>org.apache.spark</groupId>
>>>>> <artifactId>spark-streaming-kafka_2.13</artifactId> <!-- updated
>>>>> to 2.13 -->
>>>>> <version>1.6.3</version> <!-- this might also need to be updated
>>>>> as this is a very old version -->
>>>>> <scope>provided</scope>
>>>>> </dependency>
>>>>> ```
>>>>>
>>>>> 3. As mentioned in the previous message, your Java version
>>>>> (`maven.compiler.source` and `maven.compiler.target`) is also quite
>>>>> outdated. Depending on the requirements of the libraries you're using, you
>>>>> might want to update this to a newer version, such as Java 8 or above.
>>>>>
>>>>> Finally, ensure that the correct versions of these libraries exist in
>>>>> your Maven repository or are available in the central Maven repository. If
>>>>> the versions don't match, Maven will not be able to find and download the
>>>>> correct dependencies, which can lead to problems.
>>>>>
>>>>> Please note that it's crucial to maintain consistency in your Scala
>>>>> and Java versions across your project and its dependencies to avoid these
>>>>> kinds of issues. Mixing versions can lead to binary incompatibility errors,
>>>>> such as the one you're seeing.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> The differences in behavior between SBT and Maven when resolving Scala
>>>>> dependencies can be attributed to how they each handle Scala binary
>>>>> versions.
>>>>>
>>>>> SBT is specifically designed for Scala projects and has built-in
>>>>> support for handling Scala's binary compatibility issues. When you define a
>>>>> Scala library dependency in SBT, you can specify the Scala binary version
>>>>> as "_2.12", "_2.13", etc. in the library artifact name. SBT will then
>>>>> automatically replace this with the actual Scala binary version defined in
>>>>> your SBT configuration. So, if you've defined different Scala versions for
>>>>> your project and for a specific dependency, SBT will handle this gracefully.
>>>>>
>>>>> Maven, on the other hand, is more generic and does not have the same
>>>>> level of built-in support for Scala's binary versions. In Maven, the Scala
>>>>> version is typically hardcoded in the artifact name, and if this doesn't
>>>>> match the actual Scala version used in your project, it can lead to binary
>>>>> compatibility issues.
>>>>>
>>>>> This difference between SBT and Maven means that a project can work
>>>>> fine when built with SBT but fail when built with Maven, due to these Scala
>>>>> version discrepancies. To avoid these issues in Maven, you need to ensure
>>>>> that the Scala version is consistent across your project configuration and
>>>>> all your dependencies.
>>>>>
>>>>> Also, another reason for this can be because SBT often fetches
>>>>> dependencies on the fly, while Maven requires them to be explicitly
>>>>> declared. SBT can download and link the correct Scala library version even
>>>>> if it's not explicitly declared, while Maven will only use the versions
>>>>> that have been specified in the POM file.
>>>>>
>>>>> To summarize, SBT is more flexible and Scala-oriented than Maven,
>>>>> which can lead to different behavior when handling Scala dependencies. When
>>>>> using Maven for Scala projects, it's essential to ensure that all Scala
>>>>> versions match across the project.
>>>>>
>>>>> søn. 28. mai 2023 kl. 21:03 skrev Mich Talebzadeh <
>>>>> mich.talebzadeh@gmail.com>:
>>>>>
>>>>>> This compilation works fine with SBT but fails with maven!
>>>>>>
>>>>>> Spark version 3.4.0
>>>>>> Apache Maven 3.9.2 (c9616018c7a021c1c39be70fb2843d6f5f9b8a1c)
>>>>>> Java version: 11.0.1, vendor: Oracle Corporation, runtime:
>>>>>> /opt/jdk-11.0.1
>>>>>>
>>>>>> This from the pom.xml file
>>>>>>
>>>>>> <project xmlns="https://maven.apache.org/POM/4.0.0" xmlns:xsi="
>>>>>> https://www.w3.org/2001/XMLSchema-instance"
>>>>>> xsi:schemaLocation="https://maven.apache.org/POM/4.0.0
>>>>>> https://maven.apache.org/maven-v4_0_0.xsd">
>>>>>> <modelVersion>4.0.0</modelVersion>
>>>>>> <groupId>spark</groupId>
>>>>>> <version>3.0</version>
>>>>>> <name>${project.artifactId}</name>
>>>>>>
>>>>>> <properties>
>>>>>> <maven.compiler.source>1.7</maven.compiler.source>
>>>>>> <maven.compiler.target>1.7</maven.compiler.target>
>>>>>> <encoding>UTF-8</encoding>
>>>>>> <scala.version>2.12.17</scala.version>
>>>>>> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
>>>>>> </properties>
>>>>>> <dependencies>
>>>>>> <dependency>
>>>>>> <groupId>org.scala-lang</groupId>
>>>>>> <artifactId>scala-library</artifactId>
>>>>>> <version>2.13.6</version>
>>>>>> </dependency>
>>>>>> <dependency>
>>>>>> <groupId>org.apache.spark</groupId>
>>>>>> <artifactId>spark-core_2.13</artifactId>
>>>>>> <version>3.4.0</version>
>>>>>> <scope>provided</scope>
>>>>>> </dependency>
>>>>>> <dependency>
>>>>>> <groupId>org.apache.spark</groupId>
>>>>>> <artifactId>spark-sql_2.13</artifactId>
>>>>>> <version>3.4.0</version>
>>>>>> <scope>provided</scope>
>>>>>>
>>>>>> The pom file is attached. These are the errors I am getting
>>>>>>
>>>>>> [ERROR] error: error while loading package, class file
>>>>>> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/reflect/package.class)'
>>>>>> is broken
>>>>>> [ERROR] (class java.lang.RuntimeException/error reading Scala
>>>>>> signature of package.class: Scala signature package has wrong version
>>>>>> [ERROR] error: error while loading package, class file
>>>>>> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/package.class)'
>>>>>> is broken
>>>>>> [ERROR] (class java.lang.RuntimeException/error reading Scala
>>>>>> signature of package.class: Scala signature package has wrong version
>>>>>> [ERROR] error: error while loading package, class file
>>>>>> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/collection/package.class)'
>>>>>> is broken
>>>>>> [ERROR] (class java.lang.RuntimeException/error reading Scala
>>>>>> signature of package.class: Scala signature package has wrong version
>>>>>> [ERROR] three errors found
>>>>>> [ERROR] Failed to execute goal
>>>>>> org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project
>>>>>> scala: wrap: org.apache.commons.exec.ExecuteException: Process exited with
>>>>>> an error: 1(Exit value: 1) -> [Help 1]
>>>>>>
>>>>>> Any ideas will be appreciated.
>>>>>>
>>>>>> Mich Talebzadeh,
>>>>>> Lead Solutions Architect/Engineering Lead
>>>>>> Palantir Technologies Limited
>>>>>> London
>>>>>> United Kingdom
>>>>>>
>>>>>>
>>>>>> view my Linkedin profile
>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>
>>>>>>
>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility
>>>>>> for any loss, damage or destruction of data or any other property which may
>>>>>> arise from relying on this email's technical content is explicitly
>>>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>>>> arising from such loss, damage or destruction.
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Bjørn Jørgensen
>>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>>> Norge
>>>>>
>>>>> +47 480 94 297
>>>>>
>>>>
>>>
>>> --
>>> Bjørn Jørgensen
>>> Vestre Aspehaug 4, 6010 Ålesund
>>> Norge
>>>
>>> +47 480 94 297
>>>
>>
>
> --
> Bjørn Jørgensen
> Vestre Aspehaug 4, 6010 Ålesund
> Norge
>
> +47 480 94 297
>
>
Re: Re: maven with Spark 3.4.0 fails compilation
Posted by Lingzhe Sun <li...@hirain.com>.
Hi Mich,
Your part of pom looks good. Another hint would be the exception message itself:
Exception in thread "main" java.lang.NoSuchMethodError: scala.package$.Seq()Lscala/collection/immutable/Seq$;
It basically suggest that your class could not be found in your compiled code. It says that the Seq() from the scala.package whose return type is scala.collection.immutable.Seq. The reason could be that you have mutiple versions of same classes in your classpath, which typically dues to using imcompatible library versions, like here:
[WARNING] Expected all dependencies to require Scala version: 2.13.8
[WARNING] spark:ReduceByKey:3.0 requires scala version: 2.13.8
[WARNING] org.scala-lang.modules:scala-parallel-collections_2.13:1.0.4 requires scala version: 2.13.6
[WARNING] Multiple versions of scala libraries detected!
You choose to use one version of the scala library(2.13.8) while one dependency use another library(2.13.6). I'd suggest looking in the following order:
Remove the library scala-parallel-collections_2.13:1.0.4 and see if it works.
Check the line (ReduceByKey.scala:23) to check what's the package that the function belongs to. If it's using incompatible library, try remove it and see if the problem persist.
Build the artifact and check its output using IDEA build artifact and option--copy to the output dir, which also could be a fast way to see if you have included imcompatible library accidentally.
Set your maven-scala-plugin to a newer version to avoid potential issues.
Check the warnings below and set the maven.plugin.validation property to VERBOSE to see if there're potential problems: [WARNING] * org.scala-tools:maven-scala-plugin:2.15.2
[WARNING] * org.apache.maven.plugins:maven-compiler-plugin:3.10.1
[WARNING] * org.apache.maven.plugins:maven-shade-plugin:1.6
[WARNING] * org.apache.maven.plugins:maven-resources-plugin:3.3.0
[WARNING]
[WARNING] For more or less details, use 'maven.plugin.validation' property with one of the values (case insensitive): [BRIEF, DEFAULT, VERBOSE]
Some code sharing would help, if it's possible.
And TBH, I have no idea how could Seq() from a scala.package be used cause there's no such package. There could be some underlying mechanism that I do not know. Hope someone could kindly explain this.
Best Regards!
...........................................................................
Lingzhe Sun
Hirain Technology
From: Bjørn Jørgensen
Date: 2023-05-30 00:22
To: Mich Talebzadeh
CC: Lingzhe Sun; User
Subject: Re: Re: maven with Spark 3.4.0 fails compilation
<scala.version>2.13.8</scala.version>
you must change 2.13.6 to 2.13.8
man. 29. mai 2023 kl. 18:02 skrev Mich Talebzadeh <mi...@gmail.com>:
Thanks everyone. Still not much progress :(. It is becoming a bit confusing as I am getting this error
Compiling ReduceByKey
[INFO] Scanning for projects...
[INFO]
[INFO] -------------------------< spark:ReduceByKey >--------------------------
[INFO] Building ReduceByKey 3.0
[INFO] from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- resources:3.3.0:resources (default-resources) @ ReduceByKey ---
[WARNING] Using platform encoding (ANSI_X3.4-1968 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /data6/hduser/scala/ReduceByKey/src/main/resources
[INFO]
[INFO] --- compiler:3.10.1:compile (default-compile) @ ReduceByKey ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- scala:2.15.2:compile (default) @ ReduceByKey ---
[INFO] Checking for multiple versions of scala
[WARNING] Expected all dependencies to require Scala version: 2.13.8
[WARNING] spark:ReduceByKey:3.0 requires scala version: 2.13.8
[WARNING] org.scala-lang.modules:scala-parallel-collections_2.13:1.0.4 requires scala version: 2.13.6
[WARNING] Multiple versions of scala libraries detected!
[INFO] includes = [**/*.java,**/*.scala,]
[INFO] excludes = []
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- resources:3.3.0:testResources (default-testResources) @ ReduceByKey ---
[WARNING] Using platform encoding (ANSI_X3.4-1968 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /data6/hduser/scala/ReduceByKey/src/test/resources
[INFO]
[INFO] --- compiler:3.10.1:testCompile (default-testCompile) @ ReduceByKey ---
[INFO] No sources to compile
[INFO]
[INFO] --- surefire:3.0.0:test (default-test) @ ReduceByKey ---
[INFO] No tests to run.
[INFO]
[INFO] --- jar:3.3.0:jar (default-jar) @ ReduceByKey ---
[INFO] Building jar: /data6/hduser/scala/ReduceByKey/target/ReduceByKey.jar
[INFO]
[INFO] --- shade:1.6:shade (default) @ ReduceByKey ---
[INFO] Including org.scala-lang:scala-library:jar:2.13.8 in the shaded jar.
[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing /data6/hduser/scala/ReduceByKey/target/ReduceByKey.jar with /data6/hduser/scala/ReduceByKey/target/ReduceByKey-3.0-shaded.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.321 s
[INFO] Finished at: 2023-05-29T16:54:47+01:00
[INFO] ------------------------------------------------------------------------
[WARNING]
[WARNING] Plugin validation issues were detected in 4 plugin(s)
[WARNING]
[WARNING] * org.scala-tools:maven-scala-plugin:2.15.2
[WARNING] * org.apache.maven.plugins:maven-compiler-plugin:3.10.1
[WARNING] * org.apache.maven.plugins:maven-shade-plugin:1.6
[WARNING] * org.apache.maven.plugins:maven-resources-plugin:3.3.0
[WARNING]
[WARNING] For more or less details, use 'maven.plugin.validation' property with one of the values (case insensitive): [BRIEF, DEFAULT, VERBOSE]
[WARNING]
Completed compiling
Mon May 29 16:54:47 BST 2023 , Running in **local mode**
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Application is ReduceByKey
Exception in thread "main" java.lang.NoSuchMethodError: scala.package$.Seq()Lscala/collection/immutable/Seq$;
at ReduceByKey$.main(ReduceByKey.scala:23)
at ReduceByKey.main(ReduceByKey.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Part of pom.xml is here
<project xmlns="https://maven.apache.org/POM/4.0.0" xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="https://maven.apache.org/POM/4.0.0 https://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>spark</groupId>
<version>3.0</version>
<artifactId>ReduceByKey</artifactId>
<name>${project.artifactId}</name>
<properties>
<maven.compiler.source>11.0.1</maven.compiler.source>
<maven.compiler.target>11.0.1</maven.compiler.target>
<encoding>UTF-8</encoding>
<scala.version>2.13.8</scala.version> <!-- Is this correct for spark 3.4? -->
<maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
</properties>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.13.8</version> <!-- Is this correct for spark 3.4? -->
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.13</artifactId> <!-- Is this correct for spark 3.4? -->
<version>3.4.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.13</artifactId>
<version>3.4.0</version>
<scope>provided</scope>
</dependency>
Thanks
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
On Mon, 29 May 2023 at 13:44, Bjørn Jørgensen <bj...@gmail.com> wrote:
Change
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.13.11-M2</version>
</dependency>
<dependency>
to
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
man. 29. mai 2023 kl. 13:20 skrev Lingzhe Sun <li...@hirain.com>:
Hi Mich,
Spark 3.4.0 prebuilt with scala 2.13 is built with version 2.13.8. Since you are using spark-core_2.13 and spark-sql_2.13, you should stick to the major(13) and the minor version(8). Not using any of these may cause unexpected behaviour(though scala claims compatibility among minor version changes, I've encountered problem using the scala package with the same major version and different minor version. That may due to bug fixes and upgrade of scala itself.).
And although I did not encountered such problem, this can be a a pitfall for you.
Best Regards!
...........................................................................
Lingzhe Sun
Hirain Technology
From: Mich Talebzadeh
Date: 2023-05-29 17:55
To: Bjørn Jørgensen
CC: user @spark
Subject: Re: maven with Spark 3.4.0 fails compilation
Thanks for your helpful comments Bjorn.
I managed to compile the code with maven but when it run it fails with
Application is ReduceByKey
Exception in thread "main" java.lang.NoSuchMethodError: scala.package$.Seq()Lscala/collection/immutable/Seq$;
at ReduceByKey$.main(ReduceByKey.scala:23)
at ReduceByKey.main(ReduceByKey.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
I attach the pom.xml and the sample scala code is self contained and basic. Again it runs with SBT with no issues.
FYI, my scala version on host is
scala -version
Scala code runner version 2.13.6 -- Copyright 2002-2021, LAMP/EPFL and Lightbend, Inc.
I think I have a scala incompatible somewhere again
Cheers
Mich Talebzadeh,
Lead Solutions Architect/Engineering Lead
Palantir Technologies Limited
London
United Kingdom
view my Linkedin profile
https://en.everybodywiki.com/Mich_Talebzadeh
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
On Sun, 28 May 2023 at 20:29, Bjørn Jørgensen <bj...@gmail.com> wrote:
From chatgpt4
The problem appears to be that there is a mismatch between the version of Scala used by the Scala Maven plugin and the version of the Scala library defined as a dependency in your POM. You've defined your Scala version in your properties as `2.12.17` but you're pulling in `scala-library` version `2.13.6` as a dependency.
The Scala Maven plugin will be using the Scala version defined in the `scala.version` property for compilation, but then it tries to load classes from a different Scala version, hence the error.
To resolve this issue, make sure the `scala.version` property matches the version of `scala-library` defined in your dependencies. In your case, you may want to change `scala.version` to `2.13.6`.
Here's the corrected part of your POM:
```xml
<properties>
<maven.compiler.source>1.7</maven.compiler.source>
<maven.compiler.target>1.7</maven.compiler.target>
<encoding>UTF-8</encoding>
<scala.version>2.13.6</scala.version> <!-- Here's the change -->
<maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
</properties>
```
Additionally, ensure that the Scala versions in the Spark dependencies match the `scala.version` property as well. If you've updated the Scala version to `2.13.6`, the artifactIds for Spark dependencies should be `spark-core_2.13` and `spark-sql_2.13`.
Another thing to consider: your Java version defined in `maven.compiler.source` and `maven.compiler.target` is `1.7`, which is quite outdated and might not be compatible with the latest versions of these libraries. Consider updating to a more recent version of Java, such as Java 8 or above, depending on the requirements of the libraries you're using.
The same problem persists in this updated POM file - there's a mismatch in the Scala version declared in the properties and the version used in your dependencies. Here's what you need to update:
1. Update the Scala version in your properties to match the Scala library and your Spark dependencies:
```xml
<properties>
<maven.compiler.source>1.7</maven.compiler.source>
<maven.compiler.target>1.7</maven.compiler.target>
<encoding>UTF-8</encoding>
<scala.version>2.13.6</scala.version> <!-- updated Scala version -->
<maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
</properties>
```
2. Make sure all your Spark dependencies use the same Scala version. In this case, I see `spark-streaming-kafka_2.11` which should be `spark-streaming-kafka_2.13` if you're using Scala `2.13.6`.
```xml
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka_2.13</artifactId> <!-- updated to 2.13 -->
<version>1.6.3</version> <!-- this might also need to be updated as this is a very old version -->
<scope>provided</scope>
</dependency>
```
3. As mentioned in the previous message, your Java version (`maven.compiler.source` and `maven.compiler.target`) is also quite outdated. Depending on the requirements of the libraries you're using, you might want to update this to a newer version, such as Java 8 or above.
Finally, ensure that the correct versions of these libraries exist in your Maven repository or are available in the central Maven repository. If the versions don't match, Maven will not be able to find and download the correct dependencies, which can lead to problems.
Please note that it's crucial to maintain consistency in your Scala and Java versions across your project and its dependencies to avoid these kinds of issues. Mixing versions can lead to binary incompatibility errors, such as the one you're seeing.
The differences in behavior between SBT and Maven when resolving Scala dependencies can be attributed to how they each handle Scala binary versions.
SBT is specifically designed for Scala projects and has built-in support for handling Scala's binary compatibility issues. When you define a Scala library dependency in SBT, you can specify the Scala binary version as "_2.12", "_2.13", etc. in the library artifact name. SBT will then automatically replace this with the actual Scala binary version defined in your SBT configuration. So, if you've defined different Scala versions for your project and for a specific dependency, SBT will handle this gracefully.
Maven, on the other hand, is more generic and does not have the same level of built-in support for Scala's binary versions. In Maven, the Scala version is typically hardcoded in the artifact name, and if this doesn't match the actual Scala version used in your project, it can lead to binary compatibility issues.
This difference between SBT and Maven means that a project can work fine when built with SBT but fail when built with Maven, due to these Scala version discrepancies. To avoid these issues in Maven, you need to ensure that the Scala version is consistent across your project configuration and all your dependencies.
Also, another reason for this can be because SBT often fetches dependencies on the fly, while Maven requires them to be explicitly declared. SBT can download and link the correct Scala library version even if it's not explicitly declared, while Maven will only use the versions that have been specified in the POM file.
To summarize, SBT is more flexible and Scala-oriented than Maven, which can lead to different behavior when handling Scala dependencies. When using Maven for Scala projects, it's essential to ensure that all Scala versions match across the project.
søn. 28. mai 2023 kl. 21:03 skrev Mich Talebzadeh <mi...@gmail.com>:
This compilation works fine with SBT but fails with maven!
Spark version 3.4.0
Apache Maven 3.9.2 (c9616018c7a021c1c39be70fb2843d6f5f9b8a1c)
Java version: 11.0.1, vendor: Oracle Corporation, runtime: /opt/jdk-11.0.1
This from the pom.xml file
<project xmlns="https://maven.apache.org/POM/4.0.0" xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="https://maven.apache.org/POM/4.0.0 https://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>spark</groupId>
<version>3.0</version>
<name>${project.artifactId}</name>
<properties>
<maven.compiler.source>1.7</maven.compiler.source>
<maven.compiler.target>1.7</maven.compiler.target>
<encoding>UTF-8</encoding>
<scala.version>2.12.17</scala.version>
<maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
</properties>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.13.6</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.13</artifactId>
<version>3.4.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.13</artifactId>
<version>3.4.0</version>
<scope>provided</scope>
The pom file is attached. These are the errors I am getting
[ERROR] error: error while loading package, class file '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/reflect/package.class)' is broken
[ERROR] (class java.lang.RuntimeException/error reading Scala signature of package.class: Scala signature package has wrong version
[ERROR] error: error while loading package, class file '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/package.class)' is broken
[ERROR] (class java.lang.RuntimeException/error reading Scala signature of package.class: Scala signature package has wrong version
[ERROR] error: error while loading package, class file '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/collection/package.class)' is broken
[ERROR] (class java.lang.RuntimeException/error reading Scala signature of package.class: Scala signature package has wrong version
[ERROR] three errors found
[ERROR] Failed to execute goal org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project scala: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1) -> [Help 1]
Any ideas will be appreciated.
Mich Talebzadeh,
Lead Solutions Architect/Engineering Lead
Palantir Technologies Limited
London
United Kingdom
view my Linkedin profile
https://en.everybodywiki.com/Mich_Talebzadeh
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
--
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge
+47 480 94 297
--
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge
+47 480 94 297
--
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge
+47 480 94 297
Re: Re: maven with Spark 3.4.0 fails compilation
Posted by Bjørn Jørgensen <bj...@gmail.com>.
<scala.version>2.13.8</scala.version>
<https://github.com/apache/spark/blob/87a5442f7ed96b11051d8a9333476d080054e5a0/pom.xml#LL3606C46-L3606C46>
you must change 2.13.6 to 2.13.8
man. 29. mai 2023 kl. 18:02 skrev Mich Talebzadeh <mich.talebzadeh@gmail.com
>:
> Thanks everyone. Still not much progress :(. It is becoming a bit
> confusing as I am getting this error
>
> Compiling ReduceByKey
> [INFO] Scanning for projects...
> [INFO]
> [INFO] -------------------------< spark:ReduceByKey
> >--------------------------
> [INFO] Building ReduceByKey 3.0
> [INFO] from pom.xml
> [INFO] --------------------------------[ jar
> ]---------------------------------
> [INFO]
> [INFO] --- resources:3.3.0:resources (default-resources) @ ReduceByKey ---
> [WARNING] Using platform encoding (ANSI_X3.4-1968 actually) to copy
> filtered resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory
> /data6/hduser/scala/ReduceByKey/src/main/resources
> [INFO]
> [INFO] --- compiler:3.10.1:compile (default-compile) @ ReduceByKey ---
> [INFO] Nothing to compile - all classes are up to date
> [INFO]
> [INFO] --- scala:2.15.2:compile (default) @ ReduceByKey ---
> [INFO] Checking for multiple versions of scala
> [WARNING] Expected all dependencies to require Scala version: 2.13.8
> [WARNING] spark:ReduceByKey:3.0 requires scala version: 2.13.8
> [WARNING] org.scala-lang.modules:scala-parallel-collections_2.13:1.0.4
> requires scala version: 2.13.6
> [WARNING] Multiple versions of scala libraries detected!
> [INFO] includes = [**/*.java,**/*.scala,]
> [INFO] excludes = []
> [INFO] Nothing to compile - all classes are up to date
> [INFO]
> [INFO] --- resources:3.3.0:testResources (default-testResources) @
> ReduceByKey ---
> [WARNING] Using platform encoding (ANSI_X3.4-1968 actually) to copy
> filtered resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory
> /data6/hduser/scala/ReduceByKey/src/test/resources
> [INFO]
> [INFO] --- compiler:3.10.1:testCompile (default-testCompile) @ ReduceByKey
> ---
> [INFO] No sources to compile
> [INFO]
> [INFO] --- surefire:3.0.0:test (default-test) @ ReduceByKey ---
> [INFO] No tests to run.
> [INFO]
> [INFO] --- jar:3.3.0:jar (default-jar) @ ReduceByKey ---
> [INFO] Building jar: /data6/hduser/scala/ReduceByKey/target/ReduceByKey.jar
> [INFO]
> [INFO] --- shade:1.6:shade (default) @ ReduceByKey ---
> [INFO] Including org.scala-lang:scala-library:jar:2.13.8 in the shaded jar.
> [INFO] Replacing original artifact with shaded artifact.
> [INFO] Replacing /data6/hduser/scala/ReduceByKey/target/ReduceByKey.jar
> with /data6/hduser/scala/ReduceByKey/target/ReduceByKey-3.0-shaded.jar
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 3.321 s
> [INFO] Finished at: 2023-05-29T16:54:47+01:00
> [INFO]
> ------------------------------------------------------------------------
> [WARNING]
> [WARNING] Plugin validation issues were detected in 4 plugin(s)
> [WARNING]
> [WARNING] * org.scala-tools:maven-scala-plugin:2.15.2
> [WARNING] * org.apache.maven.plugins:maven-compiler-plugin:3.10.1
> [WARNING] * org.apache.maven.plugins:maven-shade-plugin:1.6
> [WARNING] * org.apache.maven.plugins:maven-resources-plugin:3.3.0
> [WARNING]
> [WARNING] For more or less details, use 'maven.plugin.validation' property
> with one of the values (case insensitive): [BRIEF, DEFAULT, VERBOSE]
> [WARNING]
> Completed compiling
> Mon May 29 16:54:47 BST 2023 , Running in **local mode**
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
>
> Application is ReduceByKey
>
> Exception in thread "main" java.lang.NoSuchMethodError:
> scala.package$.Seq()Lscala/collection/immutable/Seq$;
> at ReduceByKey$.main(ReduceByKey.scala:23)
> at ReduceByKey.main(ReduceByKey.scala)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> at org.apache.spark.deploy.SparkSubmit.org
> $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
> at
> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
> at
> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
> at
> org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
> at
> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
>
>
> Part of pom.xml is here
>
>
> <project xmlns="https://maven.apache.org/POM/4.0.0" xmlns:xsi="
> https://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="https://maven.apache.org/POM/4.0.0
> https://maven.apache.org/maven-v4_0_0.xsd">
> <modelVersion>4.0.0</modelVersion>
> <groupId>spark</groupId>
> <version>3.0</version>
> <artifactId>ReduceByKey</artifactId>
> <name>${project.artifactId}</name>
>
> <properties>
> <maven.compiler.source>11.0.1</maven.compiler.source>
> <maven.compiler.target>11.0.1</maven.compiler.target>
> <encoding>UTF-8</encoding>
> <scala.version>2.13.8</scala.version> <!-- Is this correct for spark 3.4?
> -->
> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
> </properties>
>
> <dependencies>
> <dependency>
> <groupId>org.scala-lang</groupId>
> <artifactId>scala-library</artifactId>
> <version>2.13.8</version> <!-- Is this correct for spark 3.4? -->
> </dependency>
> <dependency>
> <groupId>org.apache.spark</groupId>
> <artifactId>spark-core_2.13</artifactId> <!-- Is this correct for
> spark 3.4? -->
> <version>3.4.0</version>
> <scope>provided</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.spark</groupId>
> <artifactId>spark-sql_2.13</artifactId>
> <version>3.4.0</version>
> <scope>provided</scope>
> </dependency>
>
>
> Thanks
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Mon, 29 May 2023 at 13:44, Bjørn Jørgensen <bj...@gmail.com>
> wrote:
>
>> Change
>>
>> <dependencies>
>> <dependency>
>> <groupId>org.scala-lang</groupId>
>> <artifactId>scala-library</artifactId>
>> <version>2.13.11-M2</version>
>> </dependency>
>> <dependency>
>>
>> to
>>
>> <dependencies>
>> <dependency>
>> <groupId>org.scala-lang</groupId>
>> <artifactId>scala-library</artifactId>
>> <version>${scala.version}</version>
>> </dependency>
>>
>> man. 29. mai 2023 kl. 13:20 skrev Lingzhe Sun <li...@hirain.com>:
>>
>>> Hi Mich,
>>>
>>> Spark 3.4.0 prebuilt with scala 2.13 is built with version 2.13.8
>>> <https://github.com/apache/spark/blob/88f69d6f92860823b1a90bc162ebca2b7c8132fc/pom.xml#L170>.
>>> Since you are using spark-core_2.13 and spark-sql_2.13, you should stick to
>>> the major(13) and the minor version(8). Not using any of these may cause
>>> unexpected behaviour(though scala claims compatibility among minor version
>>> changes, I've encountered problem using the scala package with the same
>>> major version and different minor version. That may due to bug fixes and
>>> upgrade of scala itself.).
>>> And although I did not encountered such problem, this
>>> <https://stackoverflow.com/a/26411339/19476830>can be a a pitfall for
>>> you.
>>>
>>> ------------------------------
>>> Best Regards!
>>>
>>> ...........................................................................
>>> Lingzhe Sun
>>> Hirain Technology
>>>
>>>
>>> *From:* Mich Talebzadeh <mi...@gmail.com>
>>> *Date:* 2023-05-29 17:55
>>> *To:* Bjørn Jørgensen <bj...@gmail.com>
>>> *CC:* user @spark <us...@spark.apache.org>
>>> *Subject:* Re: maven with Spark 3.4.0 fails compilation
>>> Thanks for your helpful comments Bjorn.
>>>
>>> I managed to compile the code with maven but when it run it fails with
>>>
>>> Application is ReduceByKey
>>>
>>> Exception in thread "main" java.lang.NoSuchMethodError:
>>> scala.package$.Seq()Lscala/collection/immutable/Seq$;
>>> at ReduceByKey$.main(ReduceByKey.scala:23)
>>> at ReduceByKey.main(ReduceByKey.scala)
>>> at
>>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>>> Method)
>>> at
>>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> at
>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>> at
>>> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
>>> at org.apache.spark.deploy.SparkSubmit.org
>>> $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
>>> at
>>> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
>>> at
>>> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
>>> at
>>> org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
>>> at
>>> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
>>> at
>>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>
>>> I attach the pom.xml and the sample scala code is self contained and
>>> basic. Again it runs with SBT with no issues.
>>>
>>> FYI, my scala version on host is
>>>
>>> scala -version
>>> Scala code runner version 2.13.6 -- Copyright 2002-2021, LAMP/EPFL and
>>> Lightbend, Inc.
>>>
>>> I think I have a scala incompatible somewhere again
>>>
>>> Cheers
>>>
>>>
>>> Mich Talebzadeh,
>>> Lead Solutions Architect/Engineering Lead
>>> Palantir Technologies Limited
>>> London
>>> United Kingdom
>>>
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>>
>>> On Sun, 28 May 2023 at 20:29, Bjørn Jørgensen <bj...@gmail.com>
>>> wrote:
>>>
>>>> From chatgpt4
>>>>
>>>>
>>>> The problem appears to be that there is a mismatch between the version
>>>> of Scala used by the Scala Maven plugin and the version of the Scala
>>>> library defined as a dependency in your POM. You've defined your Scala
>>>> version in your properties as `2.12.17` but you're pulling in
>>>> `scala-library` version `2.13.6` as a dependency.
>>>>
>>>> The Scala Maven plugin will be using the Scala version defined in the
>>>> `scala.version` property for compilation, but then it tries to load classes
>>>> from a different Scala version, hence the error.
>>>>
>>>> To resolve this issue, make sure the `scala.version` property matches
>>>> the version of `scala-library` defined in your dependencies. In your case,
>>>> you may want to change `scala.version` to `2.13.6`.
>>>>
>>>> Here's the corrected part of your POM:
>>>>
>>>> ```xml
>>>> <properties>
>>>> <maven.compiler.source>1.7</maven.compiler.source>
>>>> <maven.compiler.target>1.7</maven.compiler.target>
>>>> <encoding>UTF-8</encoding>
>>>> <scala.version>2.13.6</scala.version> <!-- Here's the change -->
>>>> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
>>>> </properties>
>>>> ```
>>>>
>>>> Additionally, ensure that the Scala versions in the Spark dependencies
>>>> match the `scala.version` property as well. If you've updated the Scala
>>>> version to `2.13.6`, the artifactIds for Spark dependencies should be
>>>> `spark-core_2.13` and `spark-sql_2.13`.
>>>>
>>>> Another thing to consider: your Java version defined in
>>>> `maven.compiler.source` and `maven.compiler.target` is `1.7`, which is
>>>> quite outdated and might not be compatible with the latest versions of
>>>> these libraries. Consider updating to a more recent version of Java, such
>>>> as Java 8 or above, depending on the requirements of the libraries you're
>>>> using.
>>>>
>>>>
>>>>
>>>> The same problem persists in this updated POM file - there's a mismatch
>>>> in the Scala version declared in the properties and the version used in
>>>> your dependencies. Here's what you need to update:
>>>>
>>>> 1. Update the Scala version in your properties to match the Scala
>>>> library and your Spark dependencies:
>>>>
>>>> ```xml
>>>> <properties>
>>>> <maven.compiler.source>1.7</maven.compiler.source>
>>>> <maven.compiler.target>1.7</maven.compiler.target>
>>>> <encoding>UTF-8</encoding>
>>>> <scala.version>2.13.6</scala.version> <!-- updated Scala version -->
>>>> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
>>>> </properties>
>>>> ```
>>>>
>>>> 2. Make sure all your Spark dependencies use the same Scala version. In
>>>> this case, I see `spark-streaming-kafka_2.11` which should be
>>>> `spark-streaming-kafka_2.13` if you're using Scala `2.13.6`.
>>>>
>>>> ```xml
>>>> <dependency>
>>>> <groupId>org.apache.spark</groupId>
>>>> <artifactId>spark-streaming-kafka_2.13</artifactId> <!-- updated to
>>>> 2.13 -->
>>>> <version>1.6.3</version> <!-- this might also need to be updated as
>>>> this is a very old version -->
>>>> <scope>provided</scope>
>>>> </dependency>
>>>> ```
>>>>
>>>> 3. As mentioned in the previous message, your Java version
>>>> (`maven.compiler.source` and `maven.compiler.target`) is also quite
>>>> outdated. Depending on the requirements of the libraries you're using, you
>>>> might want to update this to a newer version, such as Java 8 or above.
>>>>
>>>> Finally, ensure that the correct versions of these libraries exist in
>>>> your Maven repository or are available in the central Maven repository. If
>>>> the versions don't match, Maven will not be able to find and download the
>>>> correct dependencies, which can lead to problems.
>>>>
>>>> Please note that it's crucial to maintain consistency in your Scala and
>>>> Java versions across your project and its dependencies to avoid these kinds
>>>> of issues. Mixing versions can lead to binary incompatibility errors, such
>>>> as the one you're seeing.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> The differences in behavior between SBT and Maven when resolving Scala
>>>> dependencies can be attributed to how they each handle Scala binary
>>>> versions.
>>>>
>>>> SBT is specifically designed for Scala projects and has built-in
>>>> support for handling Scala's binary compatibility issues. When you define a
>>>> Scala library dependency in SBT, you can specify the Scala binary version
>>>> as "_2.12", "_2.13", etc. in the library artifact name. SBT will then
>>>> automatically replace this with the actual Scala binary version defined in
>>>> your SBT configuration. So, if you've defined different Scala versions for
>>>> your project and for a specific dependency, SBT will handle this gracefully.
>>>>
>>>> Maven, on the other hand, is more generic and does not have the same
>>>> level of built-in support for Scala's binary versions. In Maven, the Scala
>>>> version is typically hardcoded in the artifact name, and if this doesn't
>>>> match the actual Scala version used in your project, it can lead to binary
>>>> compatibility issues.
>>>>
>>>> This difference between SBT and Maven means that a project can work
>>>> fine when built with SBT but fail when built with Maven, due to these Scala
>>>> version discrepancies. To avoid these issues in Maven, you need to ensure
>>>> that the Scala version is consistent across your project configuration and
>>>> all your dependencies.
>>>>
>>>> Also, another reason for this can be because SBT often fetches
>>>> dependencies on the fly, while Maven requires them to be explicitly
>>>> declared. SBT can download and link the correct Scala library version even
>>>> if it's not explicitly declared, while Maven will only use the versions
>>>> that have been specified in the POM file.
>>>>
>>>> To summarize, SBT is more flexible and Scala-oriented than Maven, which
>>>> can lead to different behavior when handling Scala dependencies. When using
>>>> Maven for Scala projects, it's essential to ensure that all Scala versions
>>>> match across the project.
>>>>
>>>> søn. 28. mai 2023 kl. 21:03 skrev Mich Talebzadeh <
>>>> mich.talebzadeh@gmail.com>:
>>>>
>>>>> This compilation works fine with SBT but fails with maven!
>>>>>
>>>>> Spark version 3.4.0
>>>>> Apache Maven 3.9.2 (c9616018c7a021c1c39be70fb2843d6f5f9b8a1c)
>>>>> Java version: 11.0.1, vendor: Oracle Corporation, runtime:
>>>>> /opt/jdk-11.0.1
>>>>>
>>>>> This from the pom.xml file
>>>>>
>>>>> <project xmlns="https://maven.apache.org/POM/4.0.0" xmlns:xsi="
>>>>> https://www.w3.org/2001/XMLSchema-instance"
>>>>> xsi:schemaLocation="https://maven.apache.org/POM/4.0.0
>>>>> https://maven.apache.org/maven-v4_0_0.xsd">
>>>>> <modelVersion>4.0.0</modelVersion>
>>>>> <groupId>spark</groupId>
>>>>> <version>3.0</version>
>>>>> <name>${project.artifactId}</name>
>>>>>
>>>>> <properties>
>>>>> <maven.compiler.source>1.7</maven.compiler.source>
>>>>> <maven.compiler.target>1.7</maven.compiler.target>
>>>>> <encoding>UTF-8</encoding>
>>>>> <scala.version>2.12.17</scala.version>
>>>>> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
>>>>> </properties>
>>>>> <dependencies>
>>>>> <dependency>
>>>>> <groupId>org.scala-lang</groupId>
>>>>> <artifactId>scala-library</artifactId>
>>>>> <version>2.13.6</version>
>>>>> </dependency>
>>>>> <dependency>
>>>>> <groupId>org.apache.spark</groupId>
>>>>> <artifactId>spark-core_2.13</artifactId>
>>>>> <version>3.4.0</version>
>>>>> <scope>provided</scope>
>>>>> </dependency>
>>>>> <dependency>
>>>>> <groupId>org.apache.spark</groupId>
>>>>> <artifactId>spark-sql_2.13</artifactId>
>>>>> <version>3.4.0</version>
>>>>> <scope>provided</scope>
>>>>>
>>>>> The pom file is attached. These are the errors I am getting
>>>>>
>>>>> [ERROR] error: error while loading package, class file
>>>>> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/reflect/package.class)'
>>>>> is broken
>>>>> [ERROR] (class java.lang.RuntimeException/error reading Scala
>>>>> signature of package.class: Scala signature package has wrong version
>>>>> [ERROR] error: error while loading package, class file
>>>>> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/package.class)'
>>>>> is broken
>>>>> [ERROR] (class java.lang.RuntimeException/error reading Scala
>>>>> signature of package.class: Scala signature package has wrong version
>>>>> [ERROR] error: error while loading package, class file
>>>>> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/collection/package.class)'
>>>>> is broken
>>>>> [ERROR] (class java.lang.RuntimeException/error reading Scala
>>>>> signature of package.class: Scala signature package has wrong version
>>>>> [ERROR] three errors found
>>>>> [ERROR] Failed to execute goal
>>>>> org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project
>>>>> scala: wrap: org.apache.commons.exec.ExecuteException: Process exited with
>>>>> an error: 1(Exit value: 1) -> [Help 1]
>>>>>
>>>>> Any ideas will be appreciated.
>>>>>
>>>>> Mich Talebzadeh,
>>>>> Lead Solutions Architect/Engineering Lead
>>>>> Palantir Technologies Limited
>>>>> London
>>>>> United Kingdom
>>>>>
>>>>>
>>>>> view my Linkedin profile
>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>
>>>>>
>>>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>>>
>>>>>
>>>>>
>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>>> any loss, damage or destruction of data or any other property which may
>>>>> arise from relying on this email's technical content is explicitly
>>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>>> arising from such loss, damage or destruction.
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>>>
>>>>
>>>>
>>>> --
>>>> Bjørn Jørgensen
>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>> Norge
>>>>
>>>> +47 480 94 297
>>>>
>>>
>>
>> --
>> Bjørn Jørgensen
>> Vestre Aspehaug 4, 6010 Ålesund
>> Norge
>>
>> +47 480 94 297
>>
>
--
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge
+47 480 94 297
Re: Re: maven with Spark 3.4.0 fails compilation
Posted by Mich Talebzadeh <mi...@gmail.com>.
Thanks everyone. Still not much progress :(. It is becoming a bit confusing
as I am getting this error
Compiling ReduceByKey
[INFO] Scanning for projects...
[INFO]
[INFO] -------------------------< spark:ReduceByKey
>--------------------------
[INFO] Building ReduceByKey 3.0
[INFO] from pom.xml
[INFO] --------------------------------[ jar
]---------------------------------
[INFO]
[INFO] --- resources:3.3.0:resources (default-resources) @ ReduceByKey ---
[WARNING] Using platform encoding (ANSI_X3.4-1968 actually) to copy
filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory
/data6/hduser/scala/ReduceByKey/src/main/resources
[INFO]
[INFO] --- compiler:3.10.1:compile (default-compile) @ ReduceByKey ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- scala:2.15.2:compile (default) @ ReduceByKey ---
[INFO] Checking for multiple versions of scala
[WARNING] Expected all dependencies to require Scala version: 2.13.8
[WARNING] spark:ReduceByKey:3.0 requires scala version: 2.13.8
[WARNING] org.scala-lang.modules:scala-parallel-collections_2.13:1.0.4
requires scala version: 2.13.6
[WARNING] Multiple versions of scala libraries detected!
[INFO] includes = [**/*.java,**/*.scala,]
[INFO] excludes = []
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- resources:3.3.0:testResources (default-testResources) @
ReduceByKey ---
[WARNING] Using platform encoding (ANSI_X3.4-1968 actually) to copy
filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory
/data6/hduser/scala/ReduceByKey/src/test/resources
[INFO]
[INFO] --- compiler:3.10.1:testCompile (default-testCompile) @ ReduceByKey
---
[INFO] No sources to compile
[INFO]
[INFO] --- surefire:3.0.0:test (default-test) @ ReduceByKey ---
[INFO] No tests to run.
[INFO]
[INFO] --- jar:3.3.0:jar (default-jar) @ ReduceByKey ---
[INFO] Building jar: /data6/hduser/scala/ReduceByKey/target/ReduceByKey.jar
[INFO]
[INFO] --- shade:1.6:shade (default) @ ReduceByKey ---
[INFO] Including org.scala-lang:scala-library:jar:2.13.8 in the shaded jar.
[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing /data6/hduser/scala/ReduceByKey/target/ReduceByKey.jar
with /data6/hduser/scala/ReduceByKey/target/ReduceByKey-3.0-shaded.jar
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 3.321 s
[INFO] Finished at: 2023-05-29T16:54:47+01:00
[INFO]
------------------------------------------------------------------------
[WARNING]
[WARNING] Plugin validation issues were detected in 4 plugin(s)
[WARNING]
[WARNING] * org.scala-tools:maven-scala-plugin:2.15.2
[WARNING] * org.apache.maven.plugins:maven-compiler-plugin:3.10.1
[WARNING] * org.apache.maven.plugins:maven-shade-plugin:1.6
[WARNING] * org.apache.maven.plugins:maven-resources-plugin:3.3.0
[WARNING]
[WARNING] For more or less details, use 'maven.plugin.validation' property
with one of the values (case insensitive): [BRIEF, DEFAULT, VERBOSE]
[WARNING]
Completed compiling
Mon May 29 16:54:47 BST 2023 , Running in **local mode**
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
Application is ReduceByKey
Exception in thread "main" java.lang.NoSuchMethodError:
scala.package$.Seq()Lscala/collection/immutable/Seq$;
at ReduceByKey$.main(ReduceByKey.scala:23)
at ReduceByKey.main(ReduceByKey.scala)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org
$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
at
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Part of pom.xml is here
<project xmlns="https://maven.apache.org/POM/4.0.0" xmlns:xsi="
https://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="https://maven.apache.org/POM/4.0.0
https://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>spark</groupId>
<version>3.0</version>
<artifactId>ReduceByKey</artifactId>
<name>${project.artifactId}</name>
<properties>
<maven.compiler.source>11.0.1</maven.compiler.source>
<maven.compiler.target>11.0.1</maven.compiler.target>
<encoding>UTF-8</encoding>
<scala.version>2.13.8</scala.version> <!-- Is this correct for spark 3.4?
-->
<maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
</properties>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.13.8</version> <!-- Is this correct for spark 3.4? -->
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.13</artifactId> <!-- Is this correct for spark
3.4? -->
<version>3.4.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.13</artifactId>
<version>3.4.0</version>
<scope>provided</scope>
</dependency>
Thanks
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Mon, 29 May 2023 at 13:44, Bjørn Jørgensen <bj...@gmail.com>
wrote:
> Change
>
> <dependencies>
> <dependency>
> <groupId>org.scala-lang</groupId>
> <artifactId>scala-library</artifactId>
> <version>2.13.11-M2</version>
> </dependency>
> <dependency>
>
> to
>
> <dependencies>
> <dependency>
> <groupId>org.scala-lang</groupId>
> <artifactId>scala-library</artifactId>
> <version>${scala.version}</version>
> </dependency>
>
> man. 29. mai 2023 kl. 13:20 skrev Lingzhe Sun <li...@hirain.com>:
>
>> Hi Mich,
>>
>> Spark 3.4.0 prebuilt with scala 2.13 is built with version 2.13.8
>> <https://github.com/apache/spark/blob/88f69d6f92860823b1a90bc162ebca2b7c8132fc/pom.xml#L170>.
>> Since you are using spark-core_2.13 and spark-sql_2.13, you should stick to
>> the major(13) and the minor version(8). Not using any of these may cause
>> unexpected behaviour(though scala claims compatibility among minor version
>> changes, I've encountered problem using the scala package with the same
>> major version and different minor version. That may due to bug fixes and
>> upgrade of scala itself.).
>> And although I did not encountered such problem, this
>> <https://stackoverflow.com/a/26411339/19476830>can be a a pitfall for
>> you.
>>
>> ------------------------------
>> Best Regards!
>>
>> ...........................................................................
>> Lingzhe Sun
>> Hirain Technology
>>
>>
>> *From:* Mich Talebzadeh <mi...@gmail.com>
>> *Date:* 2023-05-29 17:55
>> *To:* Bjørn Jørgensen <bj...@gmail.com>
>> *CC:* user @spark <us...@spark.apache.org>
>> *Subject:* Re: maven with Spark 3.4.0 fails compilation
>> Thanks for your helpful comments Bjorn.
>>
>> I managed to compile the code with maven but when it run it fails with
>>
>> Application is ReduceByKey
>>
>> Exception in thread "main" java.lang.NoSuchMethodError:
>> scala.package$.Seq()Lscala/collection/immutable/Seq$;
>> at ReduceByKey$.main(ReduceByKey.scala:23)
>> at ReduceByKey.main(ReduceByKey.scala)
>> at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>> at
>> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
>> at org.apache.spark.deploy.SparkSubmit.org
>> $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
>> at
>> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
>> at
>> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
>> at
>> org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
>> at
>> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
>> at
>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
>> I attach the pom.xml and the sample scala code is self contained and
>> basic. Again it runs with SBT with no issues.
>>
>> FYI, my scala version on host is
>>
>> scala -version
>> Scala code runner version 2.13.6 -- Copyright 2002-2021, LAMP/EPFL and
>> Lightbend, Inc.
>>
>> I think I have a scala incompatible somewhere again
>>
>> Cheers
>>
>>
>> Mich Talebzadeh,
>> Lead Solutions Architect/Engineering Lead
>> Palantir Technologies Limited
>> London
>> United Kingdom
>>
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>> https://en.everybodywiki.com/Mich_Talebzadeh
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Sun, 28 May 2023 at 20:29, Bjørn Jørgensen <bj...@gmail.com>
>> wrote:
>>
>>> From chatgpt4
>>>
>>>
>>> The problem appears to be that there is a mismatch between the version
>>> of Scala used by the Scala Maven plugin and the version of the Scala
>>> library defined as a dependency in your POM. You've defined your Scala
>>> version in your properties as `2.12.17` but you're pulling in
>>> `scala-library` version `2.13.6` as a dependency.
>>>
>>> The Scala Maven plugin will be using the Scala version defined in the
>>> `scala.version` property for compilation, but then it tries to load classes
>>> from a different Scala version, hence the error.
>>>
>>> To resolve this issue, make sure the `scala.version` property matches
>>> the version of `scala-library` defined in your dependencies. In your case,
>>> you may want to change `scala.version` to `2.13.6`.
>>>
>>> Here's the corrected part of your POM:
>>>
>>> ```xml
>>> <properties>
>>> <maven.compiler.source>1.7</maven.compiler.source>
>>> <maven.compiler.target>1.7</maven.compiler.target>
>>> <encoding>UTF-8</encoding>
>>> <scala.version>2.13.6</scala.version> <!-- Here's the change -->
>>> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
>>> </properties>
>>> ```
>>>
>>> Additionally, ensure that the Scala versions in the Spark dependencies
>>> match the `scala.version` property as well. If you've updated the Scala
>>> version to `2.13.6`, the artifactIds for Spark dependencies should be
>>> `spark-core_2.13` and `spark-sql_2.13`.
>>>
>>> Another thing to consider: your Java version defined in
>>> `maven.compiler.source` and `maven.compiler.target` is `1.7`, which is
>>> quite outdated and might not be compatible with the latest versions of
>>> these libraries. Consider updating to a more recent version of Java, such
>>> as Java 8 or above, depending on the requirements of the libraries you're
>>> using.
>>>
>>>
>>>
>>> The same problem persists in this updated POM file - there's a mismatch
>>> in the Scala version declared in the properties and the version used in
>>> your dependencies. Here's what you need to update:
>>>
>>> 1. Update the Scala version in your properties to match the Scala
>>> library and your Spark dependencies:
>>>
>>> ```xml
>>> <properties>
>>> <maven.compiler.source>1.7</maven.compiler.source>
>>> <maven.compiler.target>1.7</maven.compiler.target>
>>> <encoding>UTF-8</encoding>
>>> <scala.version>2.13.6</scala.version> <!-- updated Scala version -->
>>> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
>>> </properties>
>>> ```
>>>
>>> 2. Make sure all your Spark dependencies use the same Scala version. In
>>> this case, I see `spark-streaming-kafka_2.11` which should be
>>> `spark-streaming-kafka_2.13` if you're using Scala `2.13.6`.
>>>
>>> ```xml
>>> <dependency>
>>> <groupId>org.apache.spark</groupId>
>>> <artifactId>spark-streaming-kafka_2.13</artifactId> <!-- updated to
>>> 2.13 -->
>>> <version>1.6.3</version> <!-- this might also need to be updated as
>>> this is a very old version -->
>>> <scope>provided</scope>
>>> </dependency>
>>> ```
>>>
>>> 3. As mentioned in the previous message, your Java version
>>> (`maven.compiler.source` and `maven.compiler.target`) is also quite
>>> outdated. Depending on the requirements of the libraries you're using, you
>>> might want to update this to a newer version, such as Java 8 or above.
>>>
>>> Finally, ensure that the correct versions of these libraries exist in
>>> your Maven repository or are available in the central Maven repository. If
>>> the versions don't match, Maven will not be able to find and download the
>>> correct dependencies, which can lead to problems.
>>>
>>> Please note that it's crucial to maintain consistency in your Scala and
>>> Java versions across your project and its dependencies to avoid these kinds
>>> of issues. Mixing versions can lead to binary incompatibility errors, such
>>> as the one you're seeing.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> The differences in behavior between SBT and Maven when resolving Scala
>>> dependencies can be attributed to how they each handle Scala binary
>>> versions.
>>>
>>> SBT is specifically designed for Scala projects and has built-in support
>>> for handling Scala's binary compatibility issues. When you define a Scala
>>> library dependency in SBT, you can specify the Scala binary version as
>>> "_2.12", "_2.13", etc. in the library artifact name. SBT will then
>>> automatically replace this with the actual Scala binary version defined in
>>> your SBT configuration. So, if you've defined different Scala versions for
>>> your project and for a specific dependency, SBT will handle this gracefully.
>>>
>>> Maven, on the other hand, is more generic and does not have the same
>>> level of built-in support for Scala's binary versions. In Maven, the Scala
>>> version is typically hardcoded in the artifact name, and if this doesn't
>>> match the actual Scala version used in your project, it can lead to binary
>>> compatibility issues.
>>>
>>> This difference between SBT and Maven means that a project can work fine
>>> when built with SBT but fail when built with Maven, due to these Scala
>>> version discrepancies. To avoid these issues in Maven, you need to ensure
>>> that the Scala version is consistent across your project configuration and
>>> all your dependencies.
>>>
>>> Also, another reason for this can be because SBT often fetches
>>> dependencies on the fly, while Maven requires them to be explicitly
>>> declared. SBT can download and link the correct Scala library version even
>>> if it's not explicitly declared, while Maven will only use the versions
>>> that have been specified in the POM file.
>>>
>>> To summarize, SBT is more flexible and Scala-oriented than Maven, which
>>> can lead to different behavior when handling Scala dependencies. When using
>>> Maven for Scala projects, it's essential to ensure that all Scala versions
>>> match across the project.
>>>
>>> søn. 28. mai 2023 kl. 21:03 skrev Mich Talebzadeh <
>>> mich.talebzadeh@gmail.com>:
>>>
>>>> This compilation works fine with SBT but fails with maven!
>>>>
>>>> Spark version 3.4.0
>>>> Apache Maven 3.9.2 (c9616018c7a021c1c39be70fb2843d6f5f9b8a1c)
>>>> Java version: 11.0.1, vendor: Oracle Corporation, runtime:
>>>> /opt/jdk-11.0.1
>>>>
>>>> This from the pom.xml file
>>>>
>>>> <project xmlns="https://maven.apache.org/POM/4.0.0" xmlns:xsi="
>>>> https://www.w3.org/2001/XMLSchema-instance"
>>>> xsi:schemaLocation="https://maven.apache.org/POM/4.0.0
>>>> https://maven.apache.org/maven-v4_0_0.xsd">
>>>> <modelVersion>4.0.0</modelVersion>
>>>> <groupId>spark</groupId>
>>>> <version>3.0</version>
>>>> <name>${project.artifactId}</name>
>>>>
>>>> <properties>
>>>> <maven.compiler.source>1.7</maven.compiler.source>
>>>> <maven.compiler.target>1.7</maven.compiler.target>
>>>> <encoding>UTF-8</encoding>
>>>> <scala.version>2.12.17</scala.version>
>>>> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
>>>> </properties>
>>>> <dependencies>
>>>> <dependency>
>>>> <groupId>org.scala-lang</groupId>
>>>> <artifactId>scala-library</artifactId>
>>>> <version>2.13.6</version>
>>>> </dependency>
>>>> <dependency>
>>>> <groupId>org.apache.spark</groupId>
>>>> <artifactId>spark-core_2.13</artifactId>
>>>> <version>3.4.0</version>
>>>> <scope>provided</scope>
>>>> </dependency>
>>>> <dependency>
>>>> <groupId>org.apache.spark</groupId>
>>>> <artifactId>spark-sql_2.13</artifactId>
>>>> <version>3.4.0</version>
>>>> <scope>provided</scope>
>>>>
>>>> The pom file is attached. These are the errors I am getting
>>>>
>>>> [ERROR] error: error while loading package, class file
>>>> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/reflect/package.class)'
>>>> is broken
>>>> [ERROR] (class java.lang.RuntimeException/error reading Scala signature
>>>> of package.class: Scala signature package has wrong version
>>>> [ERROR] error: error while loading package, class file
>>>> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/package.class)'
>>>> is broken
>>>> [ERROR] (class java.lang.RuntimeException/error reading Scala signature
>>>> of package.class: Scala signature package has wrong version
>>>> [ERROR] error: error while loading package, class file
>>>> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/collection/package.class)'
>>>> is broken
>>>> [ERROR] (class java.lang.RuntimeException/error reading Scala signature
>>>> of package.class: Scala signature package has wrong version
>>>> [ERROR] three errors found
>>>> [ERROR] Failed to execute goal
>>>> org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project
>>>> scala: wrap: org.apache.commons.exec.ExecuteException: Process exited with
>>>> an error: 1(Exit value: 1) -> [Help 1]
>>>>
>>>> Any ideas will be appreciated.
>>>>
>>>> Mich Talebzadeh,
>>>> Lead Solutions Architect/Engineering Lead
>>>> Palantir Technologies Limited
>>>> London
>>>> United Kingdom
>>>>
>>>>
>>>> view my Linkedin profile
>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>
>>>>
>>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>>
>>>>
>>>>
>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>> any loss, damage or destruction of data or any other property which may
>>>> arise from relying on this email's technical content is explicitly
>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>> arising from such loss, damage or destruction.
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>>
>>>
>>>
>>> --
>>> Bjørn Jørgensen
>>> Vestre Aspehaug 4, 6010 Ålesund
>>> Norge
>>>
>>> +47 480 94 297
>>>
>>
>
> --
> Bjørn Jørgensen
> Vestre Aspehaug 4, 6010 Ålesund
> Norge
>
> +47 480 94 297
>
Re: Re: maven with Spark 3.4.0 fails compilation
Posted by Bjørn Jørgensen <bj...@gmail.com>.
Change
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.13.11-M2</version>
</dependency>
<dependency>
to
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
man. 29. mai 2023 kl. 13:20 skrev Lingzhe Sun <li...@hirain.com>:
> Hi Mich,
>
> Spark 3.4.0 prebuilt with scala 2.13 is built with version 2.13.8
> <https://github.com/apache/spark/blob/88f69d6f92860823b1a90bc162ebca2b7c8132fc/pom.xml#L170>.
> Since you are using spark-core_2.13 and spark-sql_2.13, you should stick to
> the major(13) and the minor version(8). Not using any of these may cause
> unexpected behaviour(though scala claims compatibility among minor version
> changes, I've encountered problem using the scala package with the same
> major version and different minor version. That may due to bug fixes and
> upgrade of scala itself.).
> And although I did not encountered such problem, this
> <https://stackoverflow.com/a/26411339/19476830>can be a a pitfall for you.
>
> ------------------------------
> Best Regards!
> ...........................................................................
> Lingzhe Sun
> Hirain Technology
>
>
> *From:* Mich Talebzadeh <mi...@gmail.com>
> *Date:* 2023-05-29 17:55
> *To:* Bjørn Jørgensen <bj...@gmail.com>
> *CC:* user @spark <us...@spark.apache.org>
> *Subject:* Re: maven with Spark 3.4.0 fails compilation
> Thanks for your helpful comments Bjorn.
>
> I managed to compile the code with maven but when it run it fails with
>
> Application is ReduceByKey
>
> Exception in thread "main" java.lang.NoSuchMethodError:
> scala.package$.Seq()Lscala/collection/immutable/Seq$;
> at ReduceByKey$.main(ReduceByKey.scala:23)
> at ReduceByKey.main(ReduceByKey.scala)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> at org.apache.spark.deploy.SparkSubmit.org
> $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
> at
> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
> at
> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
> at
> org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
> at
> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> I attach the pom.xml and the sample scala code is self contained and
> basic. Again it runs with SBT with no issues.
>
> FYI, my scala version on host is
>
> scala -version
> Scala code runner version 2.13.6 -- Copyright 2002-2021, LAMP/EPFL and
> Lightbend, Inc.
>
> I think I have a scala incompatible somewhere again
>
> Cheers
>
>
> Mich Talebzadeh,
> Lead Solutions Architect/Engineering Lead
> Palantir Technologies Limited
> London
> United Kingdom
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
> https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Sun, 28 May 2023 at 20:29, Bjørn Jørgensen <bj...@gmail.com>
> wrote:
>
>> From chatgpt4
>>
>>
>> The problem appears to be that there is a mismatch between the version of
>> Scala used by the Scala Maven plugin and the version of the Scala library
>> defined as a dependency in your POM. You've defined your Scala version in
>> your properties as `2.12.17` but you're pulling in `scala-library` version
>> `2.13.6` as a dependency.
>>
>> The Scala Maven plugin will be using the Scala version defined in the
>> `scala.version` property for compilation, but then it tries to load classes
>> from a different Scala version, hence the error.
>>
>> To resolve this issue, make sure the `scala.version` property matches the
>> version of `scala-library` defined in your dependencies. In your case, you
>> may want to change `scala.version` to `2.13.6`.
>>
>> Here's the corrected part of your POM:
>>
>> ```xml
>> <properties>
>> <maven.compiler.source>1.7</maven.compiler.source>
>> <maven.compiler.target>1.7</maven.compiler.target>
>> <encoding>UTF-8</encoding>
>> <scala.version>2.13.6</scala.version> <!-- Here's the change -->
>> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
>> </properties>
>> ```
>>
>> Additionally, ensure that the Scala versions in the Spark dependencies
>> match the `scala.version` property as well. If you've updated the Scala
>> version to `2.13.6`, the artifactIds for Spark dependencies should be
>> `spark-core_2.13` and `spark-sql_2.13`.
>>
>> Another thing to consider: your Java version defined in
>> `maven.compiler.source` and `maven.compiler.target` is `1.7`, which is
>> quite outdated and might not be compatible with the latest versions of
>> these libraries. Consider updating to a more recent version of Java, such
>> as Java 8 or above, depending on the requirements of the libraries you're
>> using.
>>
>>
>>
>> The same problem persists in this updated POM file - there's a mismatch
>> in the Scala version declared in the properties and the version used in
>> your dependencies. Here's what you need to update:
>>
>> 1. Update the Scala version in your properties to match the Scala library
>> and your Spark dependencies:
>>
>> ```xml
>> <properties>
>> <maven.compiler.source>1.7</maven.compiler.source>
>> <maven.compiler.target>1.7</maven.compiler.target>
>> <encoding>UTF-8</encoding>
>> <scala.version>2.13.6</scala.version> <!-- updated Scala version -->
>> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
>> </properties>
>> ```
>>
>> 2. Make sure all your Spark dependencies use the same Scala version. In
>> this case, I see `spark-streaming-kafka_2.11` which should be
>> `spark-streaming-kafka_2.13` if you're using Scala `2.13.6`.
>>
>> ```xml
>> <dependency>
>> <groupId>org.apache.spark</groupId>
>> <artifactId>spark-streaming-kafka_2.13</artifactId> <!-- updated to
>> 2.13 -->
>> <version>1.6.3</version> <!-- this might also need to be updated as
>> this is a very old version -->
>> <scope>provided</scope>
>> </dependency>
>> ```
>>
>> 3. As mentioned in the previous message, your Java version
>> (`maven.compiler.source` and `maven.compiler.target`) is also quite
>> outdated. Depending on the requirements of the libraries you're using, you
>> might want to update this to a newer version, such as Java 8 or above.
>>
>> Finally, ensure that the correct versions of these libraries exist in
>> your Maven repository or are available in the central Maven repository. If
>> the versions don't match, Maven will not be able to find and download the
>> correct dependencies, which can lead to problems.
>>
>> Please note that it's crucial to maintain consistency in your Scala and
>> Java versions across your project and its dependencies to avoid these kinds
>> of issues. Mixing versions can lead to binary incompatibility errors, such
>> as the one you're seeing.
>>
>>
>>
>>
>>
>>
>>
>> The differences in behavior between SBT and Maven when resolving Scala
>> dependencies can be attributed to how they each handle Scala binary
>> versions.
>>
>> SBT is specifically designed for Scala projects and has built-in support
>> for handling Scala's binary compatibility issues. When you define a Scala
>> library dependency in SBT, you can specify the Scala binary version as
>> "_2.12", "_2.13", etc. in the library artifact name. SBT will then
>> automatically replace this with the actual Scala binary version defined in
>> your SBT configuration. So, if you've defined different Scala versions for
>> your project and for a specific dependency, SBT will handle this gracefully.
>>
>> Maven, on the other hand, is more generic and does not have the same
>> level of built-in support for Scala's binary versions. In Maven, the Scala
>> version is typically hardcoded in the artifact name, and if this doesn't
>> match the actual Scala version used in your project, it can lead to binary
>> compatibility issues.
>>
>> This difference between SBT and Maven means that a project can work fine
>> when built with SBT but fail when built with Maven, due to these Scala
>> version discrepancies. To avoid these issues in Maven, you need to ensure
>> that the Scala version is consistent across your project configuration and
>> all your dependencies.
>>
>> Also, another reason for this can be because SBT often fetches
>> dependencies on the fly, while Maven requires them to be explicitly
>> declared. SBT can download and link the correct Scala library version even
>> if it's not explicitly declared, while Maven will only use the versions
>> that have been specified in the POM file.
>>
>> To summarize, SBT is more flexible and Scala-oriented than Maven, which
>> can lead to different behavior when handling Scala dependencies. When using
>> Maven for Scala projects, it's essential to ensure that all Scala versions
>> match across the project.
>>
>> søn. 28. mai 2023 kl. 21:03 skrev Mich Talebzadeh <
>> mich.talebzadeh@gmail.com>:
>>
>>> This compilation works fine with SBT but fails with maven!
>>>
>>> Spark version 3.4.0
>>> Apache Maven 3.9.2 (c9616018c7a021c1c39be70fb2843d6f5f9b8a1c)
>>> Java version: 11.0.1, vendor: Oracle Corporation, runtime:
>>> /opt/jdk-11.0.1
>>>
>>> This from the pom.xml file
>>>
>>> <project xmlns="https://maven.apache.org/POM/4.0.0" xmlns:xsi="
>>> https://www.w3.org/2001/XMLSchema-instance"
>>> xsi:schemaLocation="https://maven.apache.org/POM/4.0.0
>>> https://maven.apache.org/maven-v4_0_0.xsd">
>>> <modelVersion>4.0.0</modelVersion>
>>> <groupId>spark</groupId>
>>> <version>3.0</version>
>>> <name>${project.artifactId}</name>
>>>
>>> <properties>
>>> <maven.compiler.source>1.7</maven.compiler.source>
>>> <maven.compiler.target>1.7</maven.compiler.target>
>>> <encoding>UTF-8</encoding>
>>> <scala.version>2.12.17</scala.version>
>>> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
>>> </properties>
>>> <dependencies>
>>> <dependency>
>>> <groupId>org.scala-lang</groupId>
>>> <artifactId>scala-library</artifactId>
>>> <version>2.13.6</version>
>>> </dependency>
>>> <dependency>
>>> <groupId>org.apache.spark</groupId>
>>> <artifactId>spark-core_2.13</artifactId>
>>> <version>3.4.0</version>
>>> <scope>provided</scope>
>>> </dependency>
>>> <dependency>
>>> <groupId>org.apache.spark</groupId>
>>> <artifactId>spark-sql_2.13</artifactId>
>>> <version>3.4.0</version>
>>> <scope>provided</scope>
>>>
>>> The pom file is attached. These are the errors I am getting
>>>
>>> [ERROR] error: error while loading package, class file
>>> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/reflect/package.class)'
>>> is broken
>>> [ERROR] (class java.lang.RuntimeException/error reading Scala signature
>>> of package.class: Scala signature package has wrong version
>>> [ERROR] error: error while loading package, class file
>>> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/package.class)'
>>> is broken
>>> [ERROR] (class java.lang.RuntimeException/error reading Scala signature
>>> of package.class: Scala signature package has wrong version
>>> [ERROR] error: error while loading package, class file
>>> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/collection/package.class)'
>>> is broken
>>> [ERROR] (class java.lang.RuntimeException/error reading Scala signature
>>> of package.class: Scala signature package has wrong version
>>> [ERROR] three errors found
>>> [ERROR] Failed to execute goal
>>> org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project
>>> scala: wrap: org.apache.commons.exec.ExecuteException: Process exited with
>>> an error: 1(Exit value: 1) -> [Help 1]
>>>
>>> Any ideas will be appreciated.
>>>
>>> Mich Talebzadeh,
>>> Lead Solutions Architect/Engineering Lead
>>> Palantir Technologies Limited
>>> London
>>> United Kingdom
>>>
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>>
>>
>> --
>> Bjørn Jørgensen
>> Vestre Aspehaug 4, 6010 Ålesund
>> Norge
>>
>> +47 480 94 297
>>
>
--
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge
+47 480 94 297
Re: Re: maven with Spark 3.4.0 fails compilation
Posted by Lingzhe Sun <li...@hirain.com>.
Hi Mich,
Spark 3.4.0 prebuilt with scala 2.13 is built with version 2.13.8. Since you are using spark-core_2.13 and spark-sql_2.13, you should stick to the major(13) and the minor version(8). Not using any of these may cause unexpected behaviour(though scala claims compatibility among minor version changes, I've encountered problem using the scala package with the same major version and different minor version. That may due to bug fixes and upgrade of scala itself.).
And although I did not encountered such problem, this can be a a pitfall for you.
Best Regards!
...........................................................................
Lingzhe Sun
Hirain Technology
From: Mich Talebzadeh
Date: 2023-05-29 17:55
To: Bjørn Jørgensen
CC: user @spark
Subject: Re: maven with Spark 3.4.0 fails compilation
Thanks for your helpful comments Bjorn.
I managed to compile the code with maven but when it run it fails with
Application is ReduceByKey
Exception in thread "main" java.lang.NoSuchMethodError: scala.package$.Seq()Lscala/collection/immutable/Seq$;
at ReduceByKey$.main(ReduceByKey.scala:23)
at ReduceByKey.main(ReduceByKey.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
I attach the pom.xml and the sample scala code is self contained and basic. Again it runs with SBT with no issues.
FYI, my scala version on host is
scala -version
Scala code runner version 2.13.6 -- Copyright 2002-2021, LAMP/EPFL and Lightbend, Inc.
I think I have a scala incompatible somewhere again
Cheers
Mich Talebzadeh,
Lead Solutions Architect/Engineering Lead
Palantir Technologies Limited
London
United Kingdom
view my Linkedin profile
https://en.everybodywiki.com/Mich_Talebzadeh
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
On Sun, 28 May 2023 at 20:29, Bjørn Jørgensen <bj...@gmail.com> wrote:
From chatgpt4
The problem appears to be that there is a mismatch between the version of Scala used by the Scala Maven plugin and the version of the Scala library defined as a dependency in your POM. You've defined your Scala version in your properties as `2.12.17` but you're pulling in `scala-library` version `2.13.6` as a dependency.
The Scala Maven plugin will be using the Scala version defined in the `scala.version` property for compilation, but then it tries to load classes from a different Scala version, hence the error.
To resolve this issue, make sure the `scala.version` property matches the version of `scala-library` defined in your dependencies. In your case, you may want to change `scala.version` to `2.13.6`.
Here's the corrected part of your POM:
```xml
<properties>
<maven.compiler.source>1.7</maven.compiler.source>
<maven.compiler.target>1.7</maven.compiler.target>
<encoding>UTF-8</encoding>
<scala.version>2.13.6</scala.version> <!-- Here's the change -->
<maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
</properties>
```
Additionally, ensure that the Scala versions in the Spark dependencies match the `scala.version` property as well. If you've updated the Scala version to `2.13.6`, the artifactIds for Spark dependencies should be `spark-core_2.13` and `spark-sql_2.13`.
Another thing to consider: your Java version defined in `maven.compiler.source` and `maven.compiler.target` is `1.7`, which is quite outdated and might not be compatible with the latest versions of these libraries. Consider updating to a more recent version of Java, such as Java 8 or above, depending on the requirements of the libraries you're using.
The same problem persists in this updated POM file - there's a mismatch in the Scala version declared in the properties and the version used in your dependencies. Here's what you need to update:
1. Update the Scala version in your properties to match the Scala library and your Spark dependencies:
```xml
<properties>
<maven.compiler.source>1.7</maven.compiler.source>
<maven.compiler.target>1.7</maven.compiler.target>
<encoding>UTF-8</encoding>
<scala.version>2.13.6</scala.version> <!-- updated Scala version -->
<maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
</properties>
```
2. Make sure all your Spark dependencies use the same Scala version. In this case, I see `spark-streaming-kafka_2.11` which should be `spark-streaming-kafka_2.13` if you're using Scala `2.13.6`.
```xml
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka_2.13</artifactId> <!-- updated to 2.13 -->
<version>1.6.3</version> <!-- this might also need to be updated as this is a very old version -->
<scope>provided</scope>
</dependency>
```
3. As mentioned in the previous message, your Java version (`maven.compiler.source` and `maven.compiler.target`) is also quite outdated. Depending on the requirements of the libraries you're using, you might want to update this to a newer version, such as Java 8 or above.
Finally, ensure that the correct versions of these libraries exist in your Maven repository or are available in the central Maven repository. If the versions don't match, Maven will not be able to find and download the correct dependencies, which can lead to problems.
Please note that it's crucial to maintain consistency in your Scala and Java versions across your project and its dependencies to avoid these kinds of issues. Mixing versions can lead to binary incompatibility errors, such as the one you're seeing.
The differences in behavior between SBT and Maven when resolving Scala dependencies can be attributed to how they each handle Scala binary versions.
SBT is specifically designed for Scala projects and has built-in support for handling Scala's binary compatibility issues. When you define a Scala library dependency in SBT, you can specify the Scala binary version as "_2.12", "_2.13", etc. in the library artifact name. SBT will then automatically replace this with the actual Scala binary version defined in your SBT configuration. So, if you've defined different Scala versions for your project and for a specific dependency, SBT will handle this gracefully.
Maven, on the other hand, is more generic and does not have the same level of built-in support for Scala's binary versions. In Maven, the Scala version is typically hardcoded in the artifact name, and if this doesn't match the actual Scala version used in your project, it can lead to binary compatibility issues.
This difference between SBT and Maven means that a project can work fine when built with SBT but fail when built with Maven, due to these Scala version discrepancies. To avoid these issues in Maven, you need to ensure that the Scala version is consistent across your project configuration and all your dependencies.
Also, another reason for this can be because SBT often fetches dependencies on the fly, while Maven requires them to be explicitly declared. SBT can download and link the correct Scala library version even if it's not explicitly declared, while Maven will only use the versions that have been specified in the POM file.
To summarize, SBT is more flexible and Scala-oriented than Maven, which can lead to different behavior when handling Scala dependencies. When using Maven for Scala projects, it's essential to ensure that all Scala versions match across the project.
søn. 28. mai 2023 kl. 21:03 skrev Mich Talebzadeh <mi...@gmail.com>:
This compilation works fine with SBT but fails with maven!
Spark version 3.4.0
Apache Maven 3.9.2 (c9616018c7a021c1c39be70fb2843d6f5f9b8a1c)
Java version: 11.0.1, vendor: Oracle Corporation, runtime: /opt/jdk-11.0.1
This from the pom.xml file
<project xmlns="https://maven.apache.org/POM/4.0.0" xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="https://maven.apache.org/POM/4.0.0 https://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>spark</groupId>
<version>3.0</version>
<name>${project.artifactId}</name>
<properties>
<maven.compiler.source>1.7</maven.compiler.source>
<maven.compiler.target>1.7</maven.compiler.target>
<encoding>UTF-8</encoding>
<scala.version>2.12.17</scala.version>
<maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
</properties>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.13.6</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.13</artifactId>
<version>3.4.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.13</artifactId>
<version>3.4.0</version>
<scope>provided</scope>
The pom file is attached. These are the errors I am getting
[ERROR] error: error while loading package, class file '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/reflect/package.class)' is broken
[ERROR] (class java.lang.RuntimeException/error reading Scala signature of package.class: Scala signature package has wrong version
[ERROR] error: error while loading package, class file '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/package.class)' is broken
[ERROR] (class java.lang.RuntimeException/error reading Scala signature of package.class: Scala signature package has wrong version
[ERROR] error: error while loading package, class file '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/collection/package.class)' is broken
[ERROR] (class java.lang.RuntimeException/error reading Scala signature of package.class: Scala signature package has wrong version
[ERROR] three errors found
[ERROR] Failed to execute goal org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project scala: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1) -> [Help 1]
Any ideas will be appreciated.
Mich Talebzadeh,
Lead Solutions Architect/Engineering Lead
Palantir Technologies Limited
London
United Kingdom
view my Linkedin profile
https://en.everybodywiki.com/Mich_Talebzadeh
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
--
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge
+47 480 94 297
Re: maven with Spark 3.4.0 fails compilation
Posted by Mich Talebzadeh <mi...@gmail.com>.
Thanks for your helpful comments Bjorn.
I managed to compile the code with maven but when it run it fails with
Application is ReduceByKey
Exception in thread "main" java.lang.NoSuchMethodError:
scala.package$.Seq()Lscala/collection/immutable/Seq$;
at ReduceByKey$.main(ReduceByKey.scala:23)
at ReduceByKey.main(ReduceByKey.scala)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org
$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
at
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
I attach the pom.xml and the sample scala code is self contained and basic.
Again it runs with SBT with no issues.
FYI, my scala version on host is
scala -version
Scala code runner version 2.13.6 -- Copyright 2002-2021, LAMP/EPFL and
Lightbend, Inc.
I think I have a scala incompatible somewhere again
Cheers
Mich Talebzadeh,
Lead Solutions Architect/Engineering Lead
Palantir Technologies Limited
London
United Kingdom
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Sun, 28 May 2023 at 20:29, Bjørn Jørgensen <bj...@gmail.com>
wrote:
> From chatgpt4
>
>
> The problem appears to be that there is a mismatch between the version of
> Scala used by the Scala Maven plugin and the version of the Scala library
> defined as a dependency in your POM. You've defined your Scala version in
> your properties as `2.12.17` but you're pulling in `scala-library` version
> `2.13.6` as a dependency.
>
> The Scala Maven plugin will be using the Scala version defined in the
> `scala.version` property for compilation, but then it tries to load classes
> from a different Scala version, hence the error.
>
> To resolve this issue, make sure the `scala.version` property matches the
> version of `scala-library` defined in your dependencies. In your case, you
> may want to change `scala.version` to `2.13.6`.
>
> Here's the corrected part of your POM:
>
> ```xml
> <properties>
> <maven.compiler.source>1.7</maven.compiler.source>
> <maven.compiler.target>1.7</maven.compiler.target>
> <encoding>UTF-8</encoding>
> <scala.version>2.13.6</scala.version> <!-- Here's the change -->
> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
> </properties>
> ```
>
> Additionally, ensure that the Scala versions in the Spark dependencies
> match the `scala.version` property as well. If you've updated the Scala
> version to `2.13.6`, the artifactIds for Spark dependencies should be
> `spark-core_2.13` and `spark-sql_2.13`.
>
> Another thing to consider: your Java version defined in
> `maven.compiler.source` and `maven.compiler.target` is `1.7`, which is
> quite outdated and might not be compatible with the latest versions of
> these libraries. Consider updating to a more recent version of Java, such
> as Java 8 or above, depending on the requirements of the libraries you're
> using.
>
>
>
> The same problem persists in this updated POM file - there's a mismatch in
> the Scala version declared in the properties and the version used in your
> dependencies. Here's what you need to update:
>
> 1. Update the Scala version in your properties to match the Scala library
> and your Spark dependencies:
>
> ```xml
> <properties>
> <maven.compiler.source>1.7</maven.compiler.source>
> <maven.compiler.target>1.7</maven.compiler.target>
> <encoding>UTF-8</encoding>
> <scala.version>2.13.6</scala.version> <!-- updated Scala version -->
> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
> </properties>
> ```
>
> 2. Make sure all your Spark dependencies use the same Scala version. In
> this case, I see `spark-streaming-kafka_2.11` which should be
> `spark-streaming-kafka_2.13` if you're using Scala `2.13.6`.
>
> ```xml
> <dependency>
> <groupId>org.apache.spark</groupId>
> <artifactId>spark-streaming-kafka_2.13</artifactId> <!-- updated to
> 2.13 -->
> <version>1.6.3</version> <!-- this might also need to be updated as
> this is a very old version -->
> <scope>provided</scope>
> </dependency>
> ```
>
> 3. As mentioned in the previous message, your Java version
> (`maven.compiler.source` and `maven.compiler.target`) is also quite
> outdated. Depending on the requirements of the libraries you're using, you
> might want to update this to a newer version, such as Java 8 or above.
>
> Finally, ensure that the correct versions of these libraries exist in your
> Maven repository or are available in the central Maven repository. If the
> versions don't match, Maven will not be able to find and download the
> correct dependencies, which can lead to problems.
>
> Please note that it's crucial to maintain consistency in your Scala and
> Java versions across your project and its dependencies to avoid these kinds
> of issues. Mixing versions can lead to binary incompatibility errors, such
> as the one you're seeing.
>
>
>
>
>
>
>
> The differences in behavior between SBT and Maven when resolving Scala
> dependencies can be attributed to how they each handle Scala binary
> versions.
>
> SBT is specifically designed for Scala projects and has built-in support
> for handling Scala's binary compatibility issues. When you define a Scala
> library dependency in SBT, you can specify the Scala binary version as
> "_2.12", "_2.13", etc. in the library artifact name. SBT will then
> automatically replace this with the actual Scala binary version defined in
> your SBT configuration. So, if you've defined different Scala versions for
> your project and for a specific dependency, SBT will handle this gracefully.
>
> Maven, on the other hand, is more generic and does not have the same level
> of built-in support for Scala's binary versions. In Maven, the Scala
> version is typically hardcoded in the artifact name, and if this doesn't
> match the actual Scala version used in your project, it can lead to binary
> compatibility issues.
>
> This difference between SBT and Maven means that a project can work fine
> when built with SBT but fail when built with Maven, due to these Scala
> version discrepancies. To avoid these issues in Maven, you need to ensure
> that the Scala version is consistent across your project configuration and
> all your dependencies.
>
> Also, another reason for this can be because SBT often fetches
> dependencies on the fly, while Maven requires them to be explicitly
> declared. SBT can download and link the correct Scala library version even
> if it's not explicitly declared, while Maven will only use the versions
> that have been specified in the POM file.
>
> To summarize, SBT is more flexible and Scala-oriented than Maven, which
> can lead to different behavior when handling Scala dependencies. When using
> Maven for Scala projects, it's essential to ensure that all Scala versions
> match across the project.
>
> søn. 28. mai 2023 kl. 21:03 skrev Mich Talebzadeh <
> mich.talebzadeh@gmail.com>:
>
>> This compilation works fine with SBT but fails with maven!
>>
>> Spark version 3.4.0
>> Apache Maven 3.9.2 (c9616018c7a021c1c39be70fb2843d6f5f9b8a1c)
>> Java version: 11.0.1, vendor: Oracle Corporation, runtime: /opt/jdk-11.0.1
>>
>> This from the pom.xml file
>>
>> <project xmlns="https://maven.apache.org/POM/4.0.0" xmlns:xsi="
>> https://www.w3.org/2001/XMLSchema-instance"
>> xsi:schemaLocation="https://maven.apache.org/POM/4.0.0
>> https://maven.apache.org/maven-v4_0_0.xsd">
>> <modelVersion>4.0.0</modelVersion>
>> <groupId>spark</groupId>
>> <version>3.0</version>
>> <name>${project.artifactId}</name>
>>
>> <properties>
>> <maven.compiler.source>1.7</maven.compiler.source>
>> <maven.compiler.target>1.7</maven.compiler.target>
>> <encoding>UTF-8</encoding>
>> <scala.version>2.12.17</scala.version>
>> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
>> </properties>
>> <dependencies>
>> <dependency>
>> <groupId>org.scala-lang</groupId>
>> <artifactId>scala-library</artifactId>
>> <version>2.13.6</version>
>> </dependency>
>> <dependency>
>> <groupId>org.apache.spark</groupId>
>> <artifactId>spark-core_2.13</artifactId>
>> <version>3.4.0</version>
>> <scope>provided</scope>
>> </dependency>
>> <dependency>
>> <groupId>org.apache.spark</groupId>
>> <artifactId>spark-sql_2.13</artifactId>
>> <version>3.4.0</version>
>> <scope>provided</scope>
>>
>> The pom file is attached. These are the errors I am getting
>>
>> [ERROR] error: error while loading package, class file
>> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/reflect/package.class)'
>> is broken
>> [ERROR] (class java.lang.RuntimeException/error reading Scala signature
>> of package.class: Scala signature package has wrong version
>> [ERROR] error: error while loading package, class file
>> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/package.class)'
>> is broken
>> [ERROR] (class java.lang.RuntimeException/error reading Scala signature
>> of package.class: Scala signature package has wrong version
>> [ERROR] error: error while loading package, class file
>> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/collection/package.class)'
>> is broken
>> [ERROR] (class java.lang.RuntimeException/error reading Scala signature
>> of package.class: Scala signature package has wrong version
>> [ERROR] three errors found
>> [ERROR] Failed to execute goal
>> org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project
>> scala: wrap: org.apache.commons.exec.ExecuteException: Process exited with
>> an error: 1(Exit value: 1) -> [Help 1]
>>
>> Any ideas will be appreciated.
>>
>> Mich Talebzadeh,
>> Lead Solutions Architect/Engineering Lead
>> Palantir Technologies Limited
>> London
>> United Kingdom
>>
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>> https://en.everybodywiki.com/Mich_Talebzadeh
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
>
> --
> Bjørn Jørgensen
> Vestre Aspehaug 4, 6010 Ålesund
> Norge
>
> +47 480 94 297
>
Re: maven with Spark 3.4.0 fails compilation
Posted by Bjørn Jørgensen <bj...@gmail.com>.
From chatgpt4
The problem appears to be that there is a mismatch between the version of
Scala used by the Scala Maven plugin and the version of the Scala library
defined as a dependency in your POM. You've defined your Scala version in
your properties as `2.12.17` but you're pulling in `scala-library` version
`2.13.6` as a dependency.
The Scala Maven plugin will be using the Scala version defined in the
`scala.version` property for compilation, but then it tries to load classes
from a different Scala version, hence the error.
To resolve this issue, make sure the `scala.version` property matches the
version of `scala-library` defined in your dependencies. In your case, you
may want to change `scala.version` to `2.13.6`.
Here's the corrected part of your POM:
```xml
<properties>
<maven.compiler.source>1.7</maven.compiler.source>
<maven.compiler.target>1.7</maven.compiler.target>
<encoding>UTF-8</encoding>
<scala.version>2.13.6</scala.version> <!-- Here's the change -->
<maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
</properties>
```
Additionally, ensure that the Scala versions in the Spark dependencies
match the `scala.version` property as well. If you've updated the Scala
version to `2.13.6`, the artifactIds for Spark dependencies should be
`spark-core_2.13` and `spark-sql_2.13`.
Another thing to consider: your Java version defined in
`maven.compiler.source` and `maven.compiler.target` is `1.7`, which is
quite outdated and might not be compatible with the latest versions of
these libraries. Consider updating to a more recent version of Java, such
as Java 8 or above, depending on the requirements of the libraries you're
using.
The same problem persists in this updated POM file - there's a mismatch in
the Scala version declared in the properties and the version used in your
dependencies. Here's what you need to update:
1. Update the Scala version in your properties to match the Scala library
and your Spark dependencies:
```xml
<properties>
<maven.compiler.source>1.7</maven.compiler.source>
<maven.compiler.target>1.7</maven.compiler.target>
<encoding>UTF-8</encoding>
<scala.version>2.13.6</scala.version> <!-- updated Scala version -->
<maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
</properties>
```
2. Make sure all your Spark dependencies use the same Scala version. In
this case, I see `spark-streaming-kafka_2.11` which should be
`spark-streaming-kafka_2.13` if you're using Scala `2.13.6`.
```xml
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka_2.13</artifactId> <!-- updated to
2.13 -->
<version>1.6.3</version> <!-- this might also need to be updated as
this is a very old version -->
<scope>provided</scope>
</dependency>
```
3. As mentioned in the previous message, your Java version
(`maven.compiler.source` and `maven.compiler.target`) is also quite
outdated. Depending on the requirements of the libraries you're using, you
might want to update this to a newer version, such as Java 8 or above.
Finally, ensure that the correct versions of these libraries exist in your
Maven repository or are available in the central Maven repository. If the
versions don't match, Maven will not be able to find and download the
correct dependencies, which can lead to problems.
Please note that it's crucial to maintain consistency in your Scala and
Java versions across your project and its dependencies to avoid these kinds
of issues. Mixing versions can lead to binary incompatibility errors, such
as the one you're seeing.
The differences in behavior between SBT and Maven when resolving Scala
dependencies can be attributed to how they each handle Scala binary
versions.
SBT is specifically designed for Scala projects and has built-in support
for handling Scala's binary compatibility issues. When you define a Scala
library dependency in SBT, you can specify the Scala binary version as
"_2.12", "_2.13", etc. in the library artifact name. SBT will then
automatically replace this with the actual Scala binary version defined in
your SBT configuration. So, if you've defined different Scala versions for
your project and for a specific dependency, SBT will handle this gracefully.
Maven, on the other hand, is more generic and does not have the same level
of built-in support for Scala's binary versions. In Maven, the Scala
version is typically hardcoded in the artifact name, and if this doesn't
match the actual Scala version used in your project, it can lead to binary
compatibility issues.
This difference between SBT and Maven means that a project can work fine
when built with SBT but fail when built with Maven, due to these Scala
version discrepancies. To avoid these issues in Maven, you need to ensure
that the Scala version is consistent across your project configuration and
all your dependencies.
Also, another reason for this can be because SBT often fetches dependencies
on the fly, while Maven requires them to be explicitly declared. SBT can
download and link the correct Scala library version even if it's not
explicitly declared, while Maven will only use the versions that have been
specified in the POM file.
To summarize, SBT is more flexible and Scala-oriented than Maven, which can
lead to different behavior when handling Scala dependencies. When using
Maven for Scala projects, it's essential to ensure that all Scala versions
match across the project.
søn. 28. mai 2023 kl. 21:03 skrev Mich Talebzadeh <mich.talebzadeh@gmail.com
>:
> This compilation works fine with SBT but fails with maven!
>
> Spark version 3.4.0
> Apache Maven 3.9.2 (c9616018c7a021c1c39be70fb2843d6f5f9b8a1c)
> Java version: 11.0.1, vendor: Oracle Corporation, runtime: /opt/jdk-11.0.1
>
> This from the pom.xml file
>
> <project xmlns="https://maven.apache.org/POM/4.0.0" xmlns:xsi="
> https://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="https://maven.apache.org/POM/4.0.0
> https://maven.apache.org/maven-v4_0_0.xsd">
> <modelVersion>4.0.0</modelVersion>
> <groupId>spark</groupId>
> <version>3.0</version>
> <name>${project.artifactId}</name>
>
> <properties>
> <maven.compiler.source>1.7</maven.compiler.source>
> <maven.compiler.target>1.7</maven.compiler.target>
> <encoding>UTF-8</encoding>
> <scala.version>2.12.17</scala.version>
> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
> </properties>
> <dependencies>
> <dependency>
> <groupId>org.scala-lang</groupId>
> <artifactId>scala-library</artifactId>
> <version>2.13.6</version>
> </dependency>
> <dependency>
> <groupId>org.apache.spark</groupId>
> <artifactId>spark-core_2.13</artifactId>
> <version>3.4.0</version>
> <scope>provided</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.spark</groupId>
> <artifactId>spark-sql_2.13</artifactId>
> <version>3.4.0</version>
> <scope>provided</scope>
>
> The pom file is attached. These are the errors I am getting
>
> [ERROR] error: error while loading package, class file
> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/reflect/package.class)'
> is broken
> [ERROR] (class java.lang.RuntimeException/error reading Scala signature of
> package.class: Scala signature package has wrong version
> [ERROR] error: error while loading package, class file
> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/package.class)'
> is broken
> [ERROR] (class java.lang.RuntimeException/error reading Scala signature of
> package.class: Scala signature package has wrong version
> [ERROR] error: error while loading package, class file
> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/collection/package.class)'
> is broken
> [ERROR] (class java.lang.RuntimeException/error reading Scala signature of
> package.class: Scala signature package has wrong version
> [ERROR] three errors found
> [ERROR] Failed to execute goal
> org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project
> scala: wrap: org.apache.commons.exec.ExecuteException: Process exited with
> an error: 1(Exit value: 1) -> [Help 1]
>
> Any ideas will be appreciated.
>
> Mich Talebzadeh,
> Lead Solutions Architect/Engineering Lead
> Palantir Technologies Limited
> London
> United Kingdom
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
> https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
--
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge
+47 480 94 297