You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Aurélien Mazoyer <au...@aepsilon.com> on 2022/01/21 23:49:14 UTC
Migration to Spark 3.2
Hello,
I migrated my code to Spark 3.2 and I am facing some issues. When I run my
unit tests via Maven, I get this error:
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.spark.rdd.RDDOperationScope$
which is not super nice.
However, when I run my test via Intellij, I get the following one:
java.lang.ExceptionInInitializerError
at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
at org.apache.spark.rdd.RDD.map(RDD.scala:421)
...
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Scala
module 2.12.3 requires Jackson Databind version >= 2.12.0 and < 2.13.0
which is far better imo since it gives me some clue on what is missing in
my pom.xml file to make it work. After putting a few more dependencies, my
tests are again passing.... in intellij, but I am stuck on the same error
when I am running maven command :-/.
It seems that jdk and maven versions are the same and both are using the
same .m2 directory.
Any clue on what can be going wrong?
Thank you,
Aurelien
Re: Migration to Spark 3.2
Posted by Aurélien Mazoyer <au...@aepsilon.com>.
Hello,
Sorry for asking twice, but anyone has any idea which issue I could be
facing with this depencency problem :-/?
Thank you,
Aurelien
Le sam. 22 janv. 2022 à 00:49, Aurélien Mazoyer <au...@aepsilon.com> a
écrit :
> Hello,
>
> I migrated my code to Spark 3.2 and I am facing some issues. When I run my
> unit tests via Maven, I get this error:
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.spark.rdd.RDDOperationScope$
> which is not super nice.
>
> However, when I run my test via Intellij, I get the following one:
> java.lang.ExceptionInInitializerError
> at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
> at org.apache.spark.rdd.RDD.map(RDD.scala:421)
> ...
> Caused by: com.fasterxml.jackson.databind.JsonMappingException: Scala
> module 2.12.3 requires Jackson Databind version >= 2.12.0 and < 2.13.0
> which is far better imo since it gives me some clue on what is missing in
> my pom.xml file to make it work. After putting a few more dependencies, my
> tests are again passing.... in intellij, but I am stuck on the same error
> when I am running maven command :-/.
> It seems that jdk and maven versions are the same and both are using the
> same .m2 directory.
> Any clue on what can be going wrong?
>
> Thank you,
>
> Aurelien
>
Re: Migration to Spark 3.2
Posted by Aurélien Mazoyer <au...@aepsilon.com>.
Hi Stephen,
I managed to solve my issue, I had a conflicting version of jackson
databind that came from parent pom.
Thank you,
Aurelien
Le dim. 30 janv. 2022 à 23:28, Aurélien Mazoyer <au...@aepsilon.com> a
écrit :
> Hi Stephen,
>
> Thank you for your answer. Yes, I changed the scope to "provided" but got
> the same error :-( FYI. I am getting this error while running tests.
>
> Regards,
>
> Aurelien
>
> Le jeu. 27 janv. 2022 à 23:57, Stephen Coy <sc...@infomedia.com.au> a
> écrit :
>
>> Hi Aurélien,
>>
>> Your Jackson versions look fine.
>>
>> What happens if you change the scope of your Jackson dependencies to
>> “provided”?
>>
>> This should result in your application using the versions provided by
>> Spark and avoid this potential collision.
>>
>> Cheers,
>>
>> Steve C
>>
>> On 27 Jan 2022, at 9:48 pm, Aurélien Mazoyer <au...@aepsilon.com>
>> wrote:
>>
>> Hi Stephen,
>>
>> Thank you for your answer!
>> Here it is, it seems that jackson dependencies are correct, no? :
>>
>> Thanks,
>>
>> [INFO] com.krrier:spark-lib-full:jar:0.0.1-SNAPSHOT
>> [INFO] +- com.krrier:backend:jar:0.0.1-SNAPSHOT:compile
>> [INFO] | \- com.krrier:data:jar:0.0.1-SNAPSHOT:compile
>> [INFO] +- com.krrier:plugin-api:jar:0.0.1-SNAPSHOT:compile
>> [INFO] +- com.opencsv:opencsv:jar:4.2:compile
>> [INFO] | +- org.apache.commons:commons-lang3:jar:3.9:compile
>> [INFO] | +- org.apache.commons:commons-text:jar:1.3:compile
>> [INFO] | +- commons-beanutils:commons-beanutils:jar:1.9.3:compile
>> [INFO] | | \- commons-logging:commons-logging:jar:1.2:compile
>> [INFO] | \- org.apache.commons:commons-collections4:jar:4.1:compile
>> [INFO] +- org.apache.solr:solr-solrj:jar:7.4.0:compile
>> [INFO] | +- org.apache.commons:commons-math3:jar:3.6.1:compile
>> [INFO] | +- org.apache.httpcomponents:httpclient:jar:4.5.3:compile
>> [INFO] | +- org.apache.httpcomponents:httpcore:jar:4.4.6:compile
>> [INFO] | +- org.apache.httpcomponents:httpmime:jar:4.5.3:compile
>> [INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.11:compile
>> [INFO] | +- org.codehaus.woodstox:stax2-api:jar:3.1.4:compile
>> [INFO] | +- org.codehaus.woodstox:woodstox-core-asl:jar:4.4.1:compile
>> [INFO] | \- org.noggit:noggit:jar:0.8:compile
>> [INFO] +- com.databricks:spark-xml_2.12:jar:0.5.0:compile
>> [INFO] +- org.apache.tika:tika-parsers:jar:1.24:compile
>> [INFO] | +- org.apache.tika:tika-core:jar:1.24:compile
>> [INFO] | +- org.glassfish.jaxb:jaxb-runtime:jar:2.3.2:compile
>> [INFO] | | +- jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.2:compile
>> [INFO] | | +- org.glassfish.jaxb:txw2:jar:2.3.2:compile
>> [INFO] | | +- com.sun.istack:istack-commons-runtime:jar:3.0.8:compile
>> [INFO] | | +- org.jvnet.staxex:stax-ex:jar:1.8.1:compile
>> [INFO] | | \- com.sun.xml.fastinfoset:FastInfoset:jar:1.2.16:compile
>> [INFO] | +- com.sun.activation:jakarta.activation:jar:1.2.1:compile
>> [INFO] | +- xerces:xercesImpl:jar:2.12.0:compile
>> [INFO] | | \- xml-apis:xml-apis:jar:1.4.01:compile
>> [INFO] | +- javax.annotation:javax.annotation-api:jar:1.3.2:compile
>> [INFO] | +- org.gagravarr:vorbis-java-tika:jar:0.8:compile
>> [INFO] | +- org.tallison:jmatio:jar:1.5:compile
>> [INFO] | +- org.apache.james:apache-mime4j-core:jar:0.8.3:compile
>> [INFO] | +- org.apache.james:apache-mime4j-dom:jar:0.8.3:compile
>> [INFO] | +- org.tukaani:xz:jar:1.8:compile
>> [INFO] | +- com.epam:parso:jar:2.0.11:compile
>> [INFO] | +- org.brotli:dec:jar:0.1.2:compile
>> [INFO] | +- commons-codec:commons-codec:jar:1.13:compile
>> [INFO] | +- org.apache.pdfbox:pdfbox:jar:2.0.19:compile
>> [INFO] | | \- org.apache.pdfbox:fontbox:jar:2.0.19:compile
>> [INFO] | +- org.apache.pdfbox:pdfbox-tools:jar:2.0.19:compile
>> [INFO] | +- org.apache.pdfbox:preflight:jar:2.0.19:compile
>> [INFO] | | \- org.apache.pdfbox:xmpbox:jar:2.0.19:compile
>> [INFO] | +- org.apache.pdfbox:jempbox:jar:1.8.16:compile
>> [INFO] | +- org.bouncycastle:bcmail-jdk15on:jar:1.64:compile
>> [INFO] | | \- org.bouncycastle:bcpkix-jdk15on:jar:1.64:compile
>> [INFO] | +- org.bouncycastle:bcprov-jdk15on:jar:1.64:compile
>> [INFO] | +- org.apache.poi:poi:jar:4.1.2:compile
>> [INFO] | | \- com.zaxxer:SparseBitSet:jar:1.2:compile
>> [INFO] | +- org.apache.poi:poi-scratchpad:jar:4.1.2:compile
>> [INFO] | +- com.healthmarketscience.jackcess:jackcess:jar:3.0.1:compile
>> [INFO] | +-
>> com.healthmarketscience.jackcess:jackcess-encrypt:jar:3.0.0:compile
>> [INFO] | +- org.ccil.cowan.tagsoup:tagsoup:jar:1.2.1:compile
>> [INFO] | +- org.ow2.asm:asm:jar:7.3.1:compile
>> [INFO] | +- com.googlecode.mp4parser:isoparser:jar:1.1.22:compile
>> [INFO] | +- org.tallison:metadata-extractor:jar:2.13.0:compile
>> [INFO] | | \- org.tallison.xmp:xmpcore-shaded:jar:6.1.10:compile
>> [INFO] | | \- com.adobe.xmp:xmpcore:jar:6.1.10:compile
>> [INFO] | +- de.l3s.boilerpipe:boilerpipe:jar:1.1.0:compile
>> [INFO] | +- com.rometools:rome:jar:1.12.2:compile
>> [INFO] | | \- com.rometools:rome-utils:jar:1.12.2:compile
>> [INFO] | +- org.gagravarr:vorbis-java-core:jar:0.8:compile
>> [INFO] | +-
>> com.googlecode.juniversalchardet:juniversalchardet:jar:1.0.3:compile
>> [INFO] | +- org.codelibs:jhighlight:jar:1.0.3:compile
>> [INFO] | +- com.pff:java-libpst:jar:0.9.3:compile
>> [INFO] | +- com.github.junrar:junrar:jar:4.0.0:compile
>> [INFO] | +- org.apache.cxf:cxf-rt-rs-client:jar:3.3.5:compile
>> [INFO] | | +- org.apache.cxf:cxf-rt-transports-http:jar:3.3.5:compile
>> [INFO] | | +- org.apache.cxf:cxf-core:jar:3.3.5:compile
>> [INFO] | | | +- com.fasterxml.woodstox:woodstox-core:jar:5.0.3:compile
>> [INFO] | | | \-
>> org.apache.ws.xmlschema:xmlschema-core:jar:2.2.5:compile
>> [INFO] | | \- org.apache.cxf:cxf-rt-frontend-jaxrs:jar:3.3.5:compile
>> [INFO] | | \- org.apache.cxf:cxf-rt-security:jar:3.3.5:compile
>> [INFO] | +- org.apache.commons:commons-exec:jar:1.3:compile
>> [INFO] | +- org.apache.opennlp:opennlp-tools:jar:1.9.2:compile
>> [INFO] | +- com.googlecode.json-simple:json-simple:jar:1.1.1:compile
>> [INFO] | +- com.github.openjson:openjson:jar:1.0.11:compile
>> [INFO] | +- com.google.code.gson:gson:jar:2.8.6:compile
>> [INFO] | +- edu.ucar:netcdf4:jar:4.5.5:compile
>> [INFO] | | \- net.jcip:jcip-annotations:jar:1.0:compile
>> [INFO] | +- org.jdom:jdom2:jar:2.0.6:compile
>> [INFO] | +- com.google.guava:guava:jar:28.2-jre:compile
>> [INFO] | | +- com.google.guava:failureaccess:jar:1.0.1:compile
>> [INFO] | | +-
>> com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
>> [INFO] | | +- org.checkerframework:checker-qual:jar:2.10.0:compile
>> [INFO] | | +-
>> com.google.errorprone:error_prone_annotations:jar:2.3.4:compile
>> [INFO] | | \- com.google.j2objc:j2objc-annotations:jar:1.3:compile
>> [INFO] | +- edu.ucar:grib:jar:4.5.5:compile
>> [INFO] | | \- org.itadaki:bzip2:jar:0.9.1:compile
>> [INFO] | +- com.beust:jcommander:jar:1.78:compile
>> [INFO] | +- net.java.dev.jna:jna:jar:5.5.0:compile
>> [INFO] | +- org.jsoup:jsoup:jar:1.13.1:compile
>> [INFO] | +- com.google.protobuf:protobuf-java:jar:3.11.4:compile
>> [INFO] | +- edu.ucar:cdm:jar:4.5.5:compile
>> [INFO] | | +- edu.ucar:udunits:jar:4.5.5:compile
>> [INFO] | | +- joda-time:joda-time:jar:2.7:compile
>> [INFO] | | \- net.sf.ehcache:ehcache-core:jar:2.6.2:compile
>> [INFO] | +- org.quartz-scheduler:quartz:jar:2.3.2:compile
>> [INFO] | | \- com.zaxxer:HikariCP-java7:jar:2.4.13:compile
>> [INFO] | +- com.mchange:c3p0:jar:0.9.5.5:compile
>> [INFO] | | \- com.mchange:mchange-commons-java:jar:0.2.19:compile
>> [INFO] | +- edu.ucar:httpservices:jar:4.5.5:compile
>> [INFO] | +- org.apache.commons:commons-csv:jar:1.8:compile
>> [INFO] | +- org.apache.sis.core:sis-utility:jar:1.0:compile
>> [INFO] | | \- javax.measure:unit-api:jar:1.0:compile
>> [INFO] | +- org.apache.sis.storage:sis-netcdf:jar:1.0:compile
>> [INFO] | | +- org.apache.sis.storage:sis-storage:jar:1.0:compile
>> [INFO] | | | \- org.apache.sis.core:sis-feature:jar:1.0:compile
>> [INFO] | | \- org.apache.sis.core:sis-referencing:jar:1.0:compile
>> [INFO] | +- org.apache.sis.core:sis-metadata:jar:1.0:compile
>> [INFO] | +- org.opengis:geoapi:jar:3.0.1:compile
>> [INFO] | +- edu.usc.ir:sentiment-analysis-parser:jar:0.1:compile
>> [INFO] | +- org.apache.pdfbox:jbig2-imageio:jar:3.0.3:compile
>> [INFO] | \- com.github.jai-imageio:jai-imageio-core:jar:1.4.0:compile
>> [INFO] +- commons-cli:commons-cli:jar:1.4:compile
>> [INFO] +- org.apache.spark:spark-core_2.12:jar:3.2.0:provided
>> [INFO] | +- org.apache.avro:avro:jar:1.8.2:provided
>> [INFO] | | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:provided
>> [INFO] | | \-
>> org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:provided
>> [INFO] | +- org.apache.avro:avro-mapred:jar:1.10.2:provided
>> [INFO] | | \- org.apache.avro:avro-ipc:jar:1.10.2:provided
>> [INFO] | +- com.twitter:chill_2.12:jar:0.10.0:provided
>> [INFO] | | \- com.esotericsoftware:kryo-shaded:jar:4.0.2:provided
>> [INFO] | | \- com.esotericsoftware:minlog:jar:1.3.0:provided
>> [INFO] | +- com.twitter:chill-java:jar:0.10.0:provided
>> [INFO] | +- org.apache.xbean:xbean-asm9-shaded:jar:4.20:provided
>> [INFO] | +- org.apache.hadoop:hadoop-client-api:jar:3.3.1:provided
>> [INFO] | +- org.apache.hadoop:hadoop-client-runtime:jar:3.3.1:provided
>> [INFO] | | \-
>> org.apache.htrace:htrace-core4:jar:4.1.0-incubating:provided
>> [INFO] | +- org.apache.spark:spark-launcher_2.12:jar:3.2.0:provided
>> [INFO] | +- org.apache.spark:spark-kvstore_2.12:jar:3.2.0:provided
>> [INFO] | | \- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:provided
>> [INFO] | +- org.apache.spark:spark-network-common_2.12:jar:3.2.0:provided
>> [INFO] | | \- com.google.crypto.tink:tink:jar:1.6.0:provided
>> [INFO] | +-
>> org.apache.spark:spark-network-shuffle_2.12:jar:3.2.0:provided
>> [INFO] | +- org.apache.spark:spark-unsafe_2.12:jar:3.2.0:provided
>> [INFO] | +- javax.activation:activation:jar:1.1.1:provided
>> [INFO] | +- org.apache.curator:curator-recipes:jar:2.13.0:provided
>> [INFO] | | \- org.apache.curator:curator-framework:jar:2.13.0:provided
>> [INFO] | | \- org.apache.curator:curator-client:jar:2.13.0:provided
>> [INFO] | +- jakarta.servlet:jakarta.servlet-api:jar:4.0.3:provided
>> [INFO] | +- commons-collections:commons-collections:jar:3.2.2:compile
>> [INFO] | +- com.google.code.findbugs:jsr305:jar:3.0.0:compile
>> [INFO] | +- org.slf4j:slf4j-api:jar:1.7.7:provided
>> [INFO] | +- org.slf4j:jul-to-slf4j:jar:1.7.30:provided
>> [INFO] | +- org.slf4j:jcl-over-slf4j:jar:1.7.30:provided
>> [INFO] | +- log4j:log4j:jar:1.2.17:provided
>> [INFO] | +- org.slf4j:slf4j-log4j12:jar:1.7.7:provided
>> [INFO] | +- com.ning:compress-lzf:jar:1.0.3:provided
>> [INFO] | +- org.xerial.snappy:snappy-java:jar:1.1.8.4:provided
>> [INFO] | +- org.lz4:lz4-java:jar:1.7.1:provided
>> [INFO] | +- com.github.luben:zstd-jni:jar:1.5.0-4:provided
>> [INFO] | +- org.roaringbitmap:RoaringBitmap:jar:0.9.0:provided
>> [INFO] | | \- org.roaringbitmap:shims:jar:0.9.0:provided
>> [INFO] | +- commons-net:commons-net:jar:3.1:provided
>> [INFO] | +- org.scala-lang.modules:scala-xml_2.12:jar:1.2.0:compile
>> [INFO] | +- org.scala-lang:scala-library:jar:2.12.15:compile
>> [INFO] | +- org.scala-lang:scala-reflect:jar:2.12.15:provided
>> [INFO] | +- org.json4s:json4s-jackson_2.12:jar:3.7.0-M11:provided
>> [INFO] | | \- org.json4s:json4s-core_2.12:jar:3.7.0-M11:provided
>> [INFO] | | +- org.json4s:json4s-ast_2.12:jar:3.7.0-M11:provided
>> [INFO] | | \- org.json4s:json4s-scalap_2.12:jar:3.7.0-M11:provided
>> [INFO] | +- org.glassfish.jersey.core:jersey-client:jar:2.34:provided
>> [INFO] | | +- jakarta.ws.rs:jakarta.ws.rs-api:jar:2.1.6:compile
>> [INFO] | | \-
>> org.glassfish.hk2.external:jakarta.inject:jar:2.6.1:provided
>> [INFO] | +- org.glassfish.jersey.core:jersey-common:jar:2.34:provided
>> [INFO] | | +-
>> jakarta.annotation:jakarta.annotation-api:jar:1.3.5:provided
>> [INFO] | | \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.3:provided
>> [INFO] | +- org.glassfish.jersey.core:jersey-server:jar:2.34:provided
>> [INFO] | | \-
>> jakarta.validation:jakarta.validation-api:jar:2.0.2:provided
>> [INFO] | +-
>> org.glassfish.jersey.containers:jersey-container-servlet:jar:2.34:provided
>> [INFO] | +-
>> org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.34:provided
>> [INFO] | +- org.glassfish.jersey.inject:jersey-hk2:jar:2.34:provided
>> [INFO] | | +- org.glassfish.hk2:hk2-locator:jar:2.6.1:provided
>> [INFO] | | | +-
>> org.glassfish.hk2.external:aopalliance-repackaged:jar:2.6.1:provided
>> [INFO] | | | +- org.glassfish.hk2:hk2-api:jar:2.6.1:provided
>> [INFO] | | | \- org.glassfish.hk2:hk2-utils:jar:2.6.1:provided
>> [INFO] | | \- org.javassist:javassist:jar:3.25.0-GA:provided
>> [INFO] | +- io.netty:netty-all:jar:4.1.68.Final:provided
>> [INFO] | +- com.clearspring.analytics:stream:jar:2.9.6:provided
>> [INFO] | +- io.dropwizard.metrics:metrics-core:jar:4.2.0:provided
>> [INFO] | +- io.dropwizard.metrics:metrics-jvm:jar:4.2.0:provided
>> [INFO] | +- io.dropwizard.metrics:metrics-json:jar:4.2.0:provided
>> [INFO] | +- io.dropwizard.metrics:metrics-graphite:jar:4.2.0:provided
>> [INFO] | +- io.dropwizard.metrics:metrics-jmx:jar:4.2.0:provided
>> [INFO] | +-
>> com.fasterxml.jackson.module:jackson-module-scala_2.12:jar:2.12.3:provided
>> [INFO] | +- org.apache.ivy:ivy:jar:2.5.0:provided
>> [INFO] | +- oro:oro:jar:2.0.8:provided
>> [INFO] | +- net.razorvine:pyrolite:jar:4.30:provided
>> [INFO] | +- net.sf.py4j:py4j:jar:0.10.9.2:provided
>> [INFO] | +- org.apache.spark:spark-tags_2.12:jar:3.2.0:provided
>> [INFO] | +- org.apache.commons:commons-crypto:jar:1.1.0:provided
>> [INFO] | \- org.spark-project.spark:unused:jar:1.0.0:provided
>> [INFO] +- org.apache.spark:spark-sql_2.12:jar:3.2.0:provided
>> [INFO] | +- org.rocksdb:rocksdbjni:jar:6.20.3:provided
>> [INFO] | +- com.univocity:univocity-parsers:jar:2.9.1:provided
>> [INFO] | +- org.apache.spark:spark-sketch_2.12:jar:3.2.0:provided
>> [INFO] | +- org.apache.spark:spark-catalyst_2.12:jar:3.2.0:provided
>> [INFO] | | +-
>> org.scala-lang.modules:scala-parser-combinators_2.12:jar:1.1.2:provided
>> [INFO] | | +- org.codehaus.janino:janino:jar:3.0.16:provided
>> [INFO] | | +- org.codehaus.janino:commons-compiler:jar:3.0.16:provided
>> [INFO] | | +- org.antlr:antlr4-runtime:jar:4.7.1:provided
>> [INFO] | | +- javax.xml.bind:jaxb-api:jar:2.2.11:provided
>> [INFO] | | +- org.apache.arrow:arrow-vector:jar:2.0.0:provided
>> [INFO] | | | +- org.apache.arrow:arrow-format:jar:2.0.0:provided
>> [INFO] | | | +- org.apache.arrow:arrow-memory-core:jar:2.0.0:provided
>> [INFO] | | | \-
>> com.google.flatbuffers:flatbuffers-java:jar:1.9.0:provided
>> [INFO] | | \- org.apache.arrow:arrow-memory-netty:jar:2.0.0:provided
>> [INFO] | +- org.apache.orc:orc-core:jar:1.6.11:provided
>> [INFO] | | +- org.apache.orc:orc-shims:jar:1.6.11:provided
>> [INFO] | | +- io.airlift:aircompressor:jar:0.21:provided
>> [INFO] | | +- org.jetbrains:annotations:jar:17.0.0:provided
>> [INFO] | | \- org.threeten:threeten-extra:jar:1.5.0:provided
>> [INFO] | +- org.apache.orc:orc-mapreduce:jar:1.6.11:provided
>> [INFO] | +- org.apache.hive:hive-storage-api:jar:2.7.2:provided
>> [INFO] | +- org.apache.parquet:parquet-column:jar:1.12.1:provided
>> [INFO] | | +- org.apache.parquet:parquet-common:jar:1.12.1:provided
>> [INFO] | | | \-
>> org.apache.yetus:audience-annotations:jar:0.12.0:provided
>> [INFO] | | \- org.apache.parquet:parquet-encoding:jar:1.12.1:provided
>> [INFO] | \- org.apache.parquet:parquet-hadoop:jar:1.12.1:provided
>> [INFO] | +-
>> org.apache.parquet:parquet-format-structures:jar:1.12.1:provided
>> [INFO] | \- org.apache.parquet:parquet-jackson:jar:1.12.1:provided
>> [INFO] +- org.apache.spark:spark-avro_2.12:jar:3.2.0:provided
>> [INFO] +- org.junit.jupiter:junit-jupiter:jar:5.5.2:test
>> [INFO] | +- org.junit.jupiter:junit-jupiter-api:jar:5.5.2:test
>> [INFO] | | +- org.apiguardian:apiguardian-api:jar:1.1.0:test
>> [INFO] | | +- org.opentest4j:opentest4j:jar:1.2.0:test
>> [INFO] | | \- org.junit.platform:junit-platform-commons:jar:1.5.2:test
>> [INFO] | +- org.junit.jupiter:junit-jupiter-params:jar:5.5.2:test
>> [INFO] | \- org.junit.jupiter:junit-jupiter-engine:jar:5.5.2:test
>> [INFO] | \- org.junit.platform:junit-platform-engine:jar:1.5.2:test
>> [INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.12.6:compile
>> [INFO] +- com.fasterxml.jackson.core:jackson-core:jar:2.12.6:compile
>> [INFO] +- com.thoughtworks.paranamer:paranamer:jar:2.8:compile
>> [INFO] +-
>> com.fasterxml.jackson.core:jackson-annotations:jar:2.12.6:compile
>> [INFO] +- org.mockito:mockito-junit-jupiter:jar:2.23.0:test
>> [INFO] | \- org.mockito:mockito-core:jar:2.23.0:test
>> [INFO] | +- net.bytebuddy:byte-buddy:jar:1.9.0:test
>> [INFO] | +- net.bytebuddy:byte-buddy-agent:jar:1.9.0:test
>> [INFO] | \- org.objenesis:objenesis:jar:2.6:provided
>> [INFO] +- org.hamcrest:hamcrest-library:jar:1.3:test
>> [INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test
>> [INFO] +- com.crealytics:spark-excel_2.12:jar:3.2.0_0.16.0:compile
>> [INFO] | +- com.norbitltd:spoiwo_2.12:jar:2.0.0:compile
>> [INFO] | +- com.github.pjfanning:excel-streaming-reader:jar:3.2.4:compile
>> [INFO] | \- com.github.pjfanning:poi-shared-strings:jar:2.3.0:compile
>> [INFO] | \- com.h2database:h2:jar:2.0.202:runtime
>> [INFO] +- org.apache.commons:commons-compress:jar:1.21:compile
>> [INFO] +- commons-io:commons-io:jar:2.11.0:compile
>> [INFO] \- org.apache.poi:poi-ooxml:jar:5.2.0:compile
>> [INFO] +- org.apache.poi:poi-ooxml-lite:jar:5.2.0:compile
>> [INFO] +- org.apache.xmlbeans:xmlbeans:jar:5.0.3:compile
>> [INFO] +- com.github.virtuald:curvesapi:jar:1.06:compile
>> [INFO] \- org.apache.logging.log4j:log4j-api:jar:2.17.1:compile
>>
>>
>> Le jeu. 27 janv. 2022 à 00:39, Stephen Coy <sc...@infomedia.com.au> a
>> écrit :
>>
>>> Hi Aurélien!
>>>
>>> Please run
>>>
>>> mvn dependency:tree
>>>
>>> and check it for Jackson dependencies.
>>>
>>> Feel free to respond with the output if you have any questions about it.
>>>
>>> Cheers,
>>>
>>> Steve C
>>>
>>> > On 22 Jan 2022, at 10:49 am, Aurélien Mazoyer <au...@aepsilon.com>
>>> wrote:
>>> >
>>> > Hello,
>>> >
>>> > I migrated my code to Spark 3.2 and I am facing some issues. When I
>>> run my unit tests via Maven, I get this error:
>>> > java.lang.NoClassDefFoundError: Could not initialize class
>>> org.apache.spark.rdd.RDDOperationScope$
>>> > which is not super nice.
>>> >
>>> > However, when I run my test via Intellij, I get the following one:
>>> > java.lang.ExceptionInInitializerError
>>> > at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
>>> > at org.apache.spark.rdd.RDD.map(RDD.scala:421)
>>> > ...
>>> > Caused by: com.fasterxml.jackson.databind.JsonMappingException: Scala
>>> module 2.12.3 requires Jackson Databind version >= 2.12.0 and < 2.13.0
>>> > which is far better imo since it gives me some clue on what is missing
>>> in my pom.xml file to make it work. After putting a few more dependencies,
>>> my tests are again passing.... in intellij, but I am stuck on the same
>>> error when I am running maven command :-/.
>>> > It seems that jdk and maven versions are the same and both are using
>>> the same .m2 directory.
>>> > Any clue on what can be going wrong?
>>> >
>>> > Thank you,
>>> >
>>> > Aurelien
>>>
>>> This email contains confidential information of and is the copyright of
>>> Infomedia. It must not be forwarded, amended or disclosed without consent
>>> of the sender. If you received this message by mistake, please advise the
>>> sender and delete all copies. Security of transmission on the internet
>>> cannot be guaranteed, could be infected, intercepted, or corrupted and you
>>> should ensure you have suitable antivirus protection in place. By sending
>>> us your or any third party personal details, you consent to (or confirm you
>>> have obtained consent from such third parties) to Infomedia’s privacy
>>> policy. http://www.infomedia.com.au/privacy-policy/
>>>
>>
>>
Re: Migration to Spark 3.2
Posted by Aurélien Mazoyer <au...@aepsilon.com>.
Hi Stephen,
Thank you for your answer. Yes, I changed the scope to "provided" but got
the same error :-( FYI. I am getting this error while running tests.
Regards,
Aurelien
Le jeu. 27 janv. 2022 à 23:57, Stephen Coy <sc...@infomedia.com.au> a écrit :
> Hi Aurélien,
>
> Your Jackson versions look fine.
>
> What happens if you change the scope of your Jackson dependencies to
> “provided”?
>
> This should result in your application using the versions provided by
> Spark and avoid this potential collision.
>
> Cheers,
>
> Steve C
>
> On 27 Jan 2022, at 9:48 pm, Aurélien Mazoyer <au...@aepsilon.com>
> wrote:
>
> Hi Stephen,
>
> Thank you for your answer!
> Here it is, it seems that jackson dependencies are correct, no? :
>
> Thanks,
>
> [INFO] com.krrier:spark-lib-full:jar:0.0.1-SNAPSHOT
> [INFO] +- com.krrier:backend:jar:0.0.1-SNAPSHOT:compile
> [INFO] | \- com.krrier:data:jar:0.0.1-SNAPSHOT:compile
> [INFO] +- com.krrier:plugin-api:jar:0.0.1-SNAPSHOT:compile
> [INFO] +- com.opencsv:opencsv:jar:4.2:compile
> [INFO] | +- org.apache.commons:commons-lang3:jar:3.9:compile
> [INFO] | +- org.apache.commons:commons-text:jar:1.3:compile
> [INFO] | +- commons-beanutils:commons-beanutils:jar:1.9.3:compile
> [INFO] | | \- commons-logging:commons-logging:jar:1.2:compile
> [INFO] | \- org.apache.commons:commons-collections4:jar:4.1:compile
> [INFO] +- org.apache.solr:solr-solrj:jar:7.4.0:compile
> [INFO] | +- org.apache.commons:commons-math3:jar:3.6.1:compile
> [INFO] | +- org.apache.httpcomponents:httpclient:jar:4.5.3:compile
> [INFO] | +- org.apache.httpcomponents:httpcore:jar:4.4.6:compile
> [INFO] | +- org.apache.httpcomponents:httpmime:jar:4.5.3:compile
> [INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.11:compile
> [INFO] | +- org.codehaus.woodstox:stax2-api:jar:3.1.4:compile
> [INFO] | +- org.codehaus.woodstox:woodstox-core-asl:jar:4.4.1:compile
> [INFO] | \- org.noggit:noggit:jar:0.8:compile
> [INFO] +- com.databricks:spark-xml_2.12:jar:0.5.0:compile
> [INFO] +- org.apache.tika:tika-parsers:jar:1.24:compile
> [INFO] | +- org.apache.tika:tika-core:jar:1.24:compile
> [INFO] | +- org.glassfish.jaxb:jaxb-runtime:jar:2.3.2:compile
> [INFO] | | +- jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.2:compile
> [INFO] | | +- org.glassfish.jaxb:txw2:jar:2.3.2:compile
> [INFO] | | +- com.sun.istack:istack-commons-runtime:jar:3.0.8:compile
> [INFO] | | +- org.jvnet.staxex:stax-ex:jar:1.8.1:compile
> [INFO] | | \- com.sun.xml.fastinfoset:FastInfoset:jar:1.2.16:compile
> [INFO] | +- com.sun.activation:jakarta.activation:jar:1.2.1:compile
> [INFO] | +- xerces:xercesImpl:jar:2.12.0:compile
> [INFO] | | \- xml-apis:xml-apis:jar:1.4.01:compile
> [INFO] | +- javax.annotation:javax.annotation-api:jar:1.3.2:compile
> [INFO] | +- org.gagravarr:vorbis-java-tika:jar:0.8:compile
> [INFO] | +- org.tallison:jmatio:jar:1.5:compile
> [INFO] | +- org.apache.james:apache-mime4j-core:jar:0.8.3:compile
> [INFO] | +- org.apache.james:apache-mime4j-dom:jar:0.8.3:compile
> [INFO] | +- org.tukaani:xz:jar:1.8:compile
> [INFO] | +- com.epam:parso:jar:2.0.11:compile
> [INFO] | +- org.brotli:dec:jar:0.1.2:compile
> [INFO] | +- commons-codec:commons-codec:jar:1.13:compile
> [INFO] | +- org.apache.pdfbox:pdfbox:jar:2.0.19:compile
> [INFO] | | \- org.apache.pdfbox:fontbox:jar:2.0.19:compile
> [INFO] | +- org.apache.pdfbox:pdfbox-tools:jar:2.0.19:compile
> [INFO] | +- org.apache.pdfbox:preflight:jar:2.0.19:compile
> [INFO] | | \- org.apache.pdfbox:xmpbox:jar:2.0.19:compile
> [INFO] | +- org.apache.pdfbox:jempbox:jar:1.8.16:compile
> [INFO] | +- org.bouncycastle:bcmail-jdk15on:jar:1.64:compile
> [INFO] | | \- org.bouncycastle:bcpkix-jdk15on:jar:1.64:compile
> [INFO] | +- org.bouncycastle:bcprov-jdk15on:jar:1.64:compile
> [INFO] | +- org.apache.poi:poi:jar:4.1.2:compile
> [INFO] | | \- com.zaxxer:SparseBitSet:jar:1.2:compile
> [INFO] | +- org.apache.poi:poi-scratchpad:jar:4.1.2:compile
> [INFO] | +- com.healthmarketscience.jackcess:jackcess:jar:3.0.1:compile
> [INFO] | +-
> com.healthmarketscience.jackcess:jackcess-encrypt:jar:3.0.0:compile
> [INFO] | +- org.ccil.cowan.tagsoup:tagsoup:jar:1.2.1:compile
> [INFO] | +- org.ow2.asm:asm:jar:7.3.1:compile
> [INFO] | +- com.googlecode.mp4parser:isoparser:jar:1.1.22:compile
> [INFO] | +- org.tallison:metadata-extractor:jar:2.13.0:compile
> [INFO] | | \- org.tallison.xmp:xmpcore-shaded:jar:6.1.10:compile
> [INFO] | | \- com.adobe.xmp:xmpcore:jar:6.1.10:compile
> [INFO] | +- de.l3s.boilerpipe:boilerpipe:jar:1.1.0:compile
> [INFO] | +- com.rometools:rome:jar:1.12.2:compile
> [INFO] | | \- com.rometools:rome-utils:jar:1.12.2:compile
> [INFO] | +- org.gagravarr:vorbis-java-core:jar:0.8:compile
> [INFO] | +-
> com.googlecode.juniversalchardet:juniversalchardet:jar:1.0.3:compile
> [INFO] | +- org.codelibs:jhighlight:jar:1.0.3:compile
> [INFO] | +- com.pff:java-libpst:jar:0.9.3:compile
> [INFO] | +- com.github.junrar:junrar:jar:4.0.0:compile
> [INFO] | +- org.apache.cxf:cxf-rt-rs-client:jar:3.3.5:compile
> [INFO] | | +- org.apache.cxf:cxf-rt-transports-http:jar:3.3.5:compile
> [INFO] | | +- org.apache.cxf:cxf-core:jar:3.3.5:compile
> [INFO] | | | +- com.fasterxml.woodstox:woodstox-core:jar:5.0.3:compile
> [INFO] | | | \- org.apache.ws.xmlschema:xmlschema-core:jar:2.2.5:compile
> [INFO] | | \- org.apache.cxf:cxf-rt-frontend-jaxrs:jar:3.3.5:compile
> [INFO] | | \- org.apache.cxf:cxf-rt-security:jar:3.3.5:compile
> [INFO] | +- org.apache.commons:commons-exec:jar:1.3:compile
> [INFO] | +- org.apache.opennlp:opennlp-tools:jar:1.9.2:compile
> [INFO] | +- com.googlecode.json-simple:json-simple:jar:1.1.1:compile
> [INFO] | +- com.github.openjson:openjson:jar:1.0.11:compile
> [INFO] | +- com.google.code.gson:gson:jar:2.8.6:compile
> [INFO] | +- edu.ucar:netcdf4:jar:4.5.5:compile
> [INFO] | | \- net.jcip:jcip-annotations:jar:1.0:compile
> [INFO] | +- org.jdom:jdom2:jar:2.0.6:compile
> [INFO] | +- com.google.guava:guava:jar:28.2-jre:compile
> [INFO] | | +- com.google.guava:failureaccess:jar:1.0.1:compile
> [INFO] | | +-
> com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
> [INFO] | | +- org.checkerframework:checker-qual:jar:2.10.0:compile
> [INFO] | | +-
> com.google.errorprone:error_prone_annotations:jar:2.3.4:compile
> [INFO] | | \- com.google.j2objc:j2objc-annotations:jar:1.3:compile
> [INFO] | +- edu.ucar:grib:jar:4.5.5:compile
> [INFO] | | \- org.itadaki:bzip2:jar:0.9.1:compile
> [INFO] | +- com.beust:jcommander:jar:1.78:compile
> [INFO] | +- net.java.dev.jna:jna:jar:5.5.0:compile
> [INFO] | +- org.jsoup:jsoup:jar:1.13.1:compile
> [INFO] | +- com.google.protobuf:protobuf-java:jar:3.11.4:compile
> [INFO] | +- edu.ucar:cdm:jar:4.5.5:compile
> [INFO] | | +- edu.ucar:udunits:jar:4.5.5:compile
> [INFO] | | +- joda-time:joda-time:jar:2.7:compile
> [INFO] | | \- net.sf.ehcache:ehcache-core:jar:2.6.2:compile
> [INFO] | +- org.quartz-scheduler:quartz:jar:2.3.2:compile
> [INFO] | | \- com.zaxxer:HikariCP-java7:jar:2.4.13:compile
> [INFO] | +- com.mchange:c3p0:jar:0.9.5.5:compile
> [INFO] | | \- com.mchange:mchange-commons-java:jar:0.2.19:compile
> [INFO] | +- edu.ucar:httpservices:jar:4.5.5:compile
> [INFO] | +- org.apache.commons:commons-csv:jar:1.8:compile
> [INFO] | +- org.apache.sis.core:sis-utility:jar:1.0:compile
> [INFO] | | \- javax.measure:unit-api:jar:1.0:compile
> [INFO] | +- org.apache.sis.storage:sis-netcdf:jar:1.0:compile
> [INFO] | | +- org.apache.sis.storage:sis-storage:jar:1.0:compile
> [INFO] | | | \- org.apache.sis.core:sis-feature:jar:1.0:compile
> [INFO] | | \- org.apache.sis.core:sis-referencing:jar:1.0:compile
> [INFO] | +- org.apache.sis.core:sis-metadata:jar:1.0:compile
> [INFO] | +- org.opengis:geoapi:jar:3.0.1:compile
> [INFO] | +- edu.usc.ir:sentiment-analysis-parser:jar:0.1:compile
> [INFO] | +- org.apache.pdfbox:jbig2-imageio:jar:3.0.3:compile
> [INFO] | \- com.github.jai-imageio:jai-imageio-core:jar:1.4.0:compile
> [INFO] +- commons-cli:commons-cli:jar:1.4:compile
> [INFO] +- org.apache.spark:spark-core_2.12:jar:3.2.0:provided
> [INFO] | +- org.apache.avro:avro:jar:1.8.2:provided
> [INFO] | | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:provided
> [INFO] | | \- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:provided
> [INFO] | +- org.apache.avro:avro-mapred:jar:1.10.2:provided
> [INFO] | | \- org.apache.avro:avro-ipc:jar:1.10.2:provided
> [INFO] | +- com.twitter:chill_2.12:jar:0.10.0:provided
> [INFO] | | \- com.esotericsoftware:kryo-shaded:jar:4.0.2:provided
> [INFO] | | \- com.esotericsoftware:minlog:jar:1.3.0:provided
> [INFO] | +- com.twitter:chill-java:jar:0.10.0:provided
> [INFO] | +- org.apache.xbean:xbean-asm9-shaded:jar:4.20:provided
> [INFO] | +- org.apache.hadoop:hadoop-client-api:jar:3.3.1:provided
> [INFO] | +- org.apache.hadoop:hadoop-client-runtime:jar:3.3.1:provided
> [INFO] | | \-
> org.apache.htrace:htrace-core4:jar:4.1.0-incubating:provided
> [INFO] | +- org.apache.spark:spark-launcher_2.12:jar:3.2.0:provided
> [INFO] | +- org.apache.spark:spark-kvstore_2.12:jar:3.2.0:provided
> [INFO] | | \- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:provided
> [INFO] | +- org.apache.spark:spark-network-common_2.12:jar:3.2.0:provided
> [INFO] | | \- com.google.crypto.tink:tink:jar:1.6.0:provided
> [INFO] | +- org.apache.spark:spark-network-shuffle_2.12:jar:3.2.0:provided
> [INFO] | +- org.apache.spark:spark-unsafe_2.12:jar:3.2.0:provided
> [INFO] | +- javax.activation:activation:jar:1.1.1:provided
> [INFO] | +- org.apache.curator:curator-recipes:jar:2.13.0:provided
> [INFO] | | \- org.apache.curator:curator-framework:jar:2.13.0:provided
> [INFO] | | \- org.apache.curator:curator-client:jar:2.13.0:provided
> [INFO] | +- jakarta.servlet:jakarta.servlet-api:jar:4.0.3:provided
> [INFO] | +- commons-collections:commons-collections:jar:3.2.2:compile
> [INFO] | +- com.google.code.findbugs:jsr305:jar:3.0.0:compile
> [INFO] | +- org.slf4j:slf4j-api:jar:1.7.7:provided
> [INFO] | +- org.slf4j:jul-to-slf4j:jar:1.7.30:provided
> [INFO] | +- org.slf4j:jcl-over-slf4j:jar:1.7.30:provided
> [INFO] | +- log4j:log4j:jar:1.2.17:provided
> [INFO] | +- org.slf4j:slf4j-log4j12:jar:1.7.7:provided
> [INFO] | +- com.ning:compress-lzf:jar:1.0.3:provided
> [INFO] | +- org.xerial.snappy:snappy-java:jar:1.1.8.4:provided
> [INFO] | +- org.lz4:lz4-java:jar:1.7.1:provided
> [INFO] | +- com.github.luben:zstd-jni:jar:1.5.0-4:provided
> [INFO] | +- org.roaringbitmap:RoaringBitmap:jar:0.9.0:provided
> [INFO] | | \- org.roaringbitmap:shims:jar:0.9.0:provided
> [INFO] | +- commons-net:commons-net:jar:3.1:provided
> [INFO] | +- org.scala-lang.modules:scala-xml_2.12:jar:1.2.0:compile
> [INFO] | +- org.scala-lang:scala-library:jar:2.12.15:compile
> [INFO] | +- org.scala-lang:scala-reflect:jar:2.12.15:provided
> [INFO] | +- org.json4s:json4s-jackson_2.12:jar:3.7.0-M11:provided
> [INFO] | | \- org.json4s:json4s-core_2.12:jar:3.7.0-M11:provided
> [INFO] | | +- org.json4s:json4s-ast_2.12:jar:3.7.0-M11:provided
> [INFO] | | \- org.json4s:json4s-scalap_2.12:jar:3.7.0-M11:provided
> [INFO] | +- org.glassfish.jersey.core:jersey-client:jar:2.34:provided
> [INFO] | | +- jakarta.ws.rs:jakarta.ws.rs-api:jar:2.1.6:compile
> [INFO] | | \-
> org.glassfish.hk2.external:jakarta.inject:jar:2.6.1:provided
> [INFO] | +- org.glassfish.jersey.core:jersey-common:jar:2.34:provided
> [INFO] | | +-
> jakarta.annotation:jakarta.annotation-api:jar:1.3.5:provided
> [INFO] | | \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.3:provided
> [INFO] | +- org.glassfish.jersey.core:jersey-server:jar:2.34:provided
> [INFO] | | \-
> jakarta.validation:jakarta.validation-api:jar:2.0.2:provided
> [INFO] | +-
> org.glassfish.jersey.containers:jersey-container-servlet:jar:2.34:provided
> [INFO] | +-
> org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.34:provided
> [INFO] | +- org.glassfish.jersey.inject:jersey-hk2:jar:2.34:provided
> [INFO] | | +- org.glassfish.hk2:hk2-locator:jar:2.6.1:provided
> [INFO] | | | +-
> org.glassfish.hk2.external:aopalliance-repackaged:jar:2.6.1:provided
> [INFO] | | | +- org.glassfish.hk2:hk2-api:jar:2.6.1:provided
> [INFO] | | | \- org.glassfish.hk2:hk2-utils:jar:2.6.1:provided
> [INFO] | | \- org.javassist:javassist:jar:3.25.0-GA:provided
> [INFO] | +- io.netty:netty-all:jar:4.1.68.Final:provided
> [INFO] | +- com.clearspring.analytics:stream:jar:2.9.6:provided
> [INFO] | +- io.dropwizard.metrics:metrics-core:jar:4.2.0:provided
> [INFO] | +- io.dropwizard.metrics:metrics-jvm:jar:4.2.0:provided
> [INFO] | +- io.dropwizard.metrics:metrics-json:jar:4.2.0:provided
> [INFO] | +- io.dropwizard.metrics:metrics-graphite:jar:4.2.0:provided
> [INFO] | +- io.dropwizard.metrics:metrics-jmx:jar:4.2.0:provided
> [INFO] | +-
> com.fasterxml.jackson.module:jackson-module-scala_2.12:jar:2.12.3:provided
> [INFO] | +- org.apache.ivy:ivy:jar:2.5.0:provided
> [INFO] | +- oro:oro:jar:2.0.8:provided
> [INFO] | +- net.razorvine:pyrolite:jar:4.30:provided
> [INFO] | +- net.sf.py4j:py4j:jar:0.10.9.2:provided
> [INFO] | +- org.apache.spark:spark-tags_2.12:jar:3.2.0:provided
> [INFO] | +- org.apache.commons:commons-crypto:jar:1.1.0:provided
> [INFO] | \- org.spark-project.spark:unused:jar:1.0.0:provided
> [INFO] +- org.apache.spark:spark-sql_2.12:jar:3.2.0:provided
> [INFO] | +- org.rocksdb:rocksdbjni:jar:6.20.3:provided
> [INFO] | +- com.univocity:univocity-parsers:jar:2.9.1:provided
> [INFO] | +- org.apache.spark:spark-sketch_2.12:jar:3.2.0:provided
> [INFO] | +- org.apache.spark:spark-catalyst_2.12:jar:3.2.0:provided
> [INFO] | | +-
> org.scala-lang.modules:scala-parser-combinators_2.12:jar:1.1.2:provided
> [INFO] | | +- org.codehaus.janino:janino:jar:3.0.16:provided
> [INFO] | | +- org.codehaus.janino:commons-compiler:jar:3.0.16:provided
> [INFO] | | +- org.antlr:antlr4-runtime:jar:4.7.1:provided
> [INFO] | | +- javax.xml.bind:jaxb-api:jar:2.2.11:provided
> [INFO] | | +- org.apache.arrow:arrow-vector:jar:2.0.0:provided
> [INFO] | | | +- org.apache.arrow:arrow-format:jar:2.0.0:provided
> [INFO] | | | +- org.apache.arrow:arrow-memory-core:jar:2.0.0:provided
> [INFO] | | | \-
> com.google.flatbuffers:flatbuffers-java:jar:1.9.0:provided
> [INFO] | | \- org.apache.arrow:arrow-memory-netty:jar:2.0.0:provided
> [INFO] | +- org.apache.orc:orc-core:jar:1.6.11:provided
> [INFO] | | +- org.apache.orc:orc-shims:jar:1.6.11:provided
> [INFO] | | +- io.airlift:aircompressor:jar:0.21:provided
> [INFO] | | +- org.jetbrains:annotations:jar:17.0.0:provided
> [INFO] | | \- org.threeten:threeten-extra:jar:1.5.0:provided
> [INFO] | +- org.apache.orc:orc-mapreduce:jar:1.6.11:provided
> [INFO] | +- org.apache.hive:hive-storage-api:jar:2.7.2:provided
> [INFO] | +- org.apache.parquet:parquet-column:jar:1.12.1:provided
> [INFO] | | +- org.apache.parquet:parquet-common:jar:1.12.1:provided
> [INFO] | | | \-
> org.apache.yetus:audience-annotations:jar:0.12.0:provided
> [INFO] | | \- org.apache.parquet:parquet-encoding:jar:1.12.1:provided
> [INFO] | \- org.apache.parquet:parquet-hadoop:jar:1.12.1:provided
> [INFO] | +-
> org.apache.parquet:parquet-format-structures:jar:1.12.1:provided
> [INFO] | \- org.apache.parquet:parquet-jackson:jar:1.12.1:provided
> [INFO] +- org.apache.spark:spark-avro_2.12:jar:3.2.0:provided
> [INFO] +- org.junit.jupiter:junit-jupiter:jar:5.5.2:test
> [INFO] | +- org.junit.jupiter:junit-jupiter-api:jar:5.5.2:test
> [INFO] | | +- org.apiguardian:apiguardian-api:jar:1.1.0:test
> [INFO] | | +- org.opentest4j:opentest4j:jar:1.2.0:test
> [INFO] | | \- org.junit.platform:junit-platform-commons:jar:1.5.2:test
> [INFO] | +- org.junit.jupiter:junit-jupiter-params:jar:5.5.2:test
> [INFO] | \- org.junit.jupiter:junit-jupiter-engine:jar:5.5.2:test
> [INFO] | \- org.junit.platform:junit-platform-engine:jar:1.5.2:test
> [INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.12.6:compile
> [INFO] +- com.fasterxml.jackson.core:jackson-core:jar:2.12.6:compile
> [INFO] +- com.thoughtworks.paranamer:paranamer:jar:2.8:compile
> [INFO] +- com.fasterxml.jackson.core:jackson-annotations:jar:2.12.6:compile
> [INFO] +- org.mockito:mockito-junit-jupiter:jar:2.23.0:test
> [INFO] | \- org.mockito:mockito-core:jar:2.23.0:test
> [INFO] | +- net.bytebuddy:byte-buddy:jar:1.9.0:test
> [INFO] | +- net.bytebuddy:byte-buddy-agent:jar:1.9.0:test
> [INFO] | \- org.objenesis:objenesis:jar:2.6:provided
> [INFO] +- org.hamcrest:hamcrest-library:jar:1.3:test
> [INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test
> [INFO] +- com.crealytics:spark-excel_2.12:jar:3.2.0_0.16.0:compile
> [INFO] | +- com.norbitltd:spoiwo_2.12:jar:2.0.0:compile
> [INFO] | +- com.github.pjfanning:excel-streaming-reader:jar:3.2.4:compile
> [INFO] | \- com.github.pjfanning:poi-shared-strings:jar:2.3.0:compile
> [INFO] | \- com.h2database:h2:jar:2.0.202:runtime
> [INFO] +- org.apache.commons:commons-compress:jar:1.21:compile
> [INFO] +- commons-io:commons-io:jar:2.11.0:compile
> [INFO] \- org.apache.poi:poi-ooxml:jar:5.2.0:compile
> [INFO] +- org.apache.poi:poi-ooxml-lite:jar:5.2.0:compile
> [INFO] +- org.apache.xmlbeans:xmlbeans:jar:5.0.3:compile
> [INFO] +- com.github.virtuald:curvesapi:jar:1.06:compile
> [INFO] \- org.apache.logging.log4j:log4j-api:jar:2.17.1:compile
>
>
> Le jeu. 27 janv. 2022 à 00:39, Stephen Coy <sc...@infomedia.com.au> a
> écrit :
>
>> Hi Aurélien!
>>
>> Please run
>>
>> mvn dependency:tree
>>
>> and check it for Jackson dependencies.
>>
>> Feel free to respond with the output if you have any questions about it.
>>
>> Cheers,
>>
>> Steve C
>>
>> > On 22 Jan 2022, at 10:49 am, Aurélien Mazoyer <au...@aepsilon.com>
>> wrote:
>> >
>> > Hello,
>> >
>> > I migrated my code to Spark 3.2 and I am facing some issues. When I run
>> my unit tests via Maven, I get this error:
>> > java.lang.NoClassDefFoundError: Could not initialize class
>> org.apache.spark.rdd.RDDOperationScope$
>> > which is not super nice.
>> >
>> > However, when I run my test via Intellij, I get the following one:
>> > java.lang.ExceptionInInitializerError
>> > at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
>> > at org.apache.spark.rdd.RDD.map(RDD.scala:421)
>> > ...
>> > Caused by: com.fasterxml.jackson.databind.JsonMappingException: Scala
>> module 2.12.3 requires Jackson Databind version >= 2.12.0 and < 2.13.0
>> > which is far better imo since it gives me some clue on what is missing
>> in my pom.xml file to make it work. After putting a few more dependencies,
>> my tests are again passing.... in intellij, but I am stuck on the same
>> error when I am running maven command :-/.
>> > It seems that jdk and maven versions are the same and both are using
>> the same .m2 directory.
>> > Any clue on what can be going wrong?
>> >
>> > Thank you,
>> >
>> > Aurelien
>>
>> This email contains confidential information of and is the copyright of
>> Infomedia. It must not be forwarded, amended or disclosed without consent
>> of the sender. If you received this message by mistake, please advise the
>> sender and delete all copies. Security of transmission on the internet
>> cannot be guaranteed, could be infected, intercepted, or corrupted and you
>> should ensure you have suitable antivirus protection in place. By sending
>> us your or any third party personal details, you consent to (or confirm you
>> have obtained consent from such third parties) to Infomedia’s privacy
>> policy. http://www.infomedia.com.au/privacy-policy/
>>
>
>
Re: Migration to Spark 3.2
Posted by Stephen Coy <sc...@infomedia.com.au.INVALID>.
Hi Aurélien,
Your Jackson versions look fine.
What happens if you change the scope of your Jackson dependencies to “provided”?
This should result in your application using the versions provided by Spark and avoid this potential collision.
Cheers,
Steve C
On 27 Jan 2022, at 9:48 pm, Aurélien Mazoyer <au...@aepsilon.com>> wrote:
Hi Stephen,
Thank you for your answer!
Here it is, it seems that jackson dependencies are correct, no? :
Thanks,
[INFO] com.krrier:spark-lib-full:jar:0.0.1-SNAPSHOT
[INFO] +- com.krrier:backend:jar:0.0.1-SNAPSHOT:compile
[INFO] | \- com.krrier:data:jar:0.0.1-SNAPSHOT:compile
[INFO] +- com.krrier:plugin-api:jar:0.0.1-SNAPSHOT:compile
[INFO] +- com.opencsv:opencsv:jar:4.2:compile
[INFO] | +- org.apache.commons:commons-lang3:jar:3.9:compile
[INFO] | +- org.apache.commons:commons-text:jar:1.3:compile
[INFO] | +- commons-beanutils:commons-beanutils:jar:1.9.3:compile
[INFO] | | \- commons-logging:commons-logging:jar:1.2:compile
[INFO] | \- org.apache.commons:commons-collections4:jar:4.1:compile
[INFO] +- org.apache.solr:solr-solrj:jar:7.4.0:compile
[INFO] | +- org.apache.commons:commons-math3:jar:3.6.1:compile
[INFO] | +- org.apache.httpcomponents:httpclient:jar:4.5.3:compile
[INFO] | +- org.apache.httpcomponents:httpcore:jar:4.4.6:compile
[INFO] | +- org.apache.httpcomponents:httpmime:jar:4.5.3:compile
[INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.11:compile
[INFO] | +- org.codehaus.woodstox:stax2-api:jar:3.1.4:compile
[INFO] | +- org.codehaus.woodstox:woodstox-core-asl:jar:4.4.1:compile
[INFO] | \- org.noggit:noggit:jar:0.8:compile
[INFO] +- com.databricks:spark-xml_2.12:jar:0.5.0:compile
[INFO] +- org.apache.tika:tika-parsers:jar:1.24:compile
[INFO] | +- org.apache.tika:tika-core:jar:1.24:compile
[INFO] | +- org.glassfish.jaxb:jaxb-runtime:jar:2.3.2:compile
[INFO] | | +- jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.2:compile
[INFO] | | +- org.glassfish.jaxb:txw2:jar:2.3.2:compile
[INFO] | | +- com.sun.istack:istack-commons-runtime:jar:3.0.8:compile
[INFO] | | +- org.jvnet.staxex:stax-ex:jar:1.8.1:compile
[INFO] | | \- com.sun.xml.fastinfoset:FastInfoset:jar:1.2.16:compile
[INFO] | +- com.sun.activation:jakarta.activation:jar:1.2.1:compile
[INFO] | +- xerces:xercesImpl:jar:2.12.0:compile
[INFO] | | \- xml-apis:xml-apis:jar:1.4.01:compile
[INFO] | +- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] | +- org.gagravarr:vorbis-java-tika:jar:0.8:compile
[INFO] | +- org.tallison:jmatio:jar:1.5:compile
[INFO] | +- org.apache.james:apache-mime4j-core:jar:0.8.3:compile
[INFO] | +- org.apache.james:apache-mime4j-dom:jar:0.8.3:compile
[INFO] | +- org.tukaani:xz:jar:1.8:compile
[INFO] | +- com.epam:parso:jar:2.0.11:compile
[INFO] | +- org.brotli:dec:jar:0.1.2:compile
[INFO] | +- commons-codec:commons-codec:jar:1.13:compile
[INFO] | +- org.apache.pdfbox:pdfbox:jar:2.0.19:compile
[INFO] | | \- org.apache.pdfbox:fontbox:jar:2.0.19:compile
[INFO] | +- org.apache.pdfbox:pdfbox-tools:jar:2.0.19:compile
[INFO] | +- org.apache.pdfbox:preflight:jar:2.0.19:compile
[INFO] | | \- org.apache.pdfbox:xmpbox:jar:2.0.19:compile
[INFO] | +- org.apache.pdfbox:jempbox:jar:1.8.16:compile
[INFO] | +- org.bouncycastle:bcmail-jdk15on:jar:1.64:compile
[INFO] | | \- org.bouncycastle:bcpkix-jdk15on:jar:1.64:compile
[INFO] | +- org.bouncycastle:bcprov-jdk15on:jar:1.64:compile
[INFO] | +- org.apache.poi:poi:jar:4.1.2:compile
[INFO] | | \- com.zaxxer:SparseBitSet:jar:1.2:compile
[INFO] | +- org.apache.poi:poi-scratchpad:jar:4.1.2:compile
[INFO] | +- com.healthmarketscience.jackcess:jackcess:jar:3.0.1:compile
[INFO] | +- com.healthmarketscience.jackcess:jackcess-encrypt:jar:3.0.0:compile
[INFO] | +- org.ccil.cowan.tagsoup:tagsoup:jar:1.2.1:compile
[INFO] | +- org.ow2.asm:asm:jar:7.3.1:compile
[INFO] | +- com.googlecode.mp4parser:isoparser:jar:1.1.22:compile
[INFO] | +- org.tallison:metadata-extractor:jar:2.13.0:compile
[INFO] | | \- org.tallison.xmp:xmpcore-shaded:jar:6.1.10:compile
[INFO] | | \- com.adobe.xmp:xmpcore:jar:6.1.10:compile
[INFO] | +- de.l3s.boilerpipe:boilerpipe:jar:1.1.0:compile
[INFO] | +- com.rometools:rome:jar:1.12.2:compile
[INFO] | | \- com.rometools:rome-utils:jar:1.12.2:compile
[INFO] | +- org.gagravarr:vorbis-java-core:jar:0.8:compile
[INFO] | +- com.googlecode.juniversalchardet:juniversalchardet:jar:1.0.3:compile
[INFO] | +- org.codelibs:jhighlight:jar:1.0.3:compile
[INFO] | +- com.pff:java-libpst:jar:0.9.3:compile
[INFO] | +- com.github.junrar:junrar:jar:4.0.0:compile
[INFO] | +- org.apache.cxf:cxf-rt-rs-client:jar:3.3.5:compile
[INFO] | | +- org.apache.cxf:cxf-rt-transports-http:jar:3.3.5:compile
[INFO] | | +- org.apache.cxf:cxf-core:jar:3.3.5:compile
[INFO] | | | +- com.fasterxml.woodstox:woodstox-core:jar:5.0.3:compile
[INFO] | | | \- org.apache.ws.xmlschema:xmlschema-core:jar:2.2.5:compile
[INFO] | | \- org.apache.cxf:cxf-rt-frontend-jaxrs:jar:3.3.5:compile
[INFO] | | \- org.apache.cxf:cxf-rt-security:jar:3.3.5:compile
[INFO] | +- org.apache.commons:commons-exec:jar:1.3:compile
[INFO] | +- org.apache.opennlp:opennlp-tools:jar:1.9.2:compile
[INFO] | +- com.googlecode.json-simple:json-simple:jar:1.1.1:compile
[INFO] | +- com.github.openjson:openjson:jar:1.0.11:compile
[INFO] | +- com.google.code.gson:gson:jar:2.8.6:compile
[INFO] | +- edu.ucar:netcdf4:jar:4.5.5:compile
[INFO] | | \- net.jcip:jcip-annotations:jar:1.0:compile
[INFO] | +- org.jdom:jdom2:jar:2.0.6:compile
[INFO] | +- com.google.guava:guava:jar:28.2-jre:compile
[INFO] | | +- com.google.guava:failureaccess:jar:1.0.1:compile
[INFO] | | +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] | | +- org.checkerframework:checker-qual:jar:2.10.0:compile
[INFO] | | +- com.google.errorprone:error_prone_annotations:jar:2.3.4:compile
[INFO] | | \- com.google.j2objc:j2objc-annotations:jar:1.3:compile
[INFO] | +- edu.ucar:grib:jar:4.5.5:compile
[INFO] | | \- org.itadaki:bzip2:jar:0.9.1:compile
[INFO] | +- com.beust:jcommander:jar:1.78:compile
[INFO] | +- net.java.dev.jna:jna:jar:5.5.0:compile
[INFO] | +- org.jsoup:jsoup:jar:1.13.1:compile
[INFO] | +- com.google.protobuf:protobuf-java:jar:3.11.4:compile
[INFO] | +- edu.ucar:cdm:jar:4.5.5:compile
[INFO] | | +- edu.ucar:udunits:jar:4.5.5:compile
[INFO] | | +- joda-time:joda-time:jar:2.7:compile
[INFO] | | \- net.sf.ehcache:ehcache-core:jar:2.6.2:compile
[INFO] | +- org.quartz-scheduler:quartz:jar:2.3.2:compile
[INFO] | | \- com.zaxxer:HikariCP-java7:jar:2.4.13:compile
[INFO] | +- com.mchange:c3p0:jar:0.9.5.5:compile
[INFO] | | \- com.mchange:mchange-commons-java:jar:0.2.19:compile
[INFO] | +- edu.ucar:httpservices:jar:4.5.5:compile
[INFO] | +- org.apache.commons:commons-csv:jar:1.8:compile
[INFO] | +- org.apache.sis.core:sis-utility:jar:1.0:compile
[INFO] | | \- javax.measure:unit-api:jar:1.0:compile
[INFO] | +- org.apache.sis.storage:sis-netcdf:jar:1.0:compile
[INFO] | | +- org.apache.sis.storage:sis-storage:jar:1.0:compile
[INFO] | | | \- org.apache.sis.core:sis-feature:jar:1.0:compile
[INFO] | | \- org.apache.sis.core:sis-referencing:jar:1.0:compile
[INFO] | +- org.apache.sis.core:sis-metadata:jar:1.0:compile
[INFO] | +- org.opengis:geoapi:jar:3.0.1:compile
[INFO] | +- edu.usc.ir<http://edu.usc.ir>:sentiment-analysis-parser:jar:0.1:compile
[INFO] | +- org.apache.pdfbox:jbig2-imageio:jar:3.0.3:compile
[INFO] | \- com.github.jai-imageio:jai-imageio-core:jar:1.4.0:compile
[INFO] +- commons-cli:commons-cli:jar:1.4:compile
[INFO] +- org.apache.spark:spark-core_2.12:jar:3.2.0:provided
[INFO] | +- org.apache.avro:avro:jar:1.8.2:provided
[INFO] | | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:provided
[INFO] | | \- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:provided
[INFO] | +- org.apache.avro:avro-mapred:jar:1.10.2:provided
[INFO] | | \- org.apache.avro:avro-ipc:jar:1.10.2:provided
[INFO] | +- com.twitter:chill_2.12:jar:0.10.0:provided
[INFO] | | \- com.esotericsoftware:kryo-shaded:jar:4.0.2:provided
[INFO] | | \- com.esotericsoftware:minlog:jar:1.3.0:provided
[INFO] | +- com.twitter:chill-java:jar:0.10.0:provided
[INFO] | +- org.apache.xbean:xbean-asm9-shaded:jar:4.20:provided
[INFO] | +- org.apache.hadoop:hadoop-client-api:jar:3.3.1:provided
[INFO] | +- org.apache.hadoop:hadoop-client-runtime:jar:3.3.1:provided
[INFO] | | \- org.apache.htrace:htrace-core4:jar:4.1.0-incubating:provided
[INFO] | +- org.apache.spark:spark-launcher_2.12:jar:3.2.0:provided
[INFO] | +- org.apache.spark:spark-kvstore_2.12:jar:3.2.0:provided
[INFO] | | \- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:provided
[INFO] | +- org.apache.spark:spark-network-common_2.12:jar:3.2.0:provided
[INFO] | | \- com.google.crypto.tink:tink:jar:1.6.0:provided
[INFO] | +- org.apache.spark:spark-network-shuffle_2.12:jar:3.2.0:provided
[INFO] | +- org.apache.spark:spark-unsafe_2.12:jar:3.2.0:provided
[INFO] | +- javax.activation:activation:jar:1.1.1:provided
[INFO] | +- org.apache.curator:curator-recipes:jar:2.13.0:provided
[INFO] | | \- org.apache.curator:curator-framework:jar:2.13.0:provided
[INFO] | | \- org.apache.curator:curator-client:jar:2.13.0:provided
[INFO] | +- jakarta.servlet:jakarta.servlet-api:jar:4.0.3:provided
[INFO] | +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] | +- com.google.code.findbugs:jsr305:jar:3.0.0:compile
[INFO] | +- org.slf4j:slf4j-api:jar:1.7.7:provided
[INFO] | +- org.slf4j:jul-to-slf4j:jar:1.7.30:provided
[INFO] | +- org.slf4j:jcl-over-slf4j:jar:1.7.30:provided
[INFO] | +- log4j:log4j:jar:1.2.17:provided
[INFO] | +- org.slf4j:slf4j-log4j12:jar:1.7.7:provided
[INFO] | +- com.ning:compress-lzf:jar:1.0.3:provided
[INFO] | +- org.xerial.snappy:snappy-java:jar:1.1.8.4:provided
[INFO] | +- org.lz4:lz4-java:jar:1.7.1:provided
[INFO] | +- com.github.luben:zstd-jni:jar:1.5.0-4:provided
[INFO] | +- org.roaringbitmap:RoaringBitmap:jar:0.9.0:provided
[INFO] | | \- org.roaringbitmap:shims:jar:0.9.0:provided
[INFO] | +- commons-net:commons-net:jar:3.1:provided
[INFO] | +- org.scala-lang.modules:scala-xml_2.12:jar:1.2.0:compile
[INFO] | +- org.scala-lang:scala-library:jar:2.12.15:compile
[INFO] | +- org.scala-lang:scala-reflect:jar:2.12.15:provided
[INFO] | +- org.json4s:json4s-jackson_2.12:jar:3.7.0-M11:provided
[INFO] | | \- org.json4s:json4s-core_2.12:jar:3.7.0-M11:provided
[INFO] | | +- org.json4s:json4s-ast_2.12:jar:3.7.0-M11:provided
[INFO] | | \- org.json4s:json4s-scalap_2.12:jar:3.7.0-M11:provided
[INFO] | +- org.glassfish.jersey.core:jersey-client:jar:2.34:provided
[INFO] | | +- jakarta.ws.rs<http://jakarta.ws.rs>:jakarta.ws.rs<http://jakarta.ws.rs>-api:jar:2.1.6:compile
[INFO] | | \- org.glassfish.hk2.external:jakarta.inject:jar:2.6.1:provided
[INFO] | +- org.glassfish.jersey.core:jersey-common:jar:2.34:provided
[INFO] | | +- jakarta.annotation:jakarta.annotation-api:jar:1.3.5:provided
[INFO] | | \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.3:provided
[INFO] | +- org.glassfish.jersey.core:jersey-server:jar:2.34:provided
[INFO] | | \- jakarta.validation:jakarta.validation-api:jar:2.0.2:provided
[INFO] | +- org.glassfish.jersey.containers:jersey-container-servlet:jar:2.34:provided
[INFO] | +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.34:provided
[INFO] | +- org.glassfish.jersey.inject:jersey-hk2:jar:2.34:provided
[INFO] | | +- org.glassfish.hk2:hk2-locator:jar:2.6.1:provided
[INFO] | | | +- org.glassfish.hk2.external:aopalliance-repackaged:jar:2.6.1:provided
[INFO] | | | +- org.glassfish.hk2:hk2-api:jar:2.6.1:provided
[INFO] | | | \- org.glassfish.hk2:hk2-utils:jar:2.6.1:provided
[INFO] | | \- org.javassist:javassist:jar:3.25.0-GA:provided
[INFO] | +- io.netty:netty-all:jar:4.1.68.Final:provided
[INFO] | +- com.clearspring.analytics:stream:jar:2.9.6:provided
[INFO] | +- io.dropwizard.metrics:metrics-core:jar:4.2.0:provided
[INFO] | +- io.dropwizard.metrics:metrics-jvm:jar:4.2.0:provided
[INFO] | +- io.dropwizard.metrics:metrics-json:jar:4.2.0:provided
[INFO] | +- io.dropwizard.metrics:metrics-graphite:jar:4.2.0:provided
[INFO] | +- io.dropwizard.metrics:metrics-jmx:jar:4.2.0:provided
[INFO] | +- com.fasterxml.jackson.module:jackson-module-scala_2.12:jar:2.12.3:provided
[INFO] | +- org.apache.ivy:ivy:jar:2.5.0:provided
[INFO] | +- oro:oro:jar:2.0.8:provided
[INFO] | +- net.razorvine:pyrolite:jar:4.30:provided
[INFO] | +- net.sf.py4j:py4j:jar:0.10.9.2:provided
[INFO] | +- org.apache.spark:spark-tags_2.12:jar:3.2.0:provided
[INFO] | +- org.apache.commons:commons-crypto:jar:1.1.0:provided
[INFO] | \- org.spark-project.spark:unused:jar:1.0.0:provided
[INFO] +- org.apache.spark:spark-sql_2.12:jar:3.2.0:provided
[INFO] | +- org.rocksdb:rocksdbjni:jar:6.20.3:provided
[INFO] | +- com.univocity:univocity-parsers:jar:2.9.1:provided
[INFO] | +- org.apache.spark:spark-sketch_2.12:jar:3.2.0:provided
[INFO] | +- org.apache.spark:spark-catalyst_2.12:jar:3.2.0:provided
[INFO] | | +- org.scala-lang.modules:scala-parser-combinators_2.12:jar:1.1.2:provided
[INFO] | | +- org.codehaus.janino:janino:jar:3.0.16:provided
[INFO] | | +- org.codehaus.janino:commons-compiler:jar:3.0.16:provided
[INFO] | | +- org.antlr:antlr4-runtime:jar:4.7.1:provided
[INFO] | | +- javax.xml.bind:jaxb-api:jar:2.2.11:provided
[INFO] | | +- org.apache.arrow:arrow-vector:jar:2.0.0:provided
[INFO] | | | +- org.apache.arrow:arrow-format:jar:2.0.0:provided
[INFO] | | | +- org.apache.arrow:arrow-memory-core:jar:2.0.0:provided
[INFO] | | | \- com.google.flatbuffers:flatbuffers-java:jar:1.9.0:provided
[INFO] | | \- org.apache.arrow:arrow-memory-netty:jar:2.0.0:provided
[INFO] | +- org.apache.orc:orc-core:jar:1.6.11:provided
[INFO] | | +- org.apache.orc:orc-shims:jar:1.6.11:provided
[INFO] | | +- io.airlift:aircompressor:jar:0.21:provided
[INFO] | | +- org.jetbrains:annotations:jar:17.0.0:provided
[INFO] | | \- org.threeten:threeten-extra:jar:1.5.0:provided
[INFO] | +- org.apache.orc:orc-mapreduce:jar:1.6.11:provided
[INFO] | +- org.apache.hive:hive-storage-api:jar:2.7.2:provided
[INFO] | +- org.apache.parquet:parquet-column:jar:1.12.1:provided
[INFO] | | +- org.apache.parquet:parquet-common:jar:1.12.1:provided
[INFO] | | | \- org.apache.yetus:audience-annotations:jar:0.12.0:provided
[INFO] | | \- org.apache.parquet:parquet-encoding:jar:1.12.1:provided
[INFO] | \- org.apache.parquet:parquet-hadoop:jar:1.12.1:provided
[INFO] | +- org.apache.parquet:parquet-format-structures:jar:1.12.1:provided
[INFO] | \- org.apache.parquet:parquet-jackson:jar:1.12.1:provided
[INFO] +- org.apache.spark:spark-avro_2.12:jar:3.2.0:provided
[INFO] +- org.junit.jupiter:junit-jupiter:jar:5.5.2:test
[INFO] | +- org.junit.jupiter:junit-jupiter-api:jar:5.5.2:test
[INFO] | | +- org.apiguardian:apiguardian-api:jar:1.1.0:test
[INFO] | | +- org.opentest4j:opentest4j:jar:1.2.0:test
[INFO] | | \- org.junit.platform:junit-platform-commons:jar:1.5.2:test
[INFO] | +- org.junit.jupiter:junit-jupiter-params:jar:5.5.2:test
[INFO] | \- org.junit.jupiter:junit-jupiter-engine:jar:5.5.2:test
[INFO] | \- org.junit.platform:junit-platform-engine:jar:1.5.2:test
[INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.12.6:compile
[INFO] +- com.fasterxml.jackson.core:jackson-core:jar:2.12.6:compile
[INFO] +- com.thoughtworks.paranamer:paranamer:jar:2.8:compile
[INFO] +- com.fasterxml.jackson.core:jackson-annotations:jar:2.12.6:compile
[INFO] +- org.mockito:mockito-junit-jupiter:jar:2.23.0:test
[INFO] | \- org.mockito:mockito-core:jar:2.23.0:test
[INFO] | +- net.bytebuddy:byte-buddy:jar:1.9.0:test
[INFO] | +- net.bytebuddy:byte-buddy-agent:jar:1.9.0:test
[INFO] | \- org.objenesis:objenesis:jar:2.6:provided
[INFO] +- org.hamcrest:hamcrest-library:jar:1.3:test
[INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] +- com.crealytics:spark-excel_2.12:jar:3.2.0_0.16.0:compile
[INFO] | +- com.norbitltd:spoiwo_2.12:jar:2.0.0:compile
[INFO] | +- com.github.pjfanning:excel-streaming-reader:jar:3.2.4:compile
[INFO] | \- com.github.pjfanning:poi-shared-strings:jar:2.3.0:compile
[INFO] | \- com.h2database:h2:jar:2.0.202:runtime
[INFO] +- org.apache.commons:commons-compress:jar:1.21:compile
[INFO] +- commons-io:commons-io:jar:2.11.0:compile
[INFO] \- org.apache.poi:poi-ooxml:jar:5.2.0:compile
[INFO] +- org.apache.poi:poi-ooxml-lite:jar:5.2.0:compile
[INFO] +- org.apache.xmlbeans:xmlbeans:jar:5.0.3:compile
[INFO] +- com.github.virtuald:curvesapi:jar:1.06:compile
[INFO] \- org.apache.logging.log4j:log4j-api:jar:2.17.1:compile
Le jeu. 27 janv. 2022 à 00:39, Stephen Coy <sc...@infomedia.com.au>> a écrit :
Hi Aurélien!
Please run
mvn dependency:tree
and check it for Jackson dependencies.
Feel free to respond with the output if you have any questions about it.
Cheers,
Steve C
> On 22 Jan 2022, at 10:49 am, Aurélien Mazoyer <au...@aepsilon.com>> wrote:
>
> Hello,
>
> I migrated my code to Spark 3.2 and I am facing some issues. When I run my unit tests via Maven, I get this error:
> java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.rdd.RDDOperationScope$
> which is not super nice.
>
> However, when I run my test via Intellij, I get the following one:
> java.lang.ExceptionInInitializerError
> at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
> at org.apache.spark.rdd.RDD.map(RDD.scala:421)
> ...
> Caused by: com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.12.3 requires Jackson Databind version >= 2.12.0 and < 2.13.0
> which is far better imo since it gives me some clue on what is missing in my pom.xml file to make it work. After putting a few more dependencies, my tests are again passing.... in intellij, but I am stuck on the same error when I am running maven command :-/.
> It seems that jdk and maven versions are the same and both are using the same .m2 directory.
> Any clue on what can be going wrong?
>
> Thank you,
>
> Aurelien
This email contains confidential information of and is the copyright of Infomedia. It must not be forwarded, amended or disclosed without consent of the sender. If you received this message by mistake, please advise the sender and delete all copies. Security of transmission on the internet cannot be guaranteed, could be infected, intercepted, or corrupted and you should ensure you have suitable antivirus protection in place. By sending us your or any third party personal details, you consent to (or confirm you have obtained consent from such third parties) to Infomedia’s privacy policy. http://www.infomedia.com.au/privacy-policy/
Re: Migration to Spark 3.2
Posted by Aurélien Mazoyer <au...@aepsilon.com>.
Hi Stephen,
Thank you for your answer!
Here it is, it seems that jackson dependencies are correct, no? :
Thanks,
[INFO] com.krrier:spark-lib-full:jar:0.0.1-SNAPSHOT
[INFO] +- com.krrier:backend:jar:0.0.1-SNAPSHOT:compile
[INFO] | \- com.krrier:data:jar:0.0.1-SNAPSHOT:compile
[INFO] +- com.krrier:plugin-api:jar:0.0.1-SNAPSHOT:compile
[INFO] +- com.opencsv:opencsv:jar:4.2:compile
[INFO] | +- org.apache.commons:commons-lang3:jar:3.9:compile
[INFO] | +- org.apache.commons:commons-text:jar:1.3:compile
[INFO] | +- commons-beanutils:commons-beanutils:jar:1.9.3:compile
[INFO] | | \- commons-logging:commons-logging:jar:1.2:compile
[INFO] | \- org.apache.commons:commons-collections4:jar:4.1:compile
[INFO] +- org.apache.solr:solr-solrj:jar:7.4.0:compile
[INFO] | +- org.apache.commons:commons-math3:jar:3.6.1:compile
[INFO] | +- org.apache.httpcomponents:httpclient:jar:4.5.3:compile
[INFO] | +- org.apache.httpcomponents:httpcore:jar:4.4.6:compile
[INFO] | +- org.apache.httpcomponents:httpmime:jar:4.5.3:compile
[INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.11:compile
[INFO] | +- org.codehaus.woodstox:stax2-api:jar:3.1.4:compile
[INFO] | +- org.codehaus.woodstox:woodstox-core-asl:jar:4.4.1:compile
[INFO] | \- org.noggit:noggit:jar:0.8:compile
[INFO] +- com.databricks:spark-xml_2.12:jar:0.5.0:compile
[INFO] +- org.apache.tika:tika-parsers:jar:1.24:compile
[INFO] | +- org.apache.tika:tika-core:jar:1.24:compile
[INFO] | +- org.glassfish.jaxb:jaxb-runtime:jar:2.3.2:compile
[INFO] | | +- jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.2:compile
[INFO] | | +- org.glassfish.jaxb:txw2:jar:2.3.2:compile
[INFO] | | +- com.sun.istack:istack-commons-runtime:jar:3.0.8:compile
[INFO] | | +- org.jvnet.staxex:stax-ex:jar:1.8.1:compile
[INFO] | | \- com.sun.xml.fastinfoset:FastInfoset:jar:1.2.16:compile
[INFO] | +- com.sun.activation:jakarta.activation:jar:1.2.1:compile
[INFO] | +- xerces:xercesImpl:jar:2.12.0:compile
[INFO] | | \- xml-apis:xml-apis:jar:1.4.01:compile
[INFO] | +- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] | +- org.gagravarr:vorbis-java-tika:jar:0.8:compile
[INFO] | +- org.tallison:jmatio:jar:1.5:compile
[INFO] | +- org.apache.james:apache-mime4j-core:jar:0.8.3:compile
[INFO] | +- org.apache.james:apache-mime4j-dom:jar:0.8.3:compile
[INFO] | +- org.tukaani:xz:jar:1.8:compile
[INFO] | +- com.epam:parso:jar:2.0.11:compile
[INFO] | +- org.brotli:dec:jar:0.1.2:compile
[INFO] | +- commons-codec:commons-codec:jar:1.13:compile
[INFO] | +- org.apache.pdfbox:pdfbox:jar:2.0.19:compile
[INFO] | | \- org.apache.pdfbox:fontbox:jar:2.0.19:compile
[INFO] | +- org.apache.pdfbox:pdfbox-tools:jar:2.0.19:compile
[INFO] | +- org.apache.pdfbox:preflight:jar:2.0.19:compile
[INFO] | | \- org.apache.pdfbox:xmpbox:jar:2.0.19:compile
[INFO] | +- org.apache.pdfbox:jempbox:jar:1.8.16:compile
[INFO] | +- org.bouncycastle:bcmail-jdk15on:jar:1.64:compile
[INFO] | | \- org.bouncycastle:bcpkix-jdk15on:jar:1.64:compile
[INFO] | +- org.bouncycastle:bcprov-jdk15on:jar:1.64:compile
[INFO] | +- org.apache.poi:poi:jar:4.1.2:compile
[INFO] | | \- com.zaxxer:SparseBitSet:jar:1.2:compile
[INFO] | +- org.apache.poi:poi-scratchpad:jar:4.1.2:compile
[INFO] | +- com.healthmarketscience.jackcess:jackcess:jar:3.0.1:compile
[INFO] | +-
com.healthmarketscience.jackcess:jackcess-encrypt:jar:3.0.0:compile
[INFO] | +- org.ccil.cowan.tagsoup:tagsoup:jar:1.2.1:compile
[INFO] | +- org.ow2.asm:asm:jar:7.3.1:compile
[INFO] | +- com.googlecode.mp4parser:isoparser:jar:1.1.22:compile
[INFO] | +- org.tallison:metadata-extractor:jar:2.13.0:compile
[INFO] | | \- org.tallison.xmp:xmpcore-shaded:jar:6.1.10:compile
[INFO] | | \- com.adobe.xmp:xmpcore:jar:6.1.10:compile
[INFO] | +- de.l3s.boilerpipe:boilerpipe:jar:1.1.0:compile
[INFO] | +- com.rometools:rome:jar:1.12.2:compile
[INFO] | | \- com.rometools:rome-utils:jar:1.12.2:compile
[INFO] | +- org.gagravarr:vorbis-java-core:jar:0.8:compile
[INFO] | +-
com.googlecode.juniversalchardet:juniversalchardet:jar:1.0.3:compile
[INFO] | +- org.codelibs:jhighlight:jar:1.0.3:compile
[INFO] | +- com.pff:java-libpst:jar:0.9.3:compile
[INFO] | +- com.github.junrar:junrar:jar:4.0.0:compile
[INFO] | +- org.apache.cxf:cxf-rt-rs-client:jar:3.3.5:compile
[INFO] | | +- org.apache.cxf:cxf-rt-transports-http:jar:3.3.5:compile
[INFO] | | +- org.apache.cxf:cxf-core:jar:3.3.5:compile
[INFO] | | | +- com.fasterxml.woodstox:woodstox-core:jar:5.0.3:compile
[INFO] | | | \- org.apache.ws.xmlschema:xmlschema-core:jar:2.2.5:compile
[INFO] | | \- org.apache.cxf:cxf-rt-frontend-jaxrs:jar:3.3.5:compile
[INFO] | | \- org.apache.cxf:cxf-rt-security:jar:3.3.5:compile
[INFO] | +- org.apache.commons:commons-exec:jar:1.3:compile
[INFO] | +- org.apache.opennlp:opennlp-tools:jar:1.9.2:compile
[INFO] | +- com.googlecode.json-simple:json-simple:jar:1.1.1:compile
[INFO] | +- com.github.openjson:openjson:jar:1.0.11:compile
[INFO] | +- com.google.code.gson:gson:jar:2.8.6:compile
[INFO] | +- edu.ucar:netcdf4:jar:4.5.5:compile
[INFO] | | \- net.jcip:jcip-annotations:jar:1.0:compile
[INFO] | +- org.jdom:jdom2:jar:2.0.6:compile
[INFO] | +- com.google.guava:guava:jar:28.2-jre:compile
[INFO] | | +- com.google.guava:failureaccess:jar:1.0.1:compile
[INFO] | | +-
com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] | | +- org.checkerframework:checker-qual:jar:2.10.0:compile
[INFO] | | +-
com.google.errorprone:error_prone_annotations:jar:2.3.4:compile
[INFO] | | \- com.google.j2objc:j2objc-annotations:jar:1.3:compile
[INFO] | +- edu.ucar:grib:jar:4.5.5:compile
[INFO] | | \- org.itadaki:bzip2:jar:0.9.1:compile
[INFO] | +- com.beust:jcommander:jar:1.78:compile
[INFO] | +- net.java.dev.jna:jna:jar:5.5.0:compile
[INFO] | +- org.jsoup:jsoup:jar:1.13.1:compile
[INFO] | +- com.google.protobuf:protobuf-java:jar:3.11.4:compile
[INFO] | +- edu.ucar:cdm:jar:4.5.5:compile
[INFO] | | +- edu.ucar:udunits:jar:4.5.5:compile
[INFO] | | +- joda-time:joda-time:jar:2.7:compile
[INFO] | | \- net.sf.ehcache:ehcache-core:jar:2.6.2:compile
[INFO] | +- org.quartz-scheduler:quartz:jar:2.3.2:compile
[INFO] | | \- com.zaxxer:HikariCP-java7:jar:2.4.13:compile
[INFO] | +- com.mchange:c3p0:jar:0.9.5.5:compile
[INFO] | | \- com.mchange:mchange-commons-java:jar:0.2.19:compile
[INFO] | +- edu.ucar:httpservices:jar:4.5.5:compile
[INFO] | +- org.apache.commons:commons-csv:jar:1.8:compile
[INFO] | +- org.apache.sis.core:sis-utility:jar:1.0:compile
[INFO] | | \- javax.measure:unit-api:jar:1.0:compile
[INFO] | +- org.apache.sis.storage:sis-netcdf:jar:1.0:compile
[INFO] | | +- org.apache.sis.storage:sis-storage:jar:1.0:compile
[INFO] | | | \- org.apache.sis.core:sis-feature:jar:1.0:compile
[INFO] | | \- org.apache.sis.core:sis-referencing:jar:1.0:compile
[INFO] | +- org.apache.sis.core:sis-metadata:jar:1.0:compile
[INFO] | +- org.opengis:geoapi:jar:3.0.1:compile
[INFO] | +- edu.usc.ir:sentiment-analysis-parser:jar:0.1:compile
[INFO] | +- org.apache.pdfbox:jbig2-imageio:jar:3.0.3:compile
[INFO] | \- com.github.jai-imageio:jai-imageio-core:jar:1.4.0:compile
[INFO] +- commons-cli:commons-cli:jar:1.4:compile
[INFO] +- org.apache.spark:spark-core_2.12:jar:3.2.0:provided
[INFO] | +- org.apache.avro:avro:jar:1.8.2:provided
[INFO] | | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:provided
[INFO] | | \- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:provided
[INFO] | +- org.apache.avro:avro-mapred:jar:1.10.2:provided
[INFO] | | \- org.apache.avro:avro-ipc:jar:1.10.2:provided
[INFO] | +- com.twitter:chill_2.12:jar:0.10.0:provided
[INFO] | | \- com.esotericsoftware:kryo-shaded:jar:4.0.2:provided
[INFO] | | \- com.esotericsoftware:minlog:jar:1.3.0:provided
[INFO] | +- com.twitter:chill-java:jar:0.10.0:provided
[INFO] | +- org.apache.xbean:xbean-asm9-shaded:jar:4.20:provided
[INFO] | +- org.apache.hadoop:hadoop-client-api:jar:3.3.1:provided
[INFO] | +- org.apache.hadoop:hadoop-client-runtime:jar:3.3.1:provided
[INFO] | | \- org.apache.htrace:htrace-core4:jar:4.1.0-incubating:provided
[INFO] | +- org.apache.spark:spark-launcher_2.12:jar:3.2.0:provided
[INFO] | +- org.apache.spark:spark-kvstore_2.12:jar:3.2.0:provided
[INFO] | | \- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:provided
[INFO] | +- org.apache.spark:spark-network-common_2.12:jar:3.2.0:provided
[INFO] | | \- com.google.crypto.tink:tink:jar:1.6.0:provided
[INFO] | +- org.apache.spark:spark-network-shuffle_2.12:jar:3.2.0:provided
[INFO] | +- org.apache.spark:spark-unsafe_2.12:jar:3.2.0:provided
[INFO] | +- javax.activation:activation:jar:1.1.1:provided
[INFO] | +- org.apache.curator:curator-recipes:jar:2.13.0:provided
[INFO] | | \- org.apache.curator:curator-framework:jar:2.13.0:provided
[INFO] | | \- org.apache.curator:curator-client:jar:2.13.0:provided
[INFO] | +- jakarta.servlet:jakarta.servlet-api:jar:4.0.3:provided
[INFO] | +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] | +- com.google.code.findbugs:jsr305:jar:3.0.0:compile
[INFO] | +- org.slf4j:slf4j-api:jar:1.7.7:provided
[INFO] | +- org.slf4j:jul-to-slf4j:jar:1.7.30:provided
[INFO] | +- org.slf4j:jcl-over-slf4j:jar:1.7.30:provided
[INFO] | +- log4j:log4j:jar:1.2.17:provided
[INFO] | +- org.slf4j:slf4j-log4j12:jar:1.7.7:provided
[INFO] | +- com.ning:compress-lzf:jar:1.0.3:provided
[INFO] | +- org.xerial.snappy:snappy-java:jar:1.1.8.4:provided
[INFO] | +- org.lz4:lz4-java:jar:1.7.1:provided
[INFO] | +- com.github.luben:zstd-jni:jar:1.5.0-4:provided
[INFO] | +- org.roaringbitmap:RoaringBitmap:jar:0.9.0:provided
[INFO] | | \- org.roaringbitmap:shims:jar:0.9.0:provided
[INFO] | +- commons-net:commons-net:jar:3.1:provided
[INFO] | +- org.scala-lang.modules:scala-xml_2.12:jar:1.2.0:compile
[INFO] | +- org.scala-lang:scala-library:jar:2.12.15:compile
[INFO] | +- org.scala-lang:scala-reflect:jar:2.12.15:provided
[INFO] | +- org.json4s:json4s-jackson_2.12:jar:3.7.0-M11:provided
[INFO] | | \- org.json4s:json4s-core_2.12:jar:3.7.0-M11:provided
[INFO] | | +- org.json4s:json4s-ast_2.12:jar:3.7.0-M11:provided
[INFO] | | \- org.json4s:json4s-scalap_2.12:jar:3.7.0-M11:provided
[INFO] | +- org.glassfish.jersey.core:jersey-client:jar:2.34:provided
[INFO] | | +- jakarta.ws.rs:jakarta.ws.rs-api:jar:2.1.6:compile
[INFO] | | \- org.glassfish.hk2.external:jakarta.inject:jar:2.6.1:provided
[INFO] | +- org.glassfish.jersey.core:jersey-common:jar:2.34:provided
[INFO] | | +- jakarta.annotation:jakarta.annotation-api:jar:1.3.5:provided
[INFO] | | \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.3:provided
[INFO] | +- org.glassfish.jersey.core:jersey-server:jar:2.34:provided
[INFO] | | \- jakarta.validation:jakarta.validation-api:jar:2.0.2:provided
[INFO] | +-
org.glassfish.jersey.containers:jersey-container-servlet:jar:2.34:provided
[INFO] | +-
org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.34:provided
[INFO] | +- org.glassfish.jersey.inject:jersey-hk2:jar:2.34:provided
[INFO] | | +- org.glassfish.hk2:hk2-locator:jar:2.6.1:provided
[INFO] | | | +-
org.glassfish.hk2.external:aopalliance-repackaged:jar:2.6.1:provided
[INFO] | | | +- org.glassfish.hk2:hk2-api:jar:2.6.1:provided
[INFO] | | | \- org.glassfish.hk2:hk2-utils:jar:2.6.1:provided
[INFO] | | \- org.javassist:javassist:jar:3.25.0-GA:provided
[INFO] | +- io.netty:netty-all:jar:4.1.68.Final:provided
[INFO] | +- com.clearspring.analytics:stream:jar:2.9.6:provided
[INFO] | +- io.dropwizard.metrics:metrics-core:jar:4.2.0:provided
[INFO] | +- io.dropwizard.metrics:metrics-jvm:jar:4.2.0:provided
[INFO] | +- io.dropwizard.metrics:metrics-json:jar:4.2.0:provided
[INFO] | +- io.dropwizard.metrics:metrics-graphite:jar:4.2.0:provided
[INFO] | +- io.dropwizard.metrics:metrics-jmx:jar:4.2.0:provided
[INFO] | +-
com.fasterxml.jackson.module:jackson-module-scala_2.12:jar:2.12.3:provided
[INFO] | +- org.apache.ivy:ivy:jar:2.5.0:provided
[INFO] | +- oro:oro:jar:2.0.8:provided
[INFO] | +- net.razorvine:pyrolite:jar:4.30:provided
[INFO] | +- net.sf.py4j:py4j:jar:0.10.9.2:provided
[INFO] | +- org.apache.spark:spark-tags_2.12:jar:3.2.0:provided
[INFO] | +- org.apache.commons:commons-crypto:jar:1.1.0:provided
[INFO] | \- org.spark-project.spark:unused:jar:1.0.0:provided
[INFO] +- org.apache.spark:spark-sql_2.12:jar:3.2.0:provided
[INFO] | +- org.rocksdb:rocksdbjni:jar:6.20.3:provided
[INFO] | +- com.univocity:univocity-parsers:jar:2.9.1:provided
[INFO] | +- org.apache.spark:spark-sketch_2.12:jar:3.2.0:provided
[INFO] | +- org.apache.spark:spark-catalyst_2.12:jar:3.2.0:provided
[INFO] | | +-
org.scala-lang.modules:scala-parser-combinators_2.12:jar:1.1.2:provided
[INFO] | | +- org.codehaus.janino:janino:jar:3.0.16:provided
[INFO] | | +- org.codehaus.janino:commons-compiler:jar:3.0.16:provided
[INFO] | | +- org.antlr:antlr4-runtime:jar:4.7.1:provided
[INFO] | | +- javax.xml.bind:jaxb-api:jar:2.2.11:provided
[INFO] | | +- org.apache.arrow:arrow-vector:jar:2.0.0:provided
[INFO] | | | +- org.apache.arrow:arrow-format:jar:2.0.0:provided
[INFO] | | | +- org.apache.arrow:arrow-memory-core:jar:2.0.0:provided
[INFO] | | | \-
com.google.flatbuffers:flatbuffers-java:jar:1.9.0:provided
[INFO] | | \- org.apache.arrow:arrow-memory-netty:jar:2.0.0:provided
[INFO] | +- org.apache.orc:orc-core:jar:1.6.11:provided
[INFO] | | +- org.apache.orc:orc-shims:jar:1.6.11:provided
[INFO] | | +- io.airlift:aircompressor:jar:0.21:provided
[INFO] | | +- org.jetbrains:annotations:jar:17.0.0:provided
[INFO] | | \- org.threeten:threeten-extra:jar:1.5.0:provided
[INFO] | +- org.apache.orc:orc-mapreduce:jar:1.6.11:provided
[INFO] | +- org.apache.hive:hive-storage-api:jar:2.7.2:provided
[INFO] | +- org.apache.parquet:parquet-column:jar:1.12.1:provided
[INFO] | | +- org.apache.parquet:parquet-common:jar:1.12.1:provided
[INFO] | | | \- org.apache.yetus:audience-annotations:jar:0.12.0:provided
[INFO] | | \- org.apache.parquet:parquet-encoding:jar:1.12.1:provided
[INFO] | \- org.apache.parquet:parquet-hadoop:jar:1.12.1:provided
[INFO] | +-
org.apache.parquet:parquet-format-structures:jar:1.12.1:provided
[INFO] | \- org.apache.parquet:parquet-jackson:jar:1.12.1:provided
[INFO] +- org.apache.spark:spark-avro_2.12:jar:3.2.0:provided
[INFO] +- org.junit.jupiter:junit-jupiter:jar:5.5.2:test
[INFO] | +- org.junit.jupiter:junit-jupiter-api:jar:5.5.2:test
[INFO] | | +- org.apiguardian:apiguardian-api:jar:1.1.0:test
[INFO] | | +- org.opentest4j:opentest4j:jar:1.2.0:test
[INFO] | | \- org.junit.platform:junit-platform-commons:jar:1.5.2:test
[INFO] | +- org.junit.jupiter:junit-jupiter-params:jar:5.5.2:test
[INFO] | \- org.junit.jupiter:junit-jupiter-engine:jar:5.5.2:test
[INFO] | \- org.junit.platform:junit-platform-engine:jar:1.5.2:test
[INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.12.6:compile
[INFO] +- com.fasterxml.jackson.core:jackson-core:jar:2.12.6:compile
[INFO] +- com.thoughtworks.paranamer:paranamer:jar:2.8:compile
[INFO] +- com.fasterxml.jackson.core:jackson-annotations:jar:2.12.6:compile
[INFO] +- org.mockito:mockito-junit-jupiter:jar:2.23.0:test
[INFO] | \- org.mockito:mockito-core:jar:2.23.0:test
[INFO] | +- net.bytebuddy:byte-buddy:jar:1.9.0:test
[INFO] | +- net.bytebuddy:byte-buddy-agent:jar:1.9.0:test
[INFO] | \- org.objenesis:objenesis:jar:2.6:provided
[INFO] +- org.hamcrest:hamcrest-library:jar:1.3:test
[INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] +- com.crealytics:spark-excel_2.12:jar:3.2.0_0.16.0:compile
[INFO] | +- com.norbitltd:spoiwo_2.12:jar:2.0.0:compile
[INFO] | +- com.github.pjfanning:excel-streaming-reader:jar:3.2.4:compile
[INFO] | \- com.github.pjfanning:poi-shared-strings:jar:2.3.0:compile
[INFO] | \- com.h2database:h2:jar:2.0.202:runtime
[INFO] +- org.apache.commons:commons-compress:jar:1.21:compile
[INFO] +- commons-io:commons-io:jar:2.11.0:compile
[INFO] \- org.apache.poi:poi-ooxml:jar:5.2.0:compile
[INFO] +- org.apache.poi:poi-ooxml-lite:jar:5.2.0:compile
[INFO] +- org.apache.xmlbeans:xmlbeans:jar:5.0.3:compile
[INFO] +- com.github.virtuald:curvesapi:jar:1.06:compile
[INFO] \- org.apache.logging.log4j:log4j-api:jar:2.17.1:compile
Le jeu. 27 janv. 2022 à 00:39, Stephen Coy <sc...@infomedia.com.au> a écrit :
> Hi Aurélien!
>
> Please run
>
> mvn dependency:tree
>
> and check it for Jackson dependencies.
>
> Feel free to respond with the output if you have any questions about it.
>
> Cheers,
>
> Steve C
>
> > On 22 Jan 2022, at 10:49 am, Aurélien Mazoyer <au...@aepsilon.com>
> wrote:
> >
> > Hello,
> >
> > I migrated my code to Spark 3.2 and I am facing some issues. When I run
> my unit tests via Maven, I get this error:
> > java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.spark.rdd.RDDOperationScope$
> > which is not super nice.
> >
> > However, when I run my test via Intellij, I get the following one:
> > java.lang.ExceptionInInitializerError
> > at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
> > at org.apache.spark.rdd.RDD.map(RDD.scala:421)
> > ...
> > Caused by: com.fasterxml.jackson.databind.JsonMappingException: Scala
> module 2.12.3 requires Jackson Databind version >= 2.12.0 and < 2.13.0
> > which is far better imo since it gives me some clue on what is missing
> in my pom.xml file to make it work. After putting a few more dependencies,
> my tests are again passing.... in intellij, but I am stuck on the same
> error when I am running maven command :-/.
> > It seems that jdk and maven versions are the same and both are using the
> same .m2 directory.
> > Any clue on what can be going wrong?
> >
> > Thank you,
> >
> > Aurelien
>
> This email contains confidential information of and is the copyright of
> Infomedia. It must not be forwarded, amended or disclosed without consent
> of the sender. If you received this message by mistake, please advise the
> sender and delete all copies. Security of transmission on the internet
> cannot be guaranteed, could be infected, intercepted, or corrupted and you
> should ensure you have suitable antivirus protection in place. By sending
> us your or any third party personal details, you consent to (or confirm you
> have obtained consent from such third parties) to Infomedia’s privacy
> policy. http://www.infomedia.com.au/privacy-policy/
>
Re: Migration to Spark 3.2
Posted by Stephen Coy <sc...@infomedia.com.au.INVALID>.
Hi Aurélien!
Please run
mvn dependency:tree
and check it for Jackson dependencies.
Feel free to respond with the output if you have any questions about it.
Cheers,
Steve C
> On 22 Jan 2022, at 10:49 am, Aurélien Mazoyer <au...@aepsilon.com> wrote:
>
> Hello,
>
> I migrated my code to Spark 3.2 and I am facing some issues. When I run my unit tests via Maven, I get this error:
> java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.rdd.RDDOperationScope$
> which is not super nice.
>
> However, when I run my test via Intellij, I get the following one:
> java.lang.ExceptionInInitializerError
> at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
> at org.apache.spark.rdd.RDD.map(RDD.scala:421)
> ...
> Caused by: com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.12.3 requires Jackson Databind version >= 2.12.0 and < 2.13.0
> which is far better imo since it gives me some clue on what is missing in my pom.xml file to make it work. After putting a few more dependencies, my tests are again passing.... in intellij, but I am stuck on the same error when I am running maven command :-/.
> It seems that jdk and maven versions are the same and both are using the same .m2 directory.
> Any clue on what can be going wrong?
>
> Thank you,
>
> Aurelien
This email contains confidential information of and is the copyright of Infomedia. It must not be forwarded, amended or disclosed without consent of the sender. If you received this message by mistake, please advise the sender and delete all copies. Security of transmission on the internet cannot be guaranteed, could be infected, intercepted, or corrupted and you should ensure you have suitable antivirus protection in place. By sending us your or any third party personal details, you consent to (or confirm you have obtained consent from such third parties) to Infomedia’s privacy policy. http://www.infomedia.com.au/privacy-policy/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org