You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by asdf zxcv <be...@gmail.com> on 2016/01/26 18:34:57 UTC
NoSuchMethod from transitive dependency jackson-databind in MaxMind GeoIP2
Hi all,
I'm trying to run MaxMind GeoIP2 in a Spark task, but get a runtime error
at init due to a NoSuchMethodError for ArrayNode from
com.fasterxml.jackson.core:jackson-databind.
This succeeds locally in unit tests, but fails in Spark tasks.
I've excluded jackson-databind from all other dependencies, including
Spark, since MaxMind is using the latest version of jackson-databind
(2.7.0).
`mvn dependency:tree` shows no conflicts for jackson-databind. There are
numerous conflicts with other jackson utilities from org.codehaus, but this
shouldn't be a problem since it is in a different namespace, right?
I think this may be a problem with the shaded jar produced by
maven-shade-plugin 2.4. Here is the config from pom.xml:
pastebin.com/QzrhM5Ee
`jar tvf shaded.jar` yields an entry for the missing class,
com/fasterxml/jackson/databind/node/ArrayNode.class.
META-INF/maven/com.fasterxml.jackson.core/jackson-databind/pom.xml has the
correct version, 2.7.0. The shaded jar expanded jackson-databind into its
constituent *.class files instead of providing the jar itself; is this
normal?
My other thought is there is some issue with the classpath. Is it
sufficient that the shaded jar contains the jackson-databind *.class files?
How else can I investigate this? I know that GeoIP databse init succeeds
locally with the jackson-databind jar in ~/.m2/
I've seen numerous issues with jackson-databind due to Spark using an old
version and not shading this particular dependency, but this seems to have
been resolved. I'm currently using version 1.3.1 and get the same result
bumping the version to 1.6.0
More detail here:
https://stackoverflow.com/questions/34958520/nosuchmethodexception-in-maxmind-geoip-dependency-jackson-databind-built-with-mv
Thanks,
Ben
Re: NoSuchMethod from transitive dependency jackson-databind in
MaxMind GeoIP2
Posted by asdf zxcv <be...@gmail.com>.
Thanks Ted, I think I was able to resolve this issue without modifying
spark by setting a relocation for jackson in my shaded jar.
On Tue, Jan 26, 2016 at 12:11 PM, Ted Yu <yu...@gmail.com> wrote:
> Then maybe changing the following in pom.xml to 2.7.0 and rebuild Spark ?
>
> <fasterxml.jackson.version>2.5.3</fasterxml.jackson.version>
>
> On Tue, Jan 26, 2016 at 11:53 AM, asdf zxcv <be...@gmail.com>
> wrote:
>
>> Hmm, this did not seem to resolve the issue. I also tried adding a
>> relocation for jackson as well.
>>
>> On Tue, Jan 26, 2016 at 10:09 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> I wonder if the following change would solve the problem you described
>>> (by shading jackson.core):
>>>
>>> diff --git a/pom.xml b/pom.xml
>>> index fb77506..32a3237 100644
>>> --- a/pom.xml
>>> +++ b/pom.xml
>>> @@ -2177,6 +2177,7 @@
>>> <include>org.eclipse.jetty:jetty-util</include>
>>> <include>org.eclipse.jetty:jetty-server</include>
>>> <include>com.google.guava:guava</include>
>>> + <include>com.fasterxml.jackson.core</include>
>>> </includes>
>>> </artifactSet>
>>> <relocations>
>>>
>>> Cheers
>>>
>>> On Tue, Jan 26, 2016 at 9:34 AM, asdf zxcv <be...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I'm trying to run MaxMind GeoIP2 in a Spark task, but get a runtime
>>>> error at init due to a NoSuchMethodError for ArrayNode from com.fasterxml.jackson.core:jackson-databind.
>>>> This succeeds locally in unit tests, but fails in Spark tasks.
>>>>
>>>> I've excluded jackson-databind from all other dependencies, including
>>>> Spark, since MaxMind is using the latest version of jackson-databind
>>>> (2.7.0).
>>>>
>>>> `mvn dependency:tree` shows no conflicts for jackson-databind. There
>>>> are numerous conflicts with other jackson utilities from org.codehaus, but
>>>> this shouldn't be a problem since it is in a different namespace, right?
>>>>
>>>> I think this may be a problem with the shaded jar produced by
>>>> maven-shade-plugin 2.4. Here is the config from pom.xml:
>>>> pastebin.com/QzrhM5Ee
>>>>
>>>> `jar tvf shaded.jar` yields an entry for the missing class, com/fasterxml/jackson/databind/node/ArrayNode.class.
>>>> META-INF/maven/com.fasterxml.jackson.core/jackson-databind/pom.xml has
>>>> the correct version, 2.7.0. The shaded jar expanded jackson-databind into
>>>> its constituent *.class files instead of providing the jar itself; is this
>>>> normal?
>>>>
>>>> My other thought is there is some issue with the classpath. Is it
>>>> sufficient that the shaded jar contains the jackson-databind *.class files?
>>>> How else can I investigate this? I know that GeoIP databse init succeeds
>>>> locally with the jackson-databind jar in ~/.m2/
>>>>
>>>> I've seen numerous issues with jackson-databind due to Spark using an
>>>> old version and not shading this particular dependency, but this seems to
>>>> have been resolved. I'm currently using version 1.3.1 and get the same
>>>> result bumping the version to 1.6.0
>>>>
>>>> More detail here:
>>>> https://stackoverflow.com/questions/34958520/nosuchmethodexception-in-maxmind-geoip-dependency-jackson-databind-built-with-mv
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Ben
>>>>
>>>
>>>
>>
>
Re: NoSuchMethod from transitive dependency jackson-databind in
MaxMind GeoIP2
Posted by Ted Yu <yu...@gmail.com>.
Then maybe changing the following in pom.xml to 2.7.0 and rebuild Spark ?
<fasterxml.jackson.version>2.5.3</fasterxml.jackson.version>
On Tue, Jan 26, 2016 at 11:53 AM, asdf zxcv <be...@gmail.com>
wrote:
> Hmm, this did not seem to resolve the issue. I also tried adding a
> relocation for jackson as well.
>
> On Tue, Jan 26, 2016 at 10:09 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> I wonder if the following change would solve the problem you described
>> (by shading jackson.core):
>>
>> diff --git a/pom.xml b/pom.xml
>> index fb77506..32a3237 100644
>> --- a/pom.xml
>> +++ b/pom.xml
>> @@ -2177,6 +2177,7 @@
>> <include>org.eclipse.jetty:jetty-util</include>
>> <include>org.eclipse.jetty:jetty-server</include>
>> <include>com.google.guava:guava</include>
>> + <include>com.fasterxml.jackson.core</include>
>> </includes>
>> </artifactSet>
>> <relocations>
>>
>> Cheers
>>
>> On Tue, Jan 26, 2016 at 9:34 AM, asdf zxcv <be...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> I'm trying to run MaxMind GeoIP2 in a Spark task, but get a runtime
>>> error at init due to a NoSuchMethodError for ArrayNode from com.fasterxml.jackson.core:jackson-databind.
>>> This succeeds locally in unit tests, but fails in Spark tasks.
>>>
>>> I've excluded jackson-databind from all other dependencies, including
>>> Spark, since MaxMind is using the latest version of jackson-databind
>>> (2.7.0).
>>>
>>> `mvn dependency:tree` shows no conflicts for jackson-databind. There are
>>> numerous conflicts with other jackson utilities from org.codehaus, but this
>>> shouldn't be a problem since it is in a different namespace, right?
>>>
>>> I think this may be a problem with the shaded jar produced by
>>> maven-shade-plugin 2.4. Here is the config from pom.xml:
>>> pastebin.com/QzrhM5Ee
>>>
>>> `jar tvf shaded.jar` yields an entry for the missing class, com/fasterxml/jackson/databind/node/ArrayNode.class.
>>> META-INF/maven/com.fasterxml.jackson.core/jackson-databind/pom.xml has
>>> the correct version, 2.7.0. The shaded jar expanded jackson-databind into
>>> its constituent *.class files instead of providing the jar itself; is this
>>> normal?
>>>
>>> My other thought is there is some issue with the classpath. Is it
>>> sufficient that the shaded jar contains the jackson-databind *.class files?
>>> How else can I investigate this? I know that GeoIP databse init succeeds
>>> locally with the jackson-databind jar in ~/.m2/
>>>
>>> I've seen numerous issues with jackson-databind due to Spark using an
>>> old version and not shading this particular dependency, but this seems to
>>> have been resolved. I'm currently using version 1.3.1 and get the same
>>> result bumping the version to 1.6.0
>>>
>>> More detail here:
>>> https://stackoverflow.com/questions/34958520/nosuchmethodexception-in-maxmind-geoip-dependency-jackson-databind-built-with-mv
>>>
>>>
>>> Thanks,
>>>
>>> Ben
>>>
>>
>>
>
Re: NoSuchMethod from transitive dependency jackson-databind in
MaxMind GeoIP2
Posted by asdf zxcv <be...@gmail.com>.
Hmm, this did not seem to resolve the issue. I also tried adding a
relocation for jackson as well.
On Tue, Jan 26, 2016 at 10:09 AM, Ted Yu <yu...@gmail.com> wrote:
> I wonder if the following change would solve the problem you described (by
> shading jackson.core):
>
> diff --git a/pom.xml b/pom.xml
> index fb77506..32a3237 100644
> --- a/pom.xml
> +++ b/pom.xml
> @@ -2177,6 +2177,7 @@
> <include>org.eclipse.jetty:jetty-util</include>
> <include>org.eclipse.jetty:jetty-server</include>
> <include>com.google.guava:guava</include>
> + <include>com.fasterxml.jackson.core</include>
> </includes>
> </artifactSet>
> <relocations>
>
> Cheers
>
> On Tue, Jan 26, 2016 at 9:34 AM, asdf zxcv <be...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> I'm trying to run MaxMind GeoIP2 in a Spark task, but get a runtime error
>> at init due to a NoSuchMethodError for ArrayNode from com.fasterxml.jackson.core:jackson-databind.
>> This succeeds locally in unit tests, but fails in Spark tasks.
>>
>> I've excluded jackson-databind from all other dependencies, including
>> Spark, since MaxMind is using the latest version of jackson-databind
>> (2.7.0).
>>
>> `mvn dependency:tree` shows no conflicts for jackson-databind. There are
>> numerous conflicts with other jackson utilities from org.codehaus, but this
>> shouldn't be a problem since it is in a different namespace, right?
>>
>> I think this may be a problem with the shaded jar produced by
>> maven-shade-plugin 2.4. Here is the config from pom.xml:
>> pastebin.com/QzrhM5Ee
>>
>> `jar tvf shaded.jar` yields an entry for the missing class, com/fasterxml/jackson/databind/node/ArrayNode.class.
>> META-INF/maven/com.fasterxml.jackson.core/jackson-databind/pom.xml has
>> the correct version, 2.7.0. The shaded jar expanded jackson-databind into
>> its constituent *.class files instead of providing the jar itself; is this
>> normal?
>>
>> My other thought is there is some issue with the classpath. Is it
>> sufficient that the shaded jar contains the jackson-databind *.class files?
>> How else can I investigate this? I know that GeoIP databse init succeeds
>> locally with the jackson-databind jar in ~/.m2/
>>
>> I've seen numerous issues with jackson-databind due to Spark using an old
>> version and not shading this particular dependency, but this seems to have
>> been resolved. I'm currently using version 1.3.1 and get the same result
>> bumping the version to 1.6.0
>>
>> More detail here:
>> https://stackoverflow.com/questions/34958520/nosuchmethodexception-in-maxmind-geoip-dependency-jackson-databind-built-with-mv
>>
>>
>> Thanks,
>>
>> Ben
>>
>
>
Re: NoSuchMethod from transitive dependency jackson-databind in
MaxMind GeoIP2
Posted by Ted Yu <yu...@gmail.com>.
I wonder if the following change would solve the problem you described (by
shading jackson.core):
diff --git a/pom.xml b/pom.xml
index fb77506..32a3237 100644
--- a/pom.xml
+++ b/pom.xml
@@ -2177,6 +2177,7 @@
<include>org.eclipse.jetty:jetty-util</include>
<include>org.eclipse.jetty:jetty-server</include>
<include>com.google.guava:guava</include>
+ <include>com.fasterxml.jackson.core</include>
</includes>
</artifactSet>
<relocations>
Cheers
On Tue, Jan 26, 2016 at 9:34 AM, asdf zxcv <be...@gmail.com> wrote:
> Hi all,
>
> I'm trying to run MaxMind GeoIP2 in a Spark task, but get a runtime error
> at init due to a NoSuchMethodError for ArrayNode from com.fasterxml.jackson.core:jackson-databind.
> This succeeds locally in unit tests, but fails in Spark tasks.
>
> I've excluded jackson-databind from all other dependencies, including
> Spark, since MaxMind is using the latest version of jackson-databind
> (2.7.0).
>
> `mvn dependency:tree` shows no conflicts for jackson-databind. There are
> numerous conflicts with other jackson utilities from org.codehaus, but this
> shouldn't be a problem since it is in a different namespace, right?
>
> I think this may be a problem with the shaded jar produced by
> maven-shade-plugin 2.4. Here is the config from pom.xml:
> pastebin.com/QzrhM5Ee
>
> `jar tvf shaded.jar` yields an entry for the missing class, com/fasterxml/jackson/databind/node/ArrayNode.class.
> META-INF/maven/com.fasterxml.jackson.core/jackson-databind/pom.xml has
> the correct version, 2.7.0. The shaded jar expanded jackson-databind into
> its constituent *.class files instead of providing the jar itself; is this
> normal?
>
> My other thought is there is some issue with the classpath. Is it
> sufficient that the shaded jar contains the jackson-databind *.class files?
> How else can I investigate this? I know that GeoIP databse init succeeds
> locally with the jackson-databind jar in ~/.m2/
>
> I've seen numerous issues with jackson-databind due to Spark using an old
> version and not shading this particular dependency, but this seems to have
> been resolved. I'm currently using version 1.3.1 and get the same result
> bumping the version to 1.6.0
>
> More detail here:
> https://stackoverflow.com/questions/34958520/nosuchmethodexception-in-maxmind-geoip-dependency-jackson-databind-built-with-mv
>
>
> Thanks,
>
> Ben
>