Jackson Spark/app incompatibility and how to resolve it

[Michael Allman <mi...@videoamp.com>]

Hello,

I'm running into an issue with a Spark app I'm building, which depends on a library which depends on Jackson 2.8, which fails at runtime because Spark brings in Jackson 2.6. I'm looking for a solution. As a workaround, I've patched our build of Spark to use Jackson 2.8. That's working, however given all the trouble associated with attempting a Jackson upgrade in the past (see https://issues.apache.org/jira/browse/SPARK-14989 <https://issues.apache.org/jira/browse/SPARK-14989?jql=project%20=%20SPARK%20AND%20text%20~%20jackson> and https://github.com/apache/spark/pull/13417 <https://github.com/apache/spark/pull/13417>), I'm wondering if I should submit a PR for that. Is shading Spark's Jackson deps another option? Any other suggestions for an acceptable way to fix this incompatibility with apps using a newer version of Jackson?

FWIW, Jackson claims to support backward compatibility within minor releases (https://github.com/FasterXML/jackson-docs#on-jackson-versioning <https://github.com/FasterXML/jackson-docs#on-jackson-versioning>). So in theory, apps that depend on an upgraded Spark version should work even if they ask for an older version.

Cheers,

Michael