You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Bart van Deenen <ba...@fastmail.fm> on 2016/03/29 13:17:42 UTC
jackson DefaultScalaModule missing on deployment
Hi all
I've succesfully built a Flink streaming job, and it runs beautifully in
my IntelliJ ide, with a Flink instance started on the fly. The job eats
Kafka events, and outputs to file. All the i/o is json encoded with
Jackson.
But I'm having trouble with deploying the jar on a Flink server
(version 1.0.0)
Building the jar via sbt package generates a jar with only my classes in
it, and the Flink cluster misses some Jackson stuff, and it complains
when I try to run the jar file.
java.lang.NoClassDefFoundError:
com/fasterxml/jackson/module/scala/DefaultScalaModule
The build.sbt is pretty much identical to the quick-start:
resolvers in ThisBuild ++= Seq(Resolver.mavenLocal)
name := "Flink Project"
version := "0.1-SNAPSHOT"
organization := "org.example"
scalaVersion in ThisBuild := "2.11.7"
val flinkVersion = "1.0.0"
val flinkDependencies = Seq(
"org.apache.flink" %% "flink-scala" % flinkVersion % "provided",
"org.apache.flink" %% "flink-streaming-scala" % flinkVersion %
"provided")
lazy val root = (project in file(".")).
settings(
libraryDependencies ++= flinkDependencies
)
libraryDependencies += "org.apache.flink" %
"flink-connector-kafka-0.9_2.11" % "1.0.0"
libraryDependencies += "com.fasterxml.jackson.module" %
"jackson-module-scala_2.11" % "2.4.5"
mainClass in assembly := Some("com.kpn.cj.journeyengine.Streaming")
// make run command include the provided dependencies
run in Compile <<= Defaults.runTask(fullClasspath in Compile, mainClass
in (Compile, run), runner in (Compile, run))
// exclude Scala library from assembly
assemblyOption in assembly := (assemblyOption in
assembly).value.copy(includeScala = false)
I'm not a hero as far as Scala Build Tool is concerned, any hints on how
to make sure the jackson stuff is actually included in the flink job?
Thanks
Bart
Re: jackson DefaultScalaModule missing on deployment
Posted by Bart van Deenen <ba...@fastmail.fm>.
Thanks
Copying jars into the lib directory works fine
On Tue, Mar 29, 2016, at 13:34, Balaji Rajagopalan wrote:
> You will have to include dependent jackson jar in flink server lib
> folder, or create a fat jar.
>
> balaji
>
> On Tue, Mar 29, 2016 at 4:47 PM, Bart van Deenen
> <ba...@fastmail.fm> wrote:
>> Hi all
>>
>> I've succesfully built a Flink streaming job, and it runs
>> beautifully in
>> my IntelliJ ide, with a Flink instance started on the fly. The
>> job eats
>> Kafka events, and outputs to file. All the i/o is json encoded with
>> Jackson.
>>
>> But I'm having trouble with deploying the jar on a Flink server
>> (version 1.0.0)
>> Building the jar via sbt package generates a jar with only my
>> classes in
>> it, and the Flink cluster misses some Jackson stuff, and it
>> complains
>> when I try to run the jar file.
>>
>> java.lang.NoClassDefFoundError:
>> com/fasterxml/jackson/module/scala/DefaultScalaModule
>>
>> The build.sbt is pretty much identical to the quick-start:
>>
>> resolvers in ThisBuild ++= Seq(Resolver.mavenLocal)
>> name := "Flink Project"
>> version := "0.1-SNAPSHOT"
>> organization := "org.example"
>> scalaVersion in ThisBuild := "2.11.7"
>> val flinkVersion = "1.0.0"
>>
>> val flinkDependencies = Seq(
>> "org.apache.flink" %% "flink-scala" % flinkVersion % "provided",
>> "org.apache.flink" %% "flink-streaming-scala" % flinkVersion %
>> "provided")
>> lazy val root = (project in file(".")).
>> settings(
>> libraryDependencies ++= flinkDependencies
>> )
>> libraryDependencies += "org.apache.flink" %
>> "flink-connector-kafka-0.9_2.11" % "1.0.0"
>> libraryDependencies += "com.fasterxml.jackson.module" %
>> "jackson-module-scala_2.11" % "2.4.5"
>> mainClass in assembly := Some("com.kpn.cj.journeyengine.Streaming")
>>
>> // make run command include the provided dependencies
>> run in Compile <<= Defaults.runTask(fullClasspath in Compile,
>> mainClass
>> in (Compile, run), runner in (Compile, run))
>>
>> // exclude Scala library from assembly
>> assemblyOption in assembly := (assemblyOption in
>> assembly).value.copy(includeScala = false)
>>
>> I'm not a hero as far as Scala Build Tool is concerned, any hints
>> on how
>> to make sure the jackson stuff is actually included in the
>> flink job?
>>
>> Thanks
>>
>> Bart
Re: jackson DefaultScalaModule missing on deployment
Posted by Balaji Rajagopalan <ba...@olacabs.com>.
You will have to include dependent jackson jar in flink server lib folder,
or create a fat jar.
balaji
On Tue, Mar 29, 2016 at 4:47 PM, Bart van Deenen <ba...@fastmail.fm>
wrote:
> Hi all
>
> I've succesfully built a Flink streaming job, and it runs beautifully in
> my IntelliJ ide, with a Flink instance started on the fly. The job eats
> Kafka events, and outputs to file. All the i/o is json encoded with
> Jackson.
>
> But I'm having trouble with deploying the jar on a Flink server
> (version 1.0.0)
> Building the jar via sbt package generates a jar with only my classes in
> it, and the Flink cluster misses some Jackson stuff, and it complains
> when I try to run the jar file.
>
> java.lang.NoClassDefFoundError:
> com/fasterxml/jackson/module/scala/DefaultScalaModule
>
> The build.sbt is pretty much identical to the quick-start:
>
> resolvers in ThisBuild ++= Seq(Resolver.mavenLocal)
> name := "Flink Project"
> version := "0.1-SNAPSHOT"
> organization := "org.example"
> scalaVersion in ThisBuild := "2.11.7"
> val flinkVersion = "1.0.0"
>
> val flinkDependencies = Seq(
> "org.apache.flink" %% "flink-scala" % flinkVersion % "provided",
> "org.apache.flink" %% "flink-streaming-scala" % flinkVersion %
> "provided")
> lazy val root = (project in file(".")).
> settings(
> libraryDependencies ++= flinkDependencies
> )
> libraryDependencies += "org.apache.flink" %
> "flink-connector-kafka-0.9_2.11" % "1.0.0"
> libraryDependencies += "com.fasterxml.jackson.module" %
> "jackson-module-scala_2.11" % "2.4.5"
> mainClass in assembly := Some("com.kpn.cj.journeyengine.Streaming")
>
> // make run command include the provided dependencies
> run in Compile <<= Defaults.runTask(fullClasspath in Compile, mainClass
> in (Compile, run), runner in (Compile, run))
>
> // exclude Scala library from assembly
> assemblyOption in assembly := (assemblyOption in
> assembly).value.copy(includeScala = false)
>
> I'm not a hero as far as Scala Build Tool is concerned, any hints on how
> to make sure the jackson stuff is actually included in the flink job?
>
> Thanks
>
> Bart
>