You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Prez Cannady <re...@correlatesystems.com> on 2016/03/03 20:02:11 UTC
Attempting to refactor Kafka Example using Flink 0.10.2 with Scala 2.11
I’ve forked and am now experimenting with Robert Metzler’s kaka-example.
https://github.com/OCExercise/kafka-example <https://github.com/OCExercise/kafka-example>
Work fine from the vanilla fork (on the master branch). I performed my changes on branch enerscore-2.11, which includes:
1. Going from Flink 0.9 to Flink 0.10.2.
2. Using flink-connector-kafka-0.9_2.10 1.1-SNAPSHOT (which brings in Kafka 0.9).
3. Removed the shade plugin.
4. Using Scala 2.11
5. Instrumenting with spring-boot.
Currently running into this problem while attempting to invoke `mvn clean spring-boot:run`:
```
[INFO] --- spring-boot-maven-plugin:1.3.1.RELEASE:run (default-cli) @ kafka-example ---
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/revprez/.m2/repository/ch/qos/logback/logback-classic/1.1.3/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/revprez/.m2/repository/org/slf4j/slf4j-log4j12/1.7.13/slf4j-log4j12-1.7.13.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
[WARNING]
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.boot.maven.AbstractRunMojo$LaunchRunner.run(AbstractRunMojo.java:467)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/runtime/state/CheckpointListener
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at com.dataartisans.ReadFromKafka.main(ReadFromKafka.java:53)
... 6 more
Caused by: java.lang.ClassNotFoundException: org.apache.flink.runtime.state.CheckpointListener
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 29 more
```
My guess is that this occurs in the try block of the constructor `FlinkKafkaConsumer09(List<String> topics, KeyedDeserializationSchema<T> deserializer, Properties props)`, and is probably due to me missing something in the classpath. Any hints on how to proceed would be appreciated.
Prez Cannady
p: 617 500 3378
e: revprez@opencorrelate.org <ma...@opencorrelate.org>
GH: https://github.com/opencorrelate <https://github.com/opencorrelate>
LI: https://www.linkedin.com/in/revprez <https://www.linkedin.com/in/revprez>
Re: Attempting to refactor Kafka Example using Flink 0.10.2 with Scala 2.11
Posted by Prez Cannady <re...@correlatesystems.com>.
I’ll give that a shot, but I should also report that as of yesterday I’ve been able to get it to work with this configuration:
https://github.com/OCExercise/kafka-example/tree/example-2.11-0.9.0.1 <https://github.com/OCExercise/kafka-example/tree/example-2.11-0.9.0.1>
My primary blocking issue seems to have been attempting to publish and subscribe to an older (~0.8.x.x) Kafka instance.
Prez Cannady
p: 617 500 3378
e: revprez@opencorrelate.org <ma...@opencorrelate.org>
GH: https://github.com/opencorrelate <https://github.com/opencorrelate>
LI: https://www.linkedin.com/in/revprez <https://www.linkedin.com/in/revprez>
> On Mar 9, 2016, at 6:31 AM, Stephan Ewen <se...@apache.org> wrote:
>
> Hi!
>
> Can you use version 1.0.0 for everything? That should make all dependencies consistent.
>
> Greetings,
> Stephan
>
>
> On Wed, Mar 9, 2016 at 11:39 AM, Maximilian Michels <mxm@apache.org <ma...@apache.org>> wrote:
> Hi Prez,
>
> It appears Spring's Classloader is not set up correctly.
> Unfortunately, I'm not familiar with the way Springboot works.
>
> You added flink-connector-kafka-0.9_2.10 but also mentioned you're
> using Scala 2.11. That is bound to cause troubles :)
>
> Cheers,
> Max
>
> On Thu, Mar 3, 2016 at 8:02 PM, Prez Cannady
> <revprez@correlatesystems.com <ma...@correlatesystems.com>> wrote:
> > I’ve forked and am now experimenting with Robert Metzler’s kaka-example.
> >
> > https://github.com/OCExercise/kafka-example <https://github.com/OCExercise/kafka-example>
> >
> > Work fine from the vanilla fork (on the master branch). I performed my
> > changes on branch enerscore-2.11, which includes:
> >
> > 1. Going from Flink 0.9 to Flink 0.10.2.
> > 2. Using flink-connector-kafka-0.9_2.10 1.1-SNAPSHOT (which brings in Kafka
> > 0.9).
> > 3. Removed the shade plugin.
> > 4. Using Scala 2.11
> > 5. Instrumenting with spring-boot.
> >
> > Currently running into this problem while attempting to invoke `mvn clean
> > spring-boot:run`:
> >
> > ```
> > [INFO] --- spring-boot-maven-plugin:1.3.1.RELEASE:run (default-cli) @
> > kafka-example ---
> > SLF4J: Class path contains multiple SLF4J bindings.
> > SLF4J: Found binding in
> > [jar:file:/Users/revprez/.m2/repository/ch/qos/logback/logback-classic/1.1.3/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in
> > [jar:file:/Users/revprez/.m2/repository/org/slf4j/slf4j-log4j12/1.7.13/slf4j-log4j12-1.7.13.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings <http://www.slf4j.org/codes.html#multiple_bindings> for an
> > explanation.
> > SLF4J: Actual binding is of type
> > [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
> > [WARNING]
> > java.lang.reflect.InvocationTargetException
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:497)
> > at
> > org.springframework.boot.maven.AbstractRunMojo$LaunchRunner.run(AbstractRunMojo.java:467)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.lang.NoClassDefFoundError:
> > org/apache/flink/runtime/state/CheckpointListener
> > at java.lang.ClassLoader.defineClass1(Native Method)
> > at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> > at
> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> > at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > at java.lang.ClassLoader.defineClass1(Native Method)
> > at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> > at
> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> > at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > at com.dataartisans.ReadFromKafka.main(ReadFromKafka.java:53)
> > ... 6 more
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.flink.runtime.state.CheckpointListener
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > ... 29 more
> > ```
> >
> > My guess is that this occurs in the try block of the constructor
> > `FlinkKafkaConsumer09(List<String> topics, KeyedDeserializationSchema<T>
> > deserializer, Properties props)`, and is probably due to me missing
> > something in the classpath. Any hints on how to proceed would be
> > appreciated.
> >
> >
> > Prez Cannady
> > p: 617 500 3378
> > e: revprez@opencorrelate.org <ma...@opencorrelate.org>
> > GH: https://github.com/opencorrelate <https://github.com/opencorrelate>
> > LI: https://www.linkedin.com/in/revprez <https://www.linkedin.com/in/revprez>
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
Re: Attempting to refactor Kafka Example using Flink 0.10.2 with
Scala 2.11
Posted by Stephan Ewen <se...@apache.org>.
Hi!
Can you use version 1.0.0 for everything? That should make all dependencies
consistent.
Greetings,
Stephan
On Wed, Mar 9, 2016 at 11:39 AM, Maximilian Michels <mx...@apache.org> wrote:
> Hi Prez,
>
> It appears Spring's Classloader is not set up correctly.
> Unfortunately, I'm not familiar with the way Springboot works.
>
> You added flink-connector-kafka-0.9_2.10 but also mentioned you're
> using Scala 2.11. That is bound to cause troubles :)
>
> Cheers,
> Max
>
> On Thu, Mar 3, 2016 at 8:02 PM, Prez Cannady
> <re...@correlatesystems.com> wrote:
> > I’ve forked and am now experimenting with Robert Metzler’s kaka-example.
> >
> > https://github.com/OCExercise/kafka-example
> >
> > Work fine from the vanilla fork (on the master branch). I performed my
> > changes on branch enerscore-2.11, which includes:
> >
> > 1. Going from Flink 0.9 to Flink 0.10.2.
> > 2. Using flink-connector-kafka-0.9_2.10 1.1-SNAPSHOT (which brings in
> Kafka
> > 0.9).
> > 3. Removed the shade plugin.
> > 4. Using Scala 2.11
> > 5. Instrumenting with spring-boot.
> >
> > Currently running into this problem while attempting to invoke `mvn clean
> > spring-boot:run`:
> >
> > ```
> > [INFO] --- spring-boot-maven-plugin:1.3.1.RELEASE:run (default-cli) @
> > kafka-example ---
> > SLF4J: Class path contains multiple SLF4J bindings.
> > SLF4J: Found binding in
> >
> [jar:file:/Users/revprez/.m2/repository/ch/qos/logback/logback-classic/1.1.3/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in
> >
> [jar:file:/Users/revprez/.m2/repository/org/slf4j/slf4j-log4j12/1.7.13/slf4j-log4j12-1.7.13.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> > explanation.
> > SLF4J: Actual binding is of type
> > [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
> > [WARNING]
> > java.lang.reflect.InvocationTargetException
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:497)
> > at
> >
> org.springframework.boot.maven.AbstractRunMojo$LaunchRunner.run(AbstractRunMojo.java:467)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.lang.NoClassDefFoundError:
> > org/apache/flink/runtime/state/CheckpointListener
> > at java.lang.ClassLoader.defineClass1(Native Method)
> > at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> > at
> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> > at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > at java.lang.ClassLoader.defineClass1(Native Method)
> > at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> > at
> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> > at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > at com.dataartisans.ReadFromKafka.main(ReadFromKafka.java:53)
> > ... 6 more
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.flink.runtime.state.CheckpointListener
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > ... 29 more
> > ```
> >
> > My guess is that this occurs in the try block of the constructor
> > `FlinkKafkaConsumer09(List<String> topics, KeyedDeserializationSchema<T>
> > deserializer, Properties props)`, and is probably due to me missing
> > something in the classpath. Any hints on how to proceed would be
> > appreciated.
> >
> >
> > Prez Cannady
> > p: 617 500 3378
> > e: revprez@opencorrelate.org
> > GH: https://github.com/opencorrelate
> > LI: https://www.linkedin.com/in/revprez
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
Re: Attempting to refactor Kafka Example using Flink 0.10.2 with
Scala 2.11
Posted by Maximilian Michels <mx...@apache.org>.
Hi Prez,
It appears Spring's Classloader is not set up correctly.
Unfortunately, I'm not familiar with the way Springboot works.
You added flink-connector-kafka-0.9_2.10 but also mentioned you're
using Scala 2.11. That is bound to cause troubles :)
Cheers,
Max
On Thu, Mar 3, 2016 at 8:02 PM, Prez Cannady
<re...@correlatesystems.com> wrote:
> I’ve forked and am now experimenting with Robert Metzler’s kaka-example.
>
> https://github.com/OCExercise/kafka-example
>
> Work fine from the vanilla fork (on the master branch). I performed my
> changes on branch enerscore-2.11, which includes:
>
> 1. Going from Flink 0.9 to Flink 0.10.2.
> 2. Using flink-connector-kafka-0.9_2.10 1.1-SNAPSHOT (which brings in Kafka
> 0.9).
> 3. Removed the shade plugin.
> 4. Using Scala 2.11
> 5. Instrumenting with spring-boot.
>
> Currently running into this problem while attempting to invoke `mvn clean
> spring-boot:run`:
>
> ```
> [INFO] --- spring-boot-maven-plugin:1.3.1.RELEASE:run (default-cli) @
> kafka-example ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/Users/revprez/.m2/repository/ch/qos/logback/logback-classic/1.1.3/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/Users/revprez/.m2/repository/org/slf4j/slf4j-log4j12/1.7.13/slf4j-log4j12-1.7.13.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
> [WARNING]
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.springframework.boot.maven.AbstractRunMojo$LaunchRunner.run(AbstractRunMojo.java:467)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/flink/runtime/state/CheckpointListener
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at com.dataartisans.ReadFromKafka.main(ReadFromKafka.java:53)
> ... 6 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.runtime.state.CheckpointListener
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 29 more
> ```
>
> My guess is that this occurs in the try block of the constructor
> `FlinkKafkaConsumer09(List<String> topics, KeyedDeserializationSchema<T>
> deserializer, Properties props)`, and is probably due to me missing
> something in the classpath. Any hints on how to proceed would be
> appreciated.
>
>
> Prez Cannady
> p: 617 500 3378
> e: revprez@opencorrelate.org
> GH: https://github.com/opencorrelate
> LI: https://www.linkedin.com/in/revprez
>
>
>
>
>
>
>
>
>