You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Maximilian Michels <mx...@apache.org> on 2016/03/09 11:39:43 UTC
Re: Attempting to refactor Kafka Example using Flink 0.10.2 with
Scala 2.11
Hi Prez,
It appears Spring's Classloader is not set up correctly.
Unfortunately, I'm not familiar with the way Springboot works.
You added flink-connector-kafka-0.9_2.10 but also mentioned you're
using Scala 2.11. That is bound to cause troubles :)
Cheers,
Max
On Thu, Mar 3, 2016 at 8:02 PM, Prez Cannady
<re...@correlatesystems.com> wrote:
> I’ve forked and am now experimenting with Robert Metzler’s kaka-example.
>
> https://github.com/OCExercise/kafka-example
>
> Work fine from the vanilla fork (on the master branch). I performed my
> changes on branch enerscore-2.11, which includes:
>
> 1. Going from Flink 0.9 to Flink 0.10.2.
> 2. Using flink-connector-kafka-0.9_2.10 1.1-SNAPSHOT (which brings in Kafka
> 0.9).
> 3. Removed the shade plugin.
> 4. Using Scala 2.11
> 5. Instrumenting with spring-boot.
>
> Currently running into this problem while attempting to invoke `mvn clean
> spring-boot:run`:
>
> ```
> [INFO] --- spring-boot-maven-plugin:1.3.1.RELEASE:run (default-cli) @
> kafka-example ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/Users/revprez/.m2/repository/ch/qos/logback/logback-classic/1.1.3/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/Users/revprez/.m2/repository/org/slf4j/slf4j-log4j12/1.7.13/slf4j-log4j12-1.7.13.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
> [WARNING]
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.springframework.boot.maven.AbstractRunMojo$LaunchRunner.run(AbstractRunMojo.java:467)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/flink/runtime/state/CheckpointListener
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at com.dataartisans.ReadFromKafka.main(ReadFromKafka.java:53)
> ... 6 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.runtime.state.CheckpointListener
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 29 more
> ```
>
> My guess is that this occurs in the try block of the constructor
> `FlinkKafkaConsumer09(List<String> topics, KeyedDeserializationSchema<T>
> deserializer, Properties props)`, and is probably due to me missing
> something in the classpath. Any hints on how to proceed would be
> appreciated.
>
>
> Prez Cannady
> p: 617 500 3378
> e: revprez@opencorrelate.org
> GH: https://github.com/opencorrelate
> LI: https://www.linkedin.com/in/revprez
>
>
>
>
>
>
>
>
>
Re: Attempting to refactor Kafka Example using Flink 0.10.2 with Scala 2.11
Posted by Prez Cannady <re...@correlatesystems.com>.
I’ll give that a shot, but I should also report that as of yesterday I’ve been able to get it to work with this configuration:
https://github.com/OCExercise/kafka-example/tree/example-2.11-0.9.0.1 <https://github.com/OCExercise/kafka-example/tree/example-2.11-0.9.0.1>
My primary blocking issue seems to have been attempting to publish and subscribe to an older (~0.8.x.x) Kafka instance.
Prez Cannady
p: 617 500 3378
e: revprez@opencorrelate.org <ma...@opencorrelate.org>
GH: https://github.com/opencorrelate <https://github.com/opencorrelate>
LI: https://www.linkedin.com/in/revprez <https://www.linkedin.com/in/revprez>
> On Mar 9, 2016, at 6:31 AM, Stephan Ewen <se...@apache.org> wrote:
>
> Hi!
>
> Can you use version 1.0.0 for everything? That should make all dependencies consistent.
>
> Greetings,
> Stephan
>
>
> On Wed, Mar 9, 2016 at 11:39 AM, Maximilian Michels <mxm@apache.org <ma...@apache.org>> wrote:
> Hi Prez,
>
> It appears Spring's Classloader is not set up correctly.
> Unfortunately, I'm not familiar with the way Springboot works.
>
> You added flink-connector-kafka-0.9_2.10 but also mentioned you're
> using Scala 2.11. That is bound to cause troubles :)
>
> Cheers,
> Max
>
> On Thu, Mar 3, 2016 at 8:02 PM, Prez Cannady
> <revprez@correlatesystems.com <ma...@correlatesystems.com>> wrote:
> > I’ve forked and am now experimenting with Robert Metzler’s kaka-example.
> >
> > https://github.com/OCExercise/kafka-example <https://github.com/OCExercise/kafka-example>
> >
> > Work fine from the vanilla fork (on the master branch). I performed my
> > changes on branch enerscore-2.11, which includes:
> >
> > 1. Going from Flink 0.9 to Flink 0.10.2.
> > 2. Using flink-connector-kafka-0.9_2.10 1.1-SNAPSHOT (which brings in Kafka
> > 0.9).
> > 3. Removed the shade plugin.
> > 4. Using Scala 2.11
> > 5. Instrumenting with spring-boot.
> >
> > Currently running into this problem while attempting to invoke `mvn clean
> > spring-boot:run`:
> >
> > ```
> > [INFO] --- spring-boot-maven-plugin:1.3.1.RELEASE:run (default-cli) @
> > kafka-example ---
> > SLF4J: Class path contains multiple SLF4J bindings.
> > SLF4J: Found binding in
> > [jar:file:/Users/revprez/.m2/repository/ch/qos/logback/logback-classic/1.1.3/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in
> > [jar:file:/Users/revprez/.m2/repository/org/slf4j/slf4j-log4j12/1.7.13/slf4j-log4j12-1.7.13.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings <http://www.slf4j.org/codes.html#multiple_bindings> for an
> > explanation.
> > SLF4J: Actual binding is of type
> > [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
> > [WARNING]
> > java.lang.reflect.InvocationTargetException
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:497)
> > at
> > org.springframework.boot.maven.AbstractRunMojo$LaunchRunner.run(AbstractRunMojo.java:467)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.lang.NoClassDefFoundError:
> > org/apache/flink/runtime/state/CheckpointListener
> > at java.lang.ClassLoader.defineClass1(Native Method)
> > at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> > at
> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> > at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > at java.lang.ClassLoader.defineClass1(Native Method)
> > at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> > at
> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> > at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > at com.dataartisans.ReadFromKafka.main(ReadFromKafka.java:53)
> > ... 6 more
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.flink.runtime.state.CheckpointListener
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > ... 29 more
> > ```
> >
> > My guess is that this occurs in the try block of the constructor
> > `FlinkKafkaConsumer09(List<String> topics, KeyedDeserializationSchema<T>
> > deserializer, Properties props)`, and is probably due to me missing
> > something in the classpath. Any hints on how to proceed would be
> > appreciated.
> >
> >
> > Prez Cannady
> > p: 617 500 3378
> > e: revprez@opencorrelate.org <ma...@opencorrelate.org>
> > GH: https://github.com/opencorrelate <https://github.com/opencorrelate>
> > LI: https://www.linkedin.com/in/revprez <https://www.linkedin.com/in/revprez>
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
Re: Attempting to refactor Kafka Example using Flink 0.10.2 with
Scala 2.11
Posted by Stephan Ewen <se...@apache.org>.
Hi!
Can you use version 1.0.0 for everything? That should make all dependencies
consistent.
Greetings,
Stephan
On Wed, Mar 9, 2016 at 11:39 AM, Maximilian Michels <mx...@apache.org> wrote:
> Hi Prez,
>
> It appears Spring's Classloader is not set up correctly.
> Unfortunately, I'm not familiar with the way Springboot works.
>
> You added flink-connector-kafka-0.9_2.10 but also mentioned you're
> using Scala 2.11. That is bound to cause troubles :)
>
> Cheers,
> Max
>
> On Thu, Mar 3, 2016 at 8:02 PM, Prez Cannady
> <re...@correlatesystems.com> wrote:
> > I’ve forked and am now experimenting with Robert Metzler’s kaka-example.
> >
> > https://github.com/OCExercise/kafka-example
> >
> > Work fine from the vanilla fork (on the master branch). I performed my
> > changes on branch enerscore-2.11, which includes:
> >
> > 1. Going from Flink 0.9 to Flink 0.10.2.
> > 2. Using flink-connector-kafka-0.9_2.10 1.1-SNAPSHOT (which brings in
> Kafka
> > 0.9).
> > 3. Removed the shade plugin.
> > 4. Using Scala 2.11
> > 5. Instrumenting with spring-boot.
> >
> > Currently running into this problem while attempting to invoke `mvn clean
> > spring-boot:run`:
> >
> > ```
> > [INFO] --- spring-boot-maven-plugin:1.3.1.RELEASE:run (default-cli) @
> > kafka-example ---
> > SLF4J: Class path contains multiple SLF4J bindings.
> > SLF4J: Found binding in
> >
> [jar:file:/Users/revprez/.m2/repository/ch/qos/logback/logback-classic/1.1.3/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in
> >
> [jar:file:/Users/revprez/.m2/repository/org/slf4j/slf4j-log4j12/1.7.13/slf4j-log4j12-1.7.13.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> > explanation.
> > SLF4J: Actual binding is of type
> > [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
> > [WARNING]
> > java.lang.reflect.InvocationTargetException
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:497)
> > at
> >
> org.springframework.boot.maven.AbstractRunMojo$LaunchRunner.run(AbstractRunMojo.java:467)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.lang.NoClassDefFoundError:
> > org/apache/flink/runtime/state/CheckpointListener
> > at java.lang.ClassLoader.defineClass1(Native Method)
> > at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> > at
> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> > at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > at java.lang.ClassLoader.defineClass1(Native Method)
> > at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> > at
> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> > at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > at com.dataartisans.ReadFromKafka.main(ReadFromKafka.java:53)
> > ... 6 more
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.flink.runtime.state.CheckpointListener
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > ... 29 more
> > ```
> >
> > My guess is that this occurs in the try block of the constructor
> > `FlinkKafkaConsumer09(List<String> topics, KeyedDeserializationSchema<T>
> > deserializer, Properties props)`, and is probably due to me missing
> > something in the classpath. Any hints on how to proceed would be
> > appreciated.
> >
> >
> > Prez Cannady
> > p: 617 500 3378
> > e: revprez@opencorrelate.org
> > GH: https://github.com/opencorrelate
> > LI: https://www.linkedin.com/in/revprez
> >
> >
> >
> >
> >
> >
> >
> >
> >
>