You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by swiesman <sw...@mediamath.com> on 2016/09/27 14:14:23 UTC
Flink 1.2-SNAPSHOT fails to initialize keyed state backend
Hi all,
I am working on an analytics project and am developing against flink
1.2-SNAPSHOT. The pipeline that I have built works; ie I can ingest data,
perform operations, and output the expected result. I can also see
checkpoints being written to RocksDB using Amazon S3 as the state backend.
Whenever the application runs for more than approximately 5 - 10 minutes I
get an RuntimeException: Could not initialize keyed state backed caused by a
class not found exception, full stack trace posted below. This confuses me
because for the program to have run and output data for the first 5 - 10
minutes as it does that class was certainly loaded successfully in the past.
As well, I am submitting a single fat jar to flink that I am certain
contains the class in question.
Thank you in advance for any assistance
Seth Wiesman
java.lang.RuntimeException: Could not initialize keyed state backend.
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.open(AbstractStreamOperator.java:148)
at
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:91)
at
org.apache.flink.streaming.api.operators.StreamFlatMap.open(StreamFlatMap.java:41)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:407)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:283)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:590)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException:
com.mediamath.reporting.lib.streaming.UpsertTrigger
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:628)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
at java.io.ObjectInputStream.readClass(ObjectInputStream.java:1486)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1336)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:503)
at
org.apache.flink.api.common.state.StateDescriptor.readObject(StateDescriptor.java:311)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1909)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
at
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBRestoreOperation.restoreKVStateMetaData(RocksDBKeyedStateBackend.java:653)
at
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBRestoreOperation.restoreKeyGroupsInStateHandle(RocksDBKeyedStateBackend.java:626)
at
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBRestoreOperation.doRestore(RocksDBKeyedStateBackend.java:609)
at
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.<init>(RocksDBKeyedStateBackend.java:207)
at
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.restoreKeyedStateBackend(RocksDBStateBackend.java:269)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:778)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.open(AbstractStreamOperator.java:142)
... 6 more
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-1-2-SNAPSHOT-fails-to-initialize-keyed-state-backend-tp9208.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Flink 1.2-SNAPSHOT fails to initialize keyed state backend
Posted by Stephan Ewen <se...@apache.org>.
Hi!
This was a temporary regression on the snapshot that has been fixed a few
days ago. It should be in the snapshot repositories by now.
Can you check if the problem persists if you force an update of your
snapshots dependencies?
Greetings,
Stephan
On Tue, Sep 27, 2016 at 5:04 PM, Timo Walther <tw...@apache.org> wrote:
> Hi Seth,
>
> the 1.2-SNAPSHOT is very fragile at the moment because of multiple big
> changes for dynamic scaling.
> Maybe Stefan (in CC) has an idea what is happening with the keyed state
> backend here?
>
> Timo
>
>
> Am 27/09/16 um 16:14 schrieb swiesman:
>
> Hi all,
>>
>> I am working on an analytics project and am developing against flink
>> 1.2-SNAPSHOT. The pipeline that I have built works; ie I can ingest data,
>> perform operations, and output the expected result. I can also see
>> checkpoints being written to RocksDB using Amazon S3 as the state backend.
>> Whenever the application runs for more than approximately 5 - 10 minutes I
>> get an RuntimeException: Could not initialize keyed state backed caused
>> by a
>> class not found exception, full stack trace posted below. This confuses me
>> because for the program to have run and output data for the first 5 - 10
>> minutes as it does that class was certainly loaded successfully in the
>> past.
>> As well, I am submitting a single fat jar to flink that I am certain
>> contains the class in question.
>>
>> Thank you in advance for any assistance
>>
>> Seth Wiesman
>>
>> java.lang.RuntimeException: Could not initialize keyed state backend.
>> at
>> org.apache.flink.streaming.api.operators.AbstractStreamOpera
>> tor.open(AbstractStreamOperator.java:148)
>> at
>> org.apache.flink.streaming.api.operators.AbstractUdfStreamOp
>> erator.open(AbstractUdfStreamOperator.java:91)
>> at
>> org.apache.flink.streaming.api.operators.StreamFlatMap.open(
>> StreamFlatMap.java:41)
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.openAllO
>> perators(StreamTask.java:407)
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(
>> StreamTask.java:283)
>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:590)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.lang.ClassNotFoundException:
>> com.mediamath.reporting.lib.streaming.UpsertTrigger
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:348)
>> at java.io.ObjectInputStream.resolveClass(ObjectInputStream.
>> java:628)
>> at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream
>> .java:1620)
>> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.
>> java:1521)
>> at java.io.ObjectInputStream.readClass(ObjectInputStream.java:
>> 1486)
>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java
>> :1336)
>> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea
>> m.java:2018)
>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.
>> java:1942)
>> at
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java
>> :1353)
>> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea
>> m.java:2018)
>> at java.io.ObjectInputStream.defaultReadObject(ObjectInputStrea
>> m.java:503)
>> at
>> org.apache.flink.api.common.state.StateDescriptor.readObject
>> (StateDescriptor.java:311)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>> ssorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>> thodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass
>> .java:1058)
>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.
>> java:1909)
>> at
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java
>> :1353)
>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:
>> 373)
>> at
>> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBa
>> ckend$RocksDBRestoreOperation.restoreKVStateMetaData(RocksDB
>> KeyedStateBackend.java:653)
>> at
>> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBa
>> ckend$RocksDBRestoreOperation.restoreKeyGroupsInStateHandle(
>> RocksDBKeyedStateBackend.java:626)
>> at
>> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBa
>> ckend$RocksDBRestoreOperation.doRestore(RocksDBKeyedStateBac
>> kend.java:609)
>> at
>> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBa
>> ckend.<init>(RocksDBKeyedStateBackend.java:207)
>> at
>> org.apache.flink.contrib.streaming.state.RocksDBStateBackend
>> .restoreKeyedStateBackend(RocksDBStateBackend.java:269)
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.createKe
>> yedStateBackend(StreamTask.java:778)
>> at
>> org.apache.flink.streaming.api.operators.AbstractStreamOpera
>> tor.open(AbstractStreamOperator.java:142)
>> ... 6 more
>>
>>
>>
>>
>> --
>> View this message in context: http://apache-flink-user-maili
>> ng-list-archive.2336050.n4.nabble.com/Flink-1-2-SNAPSHOT-
>> fails-to-initialize-keyed-state-backend-tp9208.html
>> Sent from the Apache Flink User Mailing List archive. mailing list
>> archive at Nabble.com.
>>
>
>
> --
> Freundliche Grüße / Kind Regards
>
> Timo Walther
>
> Follow me: @twalthr
> https://www.linkedin.com/in/twalthr
>
>
Re: Flink 1.2-SNAPSHOT fails to initialize keyed state backend
Posted by Timo Walther <tw...@apache.org>.
Hi Seth,
the 1.2-SNAPSHOT is very fragile at the moment because of multiple big
changes for dynamic scaling.
Maybe Stefan (in CC) has an idea what is happening with the keyed state
backend here?
Timo
Am 27/09/16 um 16:14 schrieb swiesman:
> Hi all,
>
> I am working on an analytics project and am developing against flink
> 1.2-SNAPSHOT. The pipeline that I have built works; ie I can ingest data,
> perform operations, and output the expected result. I can also see
> checkpoints being written to RocksDB using Amazon S3 as the state backend.
> Whenever the application runs for more than approximately 5 - 10 minutes I
> get an RuntimeException: Could not initialize keyed state backed caused by a
> class not found exception, full stack trace posted below. This confuses me
> because for the program to have run and output data for the first 5 - 10
> minutes as it does that class was certainly loaded successfully in the past.
> As well, I am submitting a single fat jar to flink that I am certain
> contains the class in question.
>
> Thank you in advance for any assistance
>
> Seth Wiesman
>
> java.lang.RuntimeException: Could not initialize keyed state backend.
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.open(AbstractStreamOperator.java:148)
> at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:91)
> at
> org.apache.flink.streaming.api.operators.StreamFlatMap.open(StreamFlatMap.java:41)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:407)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:283)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:590)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassNotFoundException:
> com.mediamath.reporting.lib.streaming.UpsertTrigger
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:628)
> at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
> at java.io.ObjectInputStream.readClass(ObjectInputStream.java:1486)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1336)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:503)
> at
> org.apache.flink.api.common.state.StateDescriptor.readObject(StateDescriptor.java:311)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1909)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBRestoreOperation.restoreKVStateMetaData(RocksDBKeyedStateBackend.java:653)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBRestoreOperation.restoreKeyGroupsInStateHandle(RocksDBKeyedStateBackend.java:626)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBRestoreOperation.doRestore(RocksDBKeyedStateBackend.java:609)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.<init>(RocksDBKeyedStateBackend.java:207)
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.restoreKeyedStateBackend(RocksDBStateBackend.java:269)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:778)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.open(AbstractStreamOperator.java:142)
> ... 6 more
>
>
>
>
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-1-2-SNAPSHOT-fails-to-initialize-keyed-state-backend-tp9208.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
--
Freundliche Gre / Kind Regards
Timo Walther
Follow me: @twalthr
https://www.linkedin.com/in/twalthr