You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Natu Lauchande <nl...@gmail.com> on 2021/12/07 05:23:22 UTC
Re: Issue with Flink jobs after upgrading to Flink 1.13.1/Ververica 2.5 - java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
Hey Timo and Flink community,
I wonder if there is a fix for this issue. The last time I rollbacked to
version 12 of Flink and downgraded Ververica.
I am really keen to leverage the new features on the latest versions of
Ververica 2.5+ , i have tried a myriad of tricks suggested ( example :
building the image with hadoop-client libraries) :
java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getDeclaredConstructors(Class.java:2020)
at java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass.java:
1961)
at java.io.ObjectStreamClass.access$100(ObjectStreamClass.java:79)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:275)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:273)
at java.security.AccessController.doPrivileged(Native Method)
at java.io.ObjectStreamClass.getSerialVersionUID(ObjectStreamClass.java:
272)
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:694)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:
2003)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1850)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:
2160)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:
2405)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:
2187)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:
2405)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:
2187)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:
2405)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:
2187)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:
2405)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:
2187)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:
2405)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:
2187)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
at org.apache.flink.util.InstantiationUtil.deserializeObject(
InstantiationUtil.java:615)
at org.apache.flink.util.InstantiationUtil.deserializeObject(
InstantiationUtil.java:600)
at org.apache.flink.util.InstantiationUtil.deserializeObject(
InstantiationUtil.java:587)
at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(
InstantiationUtil.java:541)
at org.apache.flink.streaming.api.graph.StreamConfig
.getStreamOperatorFactory(StreamConfig.java:322)
at org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(
OperatorChain.java:159)
at org.apache.flink.streaming.runtime.tasks.StreamTask.executeRestore(
StreamTask.java:551)
at org.apache.flink.streaming.runtime.tasks.StreamTask
.runWithCleanUpOnFail(StreamTask.java:650)
at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(
StreamTask.java:540)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:759)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.
Configuration
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at org.apache.flink.util.FlinkUserCodeClassLoader
.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:64)
at org.apache.flink.util.ChildFirstClassLoader
.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:65)
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(
FlinkUserCodeClassLoader.java:48)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 48 more
This error occurs when writing to StreamFileWriting on S3 in Parquet
format.
Thanks,
Natu
On Thu, Jul 22, 2021 at 3:53 PM Timo Walther <tw...@apache.org> wrote:
> Thanks, this should definitely work with the pre-packaged connectors of
> Ververica platform.
>
> I guess we have to investigate what is going on. Until then, a
> workaround could be to add Hadoop manually and set the HADOOP_CLASSPATH
> environment variable. The root cause seems that Hadoop cannot be found.
>
> Alternatively, you could also build a custom image and include Hadoop in
> the lib folder of Flink:
>
> https://docs.ververica.com/v1.3/platform/installation/custom_images.html
>
> I hope this helps. I will get back to you if we have a fix ready.
>
> Regards,
> Timo
>
>
>
> On 22.07.21 14:30, Natu Lauchande wrote:
> > Sure.
> >
> > That's how the ddl table looks like:
> >
> > CREATETABLEtablea (
> >
> > `a` BIGINT,
> >
> > `b` BIGINT,
> >
> > `c` BIGINT
> >
> > )
> >
> > COMMENT ''
> >
> > WITH(
> >
> > 'auto-compaction'='false',
> >
> > 'connector'='filesystem',
> >
> > 'format'='parquet',
> >
> > 'parquet.block.size'='134217728',
> >
> > 'parquet.compression'='SNAPPY',
> >
> > 'parquet.dictionary.page.size'='1048576',
> >
> > 'parquet.enable.dictionary'='true',
> >
> > 'parquet.page.size'='1048576',
> >
> > 'parquet.writer.max-padding'='2097152',
> >
> > 'path'='s3a://test/test’,
> >
> > 'sink.partition-commit.delay'='1 h',
> >
> > 'sink.partition-commit.policy.kind'='success-file',
> >
> > 'sink.partition-commit.success-file.name
> > <http://sink.partition-commit.success-file.name>'='_SUCCESS',
> >
> > 'sink.partition-commit.trigger'='process-time',
> >
> > 'sink.rolling-policy.check-interval'='20 min',
> >
> > 'sink.rolling-policy.file-size'='128MB',
> >
> > 'sink.rolling-policy.rollover-interval'='2 h'
> >
> > );
> >
> >
> >
> > When a change the connector to a blackhole it immediately works without
> > errors. I have the redacted the names and paths.
> >
> >
> >
> > Thanks,
> > Natu
> >
> >
> > On Thu, Jul 22, 2021 at 2:24 PM Timo Walther <twalthr@apache.org
> > <ma...@apache.org>> wrote:
> >
> > Maybe you can share also which connector/format you are using? What
> is
> > the DDL?
> >
> > Regards,
> > Timo
> >
> >
> > On 22.07.21 14:11, Natu Lauchande wrote:
> > > Hey Timo,
> > >
> > > Thanks for the reply.
> > >
> > > No custom file as we are using Flink SQL and submitting the job
> > directly
> > > through the SQL Editor UI. We are using Flink 1.13.1 as the
> > supported
> > > flink version. No custom code all through Flink SQL on UI no jars.
> > >
> > > Thanks,
> > > Natu
> > >
> > > On Thu, Jul 22, 2021 at 2:08 PM Timo Walther <twalthr@apache.org
> > <ma...@apache.org>
> > > <mailto:twalthr@apache.org <ma...@apache.org>>> wrote:
> > >
> > > Hi Natu,
> > >
> > > Ververica Platform 2.5 has updated the bundled Hadoop version
> > but this
> > > should not result in a NoClassDefFoundError exception. How
> > are you
> > > submitting your SQL jobs? You don't use Ververica's SQL
> > service but
> > > have
> > > built a regular JAR file, right? If this is the case, can you
> > share
> > > your
> > > pom.xml file with us? The Flink version stays constant at
> 1.12?
> > >
> > > Regards,
> > > Timo
> > >
> > > On 22.07.21 12:22, Natu Lauchande wrote:
> > > > Good day Flink community,
> > > >
> > > > Apache Flink/Ververica Community Edition - Question
> > > >
> > > >
> > > > I am having an issue with my Flink SQL jobs since updating
> > > from Flink
> > > > 1.12/Ververica 2.4 to Ververica 2.5 . For all the jobs
> > running on
> > > > parquet and S3 i am getting the following error
> continuously:
> > > >
> > > > INITIALIZING to FAILED on 10.243.3.0:42337-2a3224 @
> > > > 10-243-3-0.flink-metrics.vvp-jobs.svc.cluster.local
> > (dataPort=39309).
> > > >
> > > > java.lang.NoClassDefFoundError:
> > org/apache/hadoop/conf/Configuration
> > > >
> > > > at java.lang.Class.getDeclaredConstructors0(Native Method)
> > > ~[?:1.8.0_292]
> > > >
> > > > at
> > java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
> > > > ~[?:1.8.0_292]
> > > >
> > > > at java.lang.Class.getDeclaredConstructors(Class.java:2020)
> > > ~[?:1.8.0_292]
> > > >
> > > > *....*
> > > >
> > > > at
> > java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
> > > > ~[?:1.8.0_292]
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:615)
> > >
> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:600)
> > >
> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:587)
> > >
> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:541)
> > >
> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperatorFactory(StreamConfig.java:322)
> > >
> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperator(OperatorChain.java:653)
> > >
> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperatorChain(OperatorChain.java:626)
> > >
> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:566)
> > >
> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperatorChain(OperatorChain.java:616)
> > >
> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:566)
> > >
> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperatorChain(OperatorChain.java:616)
> > >
> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:566)
> > >
> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperatorChain(OperatorChain.java:616)
> > >
> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:566)
> > >
> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(OperatorChain.java:181)
> > >
> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.flink.streaming.runtime.tasks.StreamTask.executeRestore(StreamTask.java:548)
> > >
> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:647)
> > >
> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:537)
> > >
> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
> > > >
> > > > at
> > org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:759)
> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
> > > >
> > > > at
> > org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
> > > >
> > > > at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_292]
> > > >
> > > > Caused by: java.lang.ClassNotFoundException:
> > > > org.apache.hadoop.conf.Configuration
> > > >
> > > > at
> java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> > > ~[?:1.8.0_292]
> > > >
> > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> > > ~[?:1.8.0_292]
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:64)
> > >
> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:65)
> > >
> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
> > >
> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
> > > >
> > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> > > ~[?:1.8.0_292]
> > > >
> > > > ... 57 more
> > > >
> > > > 2021-07-22 09:38:43,095 DEBUG
> > > > org.apache.flink.runtime.scheduler.SharedSlot[] - Remove
> > logical
> > > slot
> > > > (SlotRequestId{4297879e795d0516e36a7c26ccc795b2}) for
> > execution
> > > vertex
> > > > (id cbc357ccb763df2852fee8c4fc7d55f2_0) from the physical
> slot
> > > > (SlotRequestId{df7c49a6610b56f26aea214c05bcd9ed})
> > > >
> > > > 2021-07-22 09:38:43,096 DEBUG
> > > > org.apache.flink.runtime.scheduler.SharedSlot[] - Release
> > shared
> > > slot
> > > > externally
> (SlotRequestId{df7c49a6610b56f26aea214c05bcd9ed})
> > > >
> > > >
> > > > Everything works well when i roll back to Ververica v2.4,
> > has anyone
> > > > experienced this error before.
> > > >
> > > > Thanks,
> > > >
> > > > Natu
> > > >
> > >
> >
>
>
Re: Issue with Flink jobs after upgrading to Flink 1.13.1/Ververica 2.5 - java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
Posted by Natu Lauchande <nl...@gmail.com>.
Hey Ingo,
Thanks for the suggestion. It's definitely an issue with the Parquet
connector, when we try with the CSV or Blackhole connector it's all fine.
I will be trying this approach and report back.
Thanks,
Natu
On Wed, Dec 8, 2021 at 7:02 PM Ingo Bürk <in...@ververica.com> wrote:
> Hi Natu,
>
> Something you could try is removing the packaged parquet format and
> defining a custom format[1]. For this custom format you can then fix the
> dependencies by packaging all of the following into the format:
>
> * flink-sql-parquet
> * flink-shaded-hadoop-2-uber
> * hadoop-aws
> * aws-java-sdk-bundle
> * guava
>
> This isn't entirely straight-forward, unfortunately, and I haven't
> verified it. However, with Ververica Platform 2.6, to be released shortly
> after Flink 1.15, it should also work again.
>
> [1]
> https://docs.ververica.com/user_guide/sql_development/connectors.html#custom-connectors-and-formats
>
>
> Best
> Ingo
>
> On Tue, Dec 7, 2021 at 6:23 AM Natu Lauchande <nl...@gmail.com>
> wrote:
>
>> Hey Timo and Flink community,
>>
>> I wonder if there is a fix for this issue. The last time I rollbacked to
>> version 12 of Flink and downgraded Ververica.
>>
>> I am really keen to leverage the new features on the latest versions of
>> Ververica 2.5+ , i have tried a myriad of tricks suggested ( example :
>> building the image with hadoop-client libraries) :
>>
>> java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
>> at java.lang.Class.getDeclaredConstructors0(Native Method)
>> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
>> at java.lang.Class.getDeclaredConstructors(Class.java:2020)
>> at java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass
>> .java:1961)
>> at java.io.ObjectStreamClass.access$100(ObjectStreamClass.java:79)
>> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:275)
>> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:273)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.io.ObjectStreamClass.getSerialVersionUID(ObjectStreamClass
>> .java:272)
>> at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:694)
>> at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:
>> 2003)
>> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:
>> 1850)
>> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream
>> .java:2160)
>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream
>> .java:2405)
>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:
>> 2329)
>> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream
>> .java:2187)
>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream
>> .java:2405)
>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:
>> 2329)
>> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream
>> .java:2187)
>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream
>> .java:2405)
>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:
>> 2329)
>> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream
>> .java:2187)
>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream
>> .java:2405)
>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:
>> 2329)
>> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream
>> .java:2187)
>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream
>> .java:2405)
>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:
>> 2329)
>> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream
>> .java:2187)
>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)
>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
>> at org.apache.flink.util.InstantiationUtil.deserializeObject(
>> InstantiationUtil.java:615)
>> at org.apache.flink.util.InstantiationUtil.deserializeObject(
>> InstantiationUtil.java:600)
>> at org.apache.flink.util.InstantiationUtil.deserializeObject(
>> InstantiationUtil.java:587)
>> at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(
>> InstantiationUtil.java:541)
>> at org.apache.flink.streaming.api.graph.StreamConfig
>> .getStreamOperatorFactory(StreamConfig.java:322)
>> at org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(
>> OperatorChain.java:159)
>> at org.apache.flink.streaming.runtime.tasks.StreamTask
>> .executeRestore(StreamTask.java:551)
>> at org.apache.flink.streaming.runtime.tasks.StreamTask
>> .runWithCleanUpOnFail(StreamTask.java:650)
>> at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(
>> StreamTask.java:540)
>> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:759)
>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.
>> Configuration
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
>> at org.apache.flink.util.FlinkUserCodeClassLoader
>> .loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:64)
>> at org.apache.flink.util.ChildFirstClassLoader
>> .loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:65)
>> at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(
>> FlinkUserCodeClassLoader.java:48)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
>> ... 48 more
>>
>> This error occurs when writing to StreamFileWriting on S3 in Parquet
>> format.
>>
>>
>> Thanks,
>> Natu
>>
>> On Thu, Jul 22, 2021 at 3:53 PM Timo Walther <tw...@apache.org> wrote:
>>
>>> Thanks, this should definitely work with the pre-packaged connectors of
>>> Ververica platform.
>>>
>>> I guess we have to investigate what is going on. Until then, a
>>> workaround could be to add Hadoop manually and set the HADOOP_CLASSPATH
>>> environment variable. The root cause seems that Hadoop cannot be found.
>>>
>>> Alternatively, you could also build a custom image and include Hadoop in
>>> the lib folder of Flink:
>>>
>>> https://docs.ververica.com/v1.3/platform/installation/custom_images.html
>>>
>>> I hope this helps. I will get back to you if we have a fix ready.
>>>
>>> Regards,
>>> Timo
>>>
>>>
>>>
>>> On 22.07.21 14:30, Natu Lauchande wrote:
>>> > Sure.
>>> >
>>> > That's how the ddl table looks like:
>>> >
>>> > CREATETABLEtablea (
>>> >
>>> > `a` BIGINT,
>>> >
>>> > `b` BIGINT,
>>> >
>>> > `c` BIGINT
>>> >
>>> > )
>>> >
>>> > COMMENT ''
>>> >
>>> > WITH(
>>> >
>>> > 'auto-compaction'='false',
>>> >
>>> > 'connector'='filesystem',
>>> >
>>> > 'format'='parquet',
>>> >
>>> > 'parquet.block.size'='134217728',
>>> >
>>> > 'parquet.compression'='SNAPPY',
>>> >
>>> > 'parquet.dictionary.page.size'='1048576',
>>> >
>>> > 'parquet.enable.dictionary'='true',
>>> >
>>> > 'parquet.page.size'='1048576',
>>> >
>>> > 'parquet.writer.max-padding'='2097152',
>>> >
>>> > 'path'='s3a://test/test’,
>>> >
>>> > 'sink.partition-commit.delay'='1 h',
>>> >
>>> > 'sink.partition-commit.policy.kind'='success-file',
>>> >
>>> > 'sink.partition-commit.success-file.name
>>> > <http://sink.partition-commit.success-file.name>'='_SUCCESS',
>>> >
>>> > 'sink.partition-commit.trigger'='process-time',
>>> >
>>> > 'sink.rolling-policy.check-interval'='20 min',
>>> >
>>> > 'sink.rolling-policy.file-size'='128MB',
>>> >
>>> > 'sink.rolling-policy.rollover-interval'='2 h'
>>> >
>>> > );
>>> >
>>> >
>>> >
>>> > When a change the connector to a blackhole it immediately works
>>> without
>>> > errors. I have the redacted the names and paths.
>>> >
>>> >
>>> >
>>> > Thanks,
>>> > Natu
>>> >
>>> >
>>> > On Thu, Jul 22, 2021 at 2:24 PM Timo Walther <twalthr@apache.org
>>> > <ma...@apache.org>> wrote:
>>> >
>>> > Maybe you can share also which connector/format you are using?
>>> What is
>>> > the DDL?
>>> >
>>> > Regards,
>>> > Timo
>>> >
>>> >
>>> > On 22.07.21 14:11, Natu Lauchande wrote:
>>> > > Hey Timo,
>>> > >
>>> > > Thanks for the reply.
>>> > >
>>> > > No custom file as we are using Flink SQL and submitting the job
>>> > directly
>>> > > through the SQL Editor UI. We are using Flink 1.13.1 as the
>>> > supported
>>> > > flink version. No custom code all through Flink SQL on UI no
>>> jars.
>>> > >
>>> > > Thanks,
>>> > > Natu
>>> > >
>>> > > On Thu, Jul 22, 2021 at 2:08 PM Timo Walther <
>>> twalthr@apache.org
>>> > <ma...@apache.org>
>>> > > <mailto:twalthr@apache.org <ma...@apache.org>>> wrote:
>>> > >
>>> > > Hi Natu,
>>> > >
>>> > > Ververica Platform 2.5 has updated the bundled Hadoop
>>> version
>>> > but this
>>> > > should not result in a NoClassDefFoundError exception. How
>>> > are you
>>> > > submitting your SQL jobs? You don't use Ververica's SQL
>>> > service but
>>> > > have
>>> > > built a regular JAR file, right? If this is the case, can
>>> you
>>> > share
>>> > > your
>>> > > pom.xml file with us? The Flink version stays constant at
>>> 1.12?
>>> > >
>>> > > Regards,
>>> > > Timo
>>> > >
>>> > > On 22.07.21 12:22, Natu Lauchande wrote:
>>> > > > Good day Flink community,
>>> > > >
>>> > > > Apache Flink/Ververica Community Edition - Question
>>> > > >
>>> > > >
>>> > > > I am having an issue with my Flink SQL jobs since
>>> updating
>>> > > from Flink
>>> > > > 1.12/Ververica 2.4 to Ververica 2.5 . For all the jobs
>>> > running on
>>> > > > parquet and S3 i am getting the following error
>>> continuously:
>>> > > >
>>> > > > INITIALIZING to FAILED on 10.243.3.0:42337-2a3224 @
>>> > > > 10-243-3-0.flink-metrics.vvp-jobs.svc.cluster.local
>>> > (dataPort=39309).
>>> > > >
>>> > > > java.lang.NoClassDefFoundError:
>>> > org/apache/hadoop/conf/Configuration
>>> > > >
>>> > > > at java.lang.Class.getDeclaredConstructors0(Native
>>> Method)
>>> > > ~[?:1.8.0_292]
>>> > > >
>>> > > > at
>>> > java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
>>> > > > ~[?:1.8.0_292]
>>> > > >
>>> > > > at
>>> java.lang.Class.getDeclaredConstructors(Class.java:2020)
>>> > > ~[?:1.8.0_292]
>>> > > >
>>> > > > *....*
>>> > > >
>>> > > > at
>>> > java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
>>> > > > ~[?:1.8.0_292]
>>> > > >
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:615)
>>> > >
>>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>>> > > >
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:600)
>>> > >
>>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>>> > > >
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:587)
>>> > >
>>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>>> > > >
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:541)
>>> > >
>>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>>> > > >
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperatorFactory(StreamConfig.java:322)
>>> > >
>>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>>> > > >
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperator(OperatorChain.java:653)
>>> > >
>>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>>> > > >
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperatorChain(OperatorChain.java:626)
>>> > >
>>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>>> > > >
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:566)
>>> > >
>>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>>> > > >
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperatorChain(OperatorChain.java:616)
>>> > >
>>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>>> > > >
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:566)
>>> > >
>>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>>> > > >
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperatorChain(OperatorChain.java:616)
>>> > >
>>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>>> > > >
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:566)
>>> > >
>>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>>> > > >
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperatorChain(OperatorChain.java:616)
>>> > >
>>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>>> > > >
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:566)
>>> > >
>>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>>> > > >
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(OperatorChain.java:181)
>>> > >
>>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>>> > > >
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.flink.streaming.runtime.tasks.StreamTask.executeRestore(StreamTask.java:548)
>>> > >
>>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>>> > > >
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:647)
>>> > >
>>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>>> > > >
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:537)
>>> > >
>>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>>> > > >
>>> > > > at
>>> > org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:759)
>>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>>> > > >
>>> > > > at
>>> > org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
>>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>>> > > >
>>> > > > at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_292]
>>> > > >
>>> > > > Caused by: java.lang.ClassNotFoundException:
>>> > > > org.apache.hadoop.conf.Configuration
>>> > > >
>>> > > > at
>>> java.net.URLClassLoader.findClass(URLClassLoader.java:382)
>>> > > ~[?:1.8.0_292]
>>> > > >
>>> > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
>>> > > ~[?:1.8.0_292]
>>> > > >
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:64)
>>> > >
>>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>>> > > >
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:65)
>>> > >
>>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>>> > > >
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
>>> > >
>>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>>> > > >
>>> > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
>>> > > ~[?:1.8.0_292]
>>> > > >
>>> > > > ... 57 more
>>> > > >
>>> > > > 2021-07-22 09:38:43,095 DEBUG
>>> > > > org.apache.flink.runtime.scheduler.SharedSlot[] - Remove
>>> > logical
>>> > > slot
>>> > > > (SlotRequestId{4297879e795d0516e36a7c26ccc795b2}) for
>>> > execution
>>> > > vertex
>>> > > > (id cbc357ccb763df2852fee8c4fc7d55f2_0) from the
>>> physical slot
>>> > > > (SlotRequestId{df7c49a6610b56f26aea214c05bcd9ed})
>>> > > >
>>> > > > 2021-07-22 09:38:43,096 DEBUG
>>> > > > org.apache.flink.runtime.scheduler.SharedSlot[] - Release
>>> > shared
>>> > > slot
>>> > > > externally
>>> (SlotRequestId{df7c49a6610b56f26aea214c05bcd9ed})
>>> > > >
>>> > > >
>>> > > > Everything works well when i roll back to Ververica v2.4,
>>> > has anyone
>>> > > > experienced this error before.
>>> > > >
>>> > > > Thanks,
>>> > > >
>>> > > > Natu
>>> > > >
>>> > >
>>> >
>>>
>>>
Re: Issue with Flink jobs after upgrading to Flink 1.13.1/Ververica 2.5 - java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
Posted by Ingo Bürk <in...@ververica.com>.
Hi Natu,
Something you could try is removing the packaged parquet format and
defining a custom format[1]. For this custom format you can then fix the
dependencies by packaging all of the following into the format:
* flink-sql-parquet
* flink-shaded-hadoop-2-uber
* hadoop-aws
* aws-java-sdk-bundle
* guava
This isn't entirely straight-forward, unfortunately, and I haven't verified
it. However, with Ververica Platform 2.6, to be released shortly after
Flink 1.15, it should also work again.
[1]
https://docs.ververica.com/user_guide/sql_development/connectors.html#custom-connectors-and-formats
Best
Ingo
On Tue, Dec 7, 2021 at 6:23 AM Natu Lauchande <nl...@gmail.com> wrote:
> Hey Timo and Flink community,
>
> I wonder if there is a fix for this issue. The last time I rollbacked to
> version 12 of Flink and downgraded Ververica.
>
> I am really keen to leverage the new features on the latest versions of
> Ververica 2.5+ , i have tried a myriad of tricks suggested ( example :
> building the image with hadoop-client libraries) :
>
> java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
> at java.lang.Class.getDeclaredConstructors(Class.java:2020)
> at java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass
> .java:1961)
> at java.io.ObjectStreamClass.access$100(ObjectStreamClass.java:79)
> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:275)
> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:273)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.io.ObjectStreamClass.getSerialVersionUID(ObjectStreamClass
> .java:272)
> at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:694)
> at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:
> 2003)
> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1850
> )
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream
> .java:2160)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:
> 2405)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:
> 2329)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream
> .java:2187)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:
> 2405)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:
> 2329)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream
> .java:2187)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:
> 2405)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:
> 2329)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream
> .java:2187)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:
> 2405)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:
> 2329)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream
> .java:2187)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:
> 2405)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:
> 2329)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream
> .java:2187)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
> at org.apache.flink.util.InstantiationUtil.deserializeObject(
> InstantiationUtil.java:615)
> at org.apache.flink.util.InstantiationUtil.deserializeObject(
> InstantiationUtil.java:600)
> at org.apache.flink.util.InstantiationUtil.deserializeObject(
> InstantiationUtil.java:587)
> at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(
> InstantiationUtil.java:541)
> at org.apache.flink.streaming.api.graph.StreamConfig
> .getStreamOperatorFactory(StreamConfig.java:322)
> at org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(
> OperatorChain.java:159)
> at org.apache.flink.streaming.runtime.tasks.StreamTask.executeRestore(
> StreamTask.java:551)
> at org.apache.flink.streaming.runtime.tasks.StreamTask
> .runWithCleanUpOnFail(StreamTask.java:650)
> at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(
> StreamTask.java:540)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:759)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.
> Configuration
> at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at org.apache.flink.util.FlinkUserCodeClassLoader
> .loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:64)
> at org.apache.flink.util.ChildFirstClassLoader
> .loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:65)
> at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(
> FlinkUserCodeClassLoader.java:48)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 48 more
>
> This error occurs when writing to StreamFileWriting on S3 in Parquet
> format.
>
>
> Thanks,
> Natu
>
> On Thu, Jul 22, 2021 at 3:53 PM Timo Walther <tw...@apache.org> wrote:
>
>> Thanks, this should definitely work with the pre-packaged connectors of
>> Ververica platform.
>>
>> I guess we have to investigate what is going on. Until then, a
>> workaround could be to add Hadoop manually and set the HADOOP_CLASSPATH
>> environment variable. The root cause seems that Hadoop cannot be found.
>>
>> Alternatively, you could also build a custom image and include Hadoop in
>> the lib folder of Flink:
>>
>> https://docs.ververica.com/v1.3/platform/installation/custom_images.html
>>
>> I hope this helps. I will get back to you if we have a fix ready.
>>
>> Regards,
>> Timo
>>
>>
>>
>> On 22.07.21 14:30, Natu Lauchande wrote:
>> > Sure.
>> >
>> > That's how the ddl table looks like:
>> >
>> > CREATETABLEtablea (
>> >
>> > `a` BIGINT,
>> >
>> > `b` BIGINT,
>> >
>> > `c` BIGINT
>> >
>> > )
>> >
>> > COMMENT ''
>> >
>> > WITH(
>> >
>> > 'auto-compaction'='false',
>> >
>> > 'connector'='filesystem',
>> >
>> > 'format'='parquet',
>> >
>> > 'parquet.block.size'='134217728',
>> >
>> > 'parquet.compression'='SNAPPY',
>> >
>> > 'parquet.dictionary.page.size'='1048576',
>> >
>> > 'parquet.enable.dictionary'='true',
>> >
>> > 'parquet.page.size'='1048576',
>> >
>> > 'parquet.writer.max-padding'='2097152',
>> >
>> > 'path'='s3a://test/test’,
>> >
>> > 'sink.partition-commit.delay'='1 h',
>> >
>> > 'sink.partition-commit.policy.kind'='success-file',
>> >
>> > 'sink.partition-commit.success-file.name
>> > <http://sink.partition-commit.success-file.name>'='_SUCCESS',
>> >
>> > 'sink.partition-commit.trigger'='process-time',
>> >
>> > 'sink.rolling-policy.check-interval'='20 min',
>> >
>> > 'sink.rolling-policy.file-size'='128MB',
>> >
>> > 'sink.rolling-policy.rollover-interval'='2 h'
>> >
>> > );
>> >
>> >
>> >
>> > When a change the connector to a blackhole it immediately works without
>> > errors. I have the redacted the names and paths.
>> >
>> >
>> >
>> > Thanks,
>> > Natu
>> >
>> >
>> > On Thu, Jul 22, 2021 at 2:24 PM Timo Walther <twalthr@apache.org
>> > <ma...@apache.org>> wrote:
>> >
>> > Maybe you can share also which connector/format you are using? What
>> is
>> > the DDL?
>> >
>> > Regards,
>> > Timo
>> >
>> >
>> > On 22.07.21 14:11, Natu Lauchande wrote:
>> > > Hey Timo,
>> > >
>> > > Thanks for the reply.
>> > >
>> > > No custom file as we are using Flink SQL and submitting the job
>> > directly
>> > > through the SQL Editor UI. We are using Flink 1.13.1 as the
>> > supported
>> > > flink version. No custom code all through Flink SQL on UI no
>> jars.
>> > >
>> > > Thanks,
>> > > Natu
>> > >
>> > > On Thu, Jul 22, 2021 at 2:08 PM Timo Walther <twalthr@apache.org
>> > <ma...@apache.org>
>> > > <mailto:twalthr@apache.org <ma...@apache.org>>> wrote:
>> > >
>> > > Hi Natu,
>> > >
>> > > Ververica Platform 2.5 has updated the bundled Hadoop version
>> > but this
>> > > should not result in a NoClassDefFoundError exception. How
>> > are you
>> > > submitting your SQL jobs? You don't use Ververica's SQL
>> > service but
>> > > have
>> > > built a regular JAR file, right? If this is the case, can you
>> > share
>> > > your
>> > > pom.xml file with us? The Flink version stays constant at
>> 1.12?
>> > >
>> > > Regards,
>> > > Timo
>> > >
>> > > On 22.07.21 12:22, Natu Lauchande wrote:
>> > > > Good day Flink community,
>> > > >
>> > > > Apache Flink/Ververica Community Edition - Question
>> > > >
>> > > >
>> > > > I am having an issue with my Flink SQL jobs since updating
>> > > from Flink
>> > > > 1.12/Ververica 2.4 to Ververica 2.5 . For all the jobs
>> > running on
>> > > > parquet and S3 i am getting the following error
>> continuously:
>> > > >
>> > > > INITIALIZING to FAILED on 10.243.3.0:42337-2a3224 @
>> > > > 10-243-3-0.flink-metrics.vvp-jobs.svc.cluster.local
>> > (dataPort=39309).
>> > > >
>> > > > java.lang.NoClassDefFoundError:
>> > org/apache/hadoop/conf/Configuration
>> > > >
>> > > > at java.lang.Class.getDeclaredConstructors0(Native Method)
>> > > ~[?:1.8.0_292]
>> > > >
>> > > > at
>> > java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
>> > > > ~[?:1.8.0_292]
>> > > >
>> > > > at
>> java.lang.Class.getDeclaredConstructors(Class.java:2020)
>> > > ~[?:1.8.0_292]
>> > > >
>> > > > *....*
>> > > >
>> > > > at
>> > java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
>> > > > ~[?:1.8.0_292]
>> > > >
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:615)
>> > >
>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>> > > >
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:600)
>> > >
>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>> > > >
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:587)
>> > >
>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>> > > >
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:541)
>> > >
>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>> > > >
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperatorFactory(StreamConfig.java:322)
>> > >
>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>> > > >
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperator(OperatorChain.java:653)
>> > >
>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>> > > >
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperatorChain(OperatorChain.java:626)
>> > >
>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>> > > >
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:566)
>> > >
>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>> > > >
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperatorChain(OperatorChain.java:616)
>> > >
>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>> > > >
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:566)
>> > >
>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>> > > >
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperatorChain(OperatorChain.java:616)
>> > >
>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>> > > >
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:566)
>> > >
>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>> > > >
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperatorChain(OperatorChain.java:616)
>> > >
>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>> > > >
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:566)
>> > >
>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>> > > >
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(OperatorChain.java:181)
>> > >
>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>> > > >
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.flink.streaming.runtime.tasks.StreamTask.executeRestore(StreamTask.java:548)
>> > >
>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>> > > >
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:647)
>> > >
>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>> > > >
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:537)
>> > >
>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>> > > >
>> > > > at
>> > org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:759)
>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>> > > >
>> > > > at
>> > org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>> > > >
>> > > > at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_292]
>> > > >
>> > > > Caused by: java.lang.ClassNotFoundException:
>> > > > org.apache.hadoop.conf.Configuration
>> > > >
>> > > > at
>> java.net.URLClassLoader.findClass(URLClassLoader.java:382)
>> > > ~[?:1.8.0_292]
>> > > >
>> > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
>> > > ~[?:1.8.0_292]
>> > > >
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:64)
>> > >
>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>> > > >
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:65)
>> > >
>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>> > > >
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
>> > >
>> > > > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1[]
>> > > >
>> > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
>> > > ~[?:1.8.0_292]
>> > > >
>> > > > ... 57 more
>> > > >
>> > > > 2021-07-22 09:38:43,095 DEBUG
>> > > > org.apache.flink.runtime.scheduler.SharedSlot[] - Remove
>> > logical
>> > > slot
>> > > > (SlotRequestId{4297879e795d0516e36a7c26ccc795b2}) for
>> > execution
>> > > vertex
>> > > > (id cbc357ccb763df2852fee8c4fc7d55f2_0) from the physical
>> slot
>> > > > (SlotRequestId{df7c49a6610b56f26aea214c05bcd9ed})
>> > > >
>> > > > 2021-07-22 09:38:43,096 DEBUG
>> > > > org.apache.flink.runtime.scheduler.SharedSlot[] - Release
>> > shared
>> > > slot
>> > > > externally
>> (SlotRequestId{df7c49a6610b56f26aea214c05bcd9ed})
>> > > >
>> > > >
>> > > > Everything works well when i roll back to Ververica v2.4,
>> > has anyone
>> > > > experienced this error before.
>> > > >
>> > > > Thanks,
>> > > >
>> > > > Natu
>> > > >
>> > >
>> >
>>
>>