You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by "Weiss, Christian" <Ch...@sva.de> on 2021/11/02 14:00:21 UTC
ClassNotFoundException on FetchHDFS
Hi guys,
we did an upgrade to NiFi 1.14.0 from 1.11.4 and migrated some our Flows for testing.
Unfortunately the FetchHDFS Processor we are using is throwing the following Exception:
java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.util.JsonSerialization
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.writeJson(KMSClientProvider.java:255)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:551)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:540)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:829)
at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:354)
at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:350)
at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:175)
at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:350)
at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:532)
at org.apache.hadoop.hdfs.HdfsKMSUtil.decryptEncryptedDataEncryptionKey(HdfsKMSUtil.java:216)
at org.apache.hadoop.hdfs.HdfsKMSUtil.createWrappedInputStream(HdfsKMSUtil.java:196)
at org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:959)
at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:335)
at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:329)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:346)
at org.apache.nifi.processors.hadoop.FetchHDFS$1.run(FetchHDFS.java:161)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1822)
at org.apache.nifi.processors.hadoop.FetchHDFS.onTrigger(FetchHDFS.java:140)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1202)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Seems like a missing package in the current NiFi build.
What did you think?
Thanks,
Christian
----------------------------
Das SVA Mail-System ist mit einem Mailverschlusselungs-Gateway ausgestattet. Wenn Sie moechten, dass an Sie gerichtete E-Mails verschluesselt werden, senden Sie einfach eine S/MIME-signierte E-Mail oder Ihren PGP Public Key an christian.weiss@sva.de.
The SVA mail system is equipped with an email encryption gateway. If you want email sent to you to be encrypted please send a S/MIME signed email or your PGP public key to christian.weiss@sva.de.
AW: ClassNotFoundException on FetchHDFS
Posted by "Weiss, Christian" <Ch...@sva.de>.
Hi David,
with NiFi 1.15.0 RC2 FetchHDFS is working as expected, again.
Thanks for your help!
Regards,
Christian
Von: David Handermann <ex...@apache.org>
Gesendet: Dienstag, 2. November 2021 18:05
An: users@nifi.apache.org
Betreff: Re: ClassNotFoundException on FetchHDFS
Christian,
Thanks for reporting this issue and providing the stack trace. It sounds like this might an issue that was resolved not too long after the release of version 1.14.0:
https://issues.apache.org/jira/browse/NIFI-8938
This particular issue is resolved in the current main branch and will be part of version 1.15.0, which is currently being evaluated. If you are in a position to run some additional tests, you can download 1.15.0 Release Candidate 2 from the following location:
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/
It would be great if you could evaluate whether or not this issue is resolved in 1.15.0 RC2.
Regards,
David Handermann
On Tue, Nov 2, 2021 at 9:00 AM Weiss, Christian <Ch...@sva.de>> wrote:
Hi guys,
we did an upgrade to NiFi 1.14.0 from 1.11.4 and migrated some our Flows for testing.
Unfortunately the FetchHDFS Processor we are using is throwing the following Exception:
java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.util.JsonSerialization
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.writeJson(KMSClientProvider.java:255)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:551)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:540)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:829)
at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:354)
at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:350)
at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:175)
at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:350)
at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:532)
at org.apache.hadoop.hdfs.HdfsKMSUtil.decryptEncryptedDataEncryptionKey(HdfsKMSUtil.java:216)
at org.apache.hadoop.hdfs.HdfsKMSUtil.createWrappedInputStream(HdfsKMSUtil.java:196)
at org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:959)
at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:335)
at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:329)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:346)
at org.apache.nifi.processors.hadoop.FetchHDFS$1.run(FetchHDFS.java:161)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1822)
at org.apache.nifi.processors.hadoop.FetchHDFS.onTrigger(FetchHDFS.java:140)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1202)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Seems like a missing package in the current NiFi build.
What did you think?
Thanks,
Christian
Das SVA Mail-System ist mit einem Mailverschluesselungs-Gateway ausgestattet. Wenn Sie moechten, dass an Sie gerichtete E-Mails verschluesselt werden, senden Sie einfach eine S/MIME-signierte E-Mail oder Ihren PGP Public Key an christian.weiss@sva.de<ma...@sva.de>.
The SVA mail system is equipped with an email encryption gateway. If you want email sent to you to be encrypted please send a S/MIME signed email or your PGP public key to christian.weiss@sva.de<ma...@sva.de>.
Re: ClassNotFoundException on FetchHDFS
Posted by David Handermann <ex...@apache.org>.
Christian,
Thanks for reporting this issue and providing the stack trace. It sounds
like this might an issue that was resolved not too long after the release
of version 1.14.0:
https://issues.apache.org/jira/browse/NIFI-8938
This particular issue is resolved in the current main branch and will be
part of version 1.15.0, which is currently being evaluated. If you are in
a position to run some additional tests, you can download 1.15.0 Release
Candidate 2 from the following location:
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/
It would be great if you could evaluate whether or not this issue is
resolved in 1.15.0 RC2.
Regards,
David Handermann
On Tue, Nov 2, 2021 at 9:00 AM Weiss, Christian <Ch...@sva.de>
wrote:
> Hi guys,
>
>
>
> we did an upgrade to NiFi 1.14.0 from 1.11.4 and migrated some our Flows
> for testing.
>
> Unfortunately the FetchHDFS Processor we are using is throwing the
> following Exception:
>
>
>
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.util.JsonSerialization
>
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.writeJson(KMSClientProvider.java:255)
>
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:551)
>
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:540)
>
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:829)
>
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:354)
>
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:350)
>
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:175)
>
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:350)
>
> at
> org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:532)
>
> at
> org.apache.hadoop.hdfs.HdfsKMSUtil.decryptEncryptedDataEncryptionKey(HdfsKMSUtil.java:216)
>
> at
> org.apache.hadoop.hdfs.HdfsKMSUtil.createWrappedInputStream(HdfsKMSUtil.java:196)
>
> at
> org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:959)
>
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:335)
>
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:329)
>
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:346)
>
> at
> org.apache.nifi.processors.hadoop.FetchHDFS$1.run(FetchHDFS.java:161)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:360)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1822)
>
> at
> org.apache.nifi.processors.hadoop.FetchHDFS.onTrigger(FetchHDFS.java:140)
>
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1202)
>
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
>
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103)
>
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> Seems like a missing package in the current NiFi build.
>
> What did you think?
>
>
>
> Thanks,
>
> Christian
>
> Das SVA Mail-System ist mit einem Mailverschluesselungs-Gateway
> ausgestattet. Wenn Sie moechten, dass an Sie gerichtete E-Mails
> verschluesselt werden, senden Sie einfach eine S/MIME-signierte E-Mail oder
> Ihren PGP Public Key an christian.weiss@sva.de.
>
> The SVA mail system is equipped with an email encryption gateway. If you
> want email sent to you to be encrypted please send a S/MIME signed email or
> your PGP public key to christian.weiss@sva.de.
>