You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Kumiko Yada <Ku...@ds-iq.com> on 2016/05/06 04:52:36 UTC
Cannot Authenticate the Azure Datalake Store
Hello,
I'm writing the custom processor to create a file in the Azure Datalake Store. I wrote the sample java program to do using this sample code, https://azure.microsoft.com/en-us/documentation/articles/data-lake-store-get-started-java-sdk/. This is working fine. However, when I wrote the code in the custom processor, it's failing if the Authenticate code with this this error when this custom processor is running. I didn't get any build error when I built this custom processor. Do you have any ideas why I'm getting this error? Is there any file I need to copy to the nifi folder other than nar file for the custom processor?
2016-05-05 20:37:37,338 ERROR [Timer-Driven Process Thread-4] d.p.custom.CreateFileAzureDatalakeStore CreateFileAzureDatalakeStore[id=f89e5860-1d99-4d20-be71-769f6bb775c7] CreateFileAzureDatalakeStore[id=f89e5860-1d99-4d20-be71-769f6bb775c7] failed to process due to java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method <init>()V not found; rolling back session: java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method <init>()V not found
2016-05-05 20:37:37,342 INFO [Flow Service Tasks Thread-2] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@3dcbc96c // Another save pending = false
2016-05-05 20:37:37,342 ERROR [Timer-Driven Process Thread-4] d.p.custom.CreateFileAzureDatalakeStore
java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method <init>()V not found
at com.microsoft.azure.management.datalake.store.DataLakeStoreAccountManagementClientImpl.<init>(DataLakeStoreAccountManagementClientImpl.java:176) ~[na:na]
at com.microsoft.azure.management.datalake.store.DataLakeStoreAccountManagementClientImpl.<init>(DataLakeStoreAccountManagementClientImpl.java:166) ~[na:na]
at dsiq.processors.custom.CreateFileAzureDatalakeStore.SetupClients(CreateFileAzureDatalakeStore.java:286) ~[na:na]
at dsiq.processors.custom.CreateFileAzureDatalakeStore.onTrigger(CreateFileAzureDatalakeStore.java:269) ~[na:na]
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1057) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_77]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_77]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_77]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_77]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
2016-05-05 20:37:37,346 ERROR [Timer-Driven Process Thread-4] d.p.custom.CreateFileAzureDatalakeStore CreateFileAzureDatalakeStore[id=f89e5860-1d99-4d20-be71-769f6bb775c7] CreateFileAzureDatalakeStore[id=f89e5860-1d99-4d20-be71-769f6bb775c7] failed to process session due to java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method <init>()V not found: java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method <init>()V not found
2016-05-05 20:37:37,347 ERROR [Timer-Driven Process Thread-4] d.p.custom.CreateFileAzureDatalakeStore
java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method <init>()V not found
at com.microsoft.azure.management.datalake.store.DataLakeStoreAccountManagementClientImpl.<init>(DataLakeStoreAccountManagementClientImpl.java:176) ~[na:na]
at com.microsoft.azure.management.datalake.store.DataLakeStoreAccountManagementClientImpl.<init>(DataLakeStoreAccountManagementClientImpl.java:166) ~[na:na]
at dsiq.processors.custom.CreateFileAzureDatalakeStore.SetupClients(CreateFileAzureDatalakeStore.java:286) ~[na:na]
at dsiq.processors.custom.CreateFileAzureDatalakeStore.onTrigger(CreateFileAzureDatalakeStore.java:269) ~[na:na]
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1057) ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_77]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_77]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_77]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_77]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
ApplicationTokenCredentials creds = new ApplicationTokenCredentials(_clientId, _tenantId, _clientSecret, null);
_adlsClient = new DataLakeStoreAccountManagementClientImpl(creds); ---------- failing in this code
_adlsFileSystemClient = new DataLakeStoreFileSystemManagementClientImpl(creds);
_adlsClient.setSubscriptionId(_subId);
Thanks
Kumiko
RE: Cannot Authenticate the Azure Datalake Store
Posted by Kumiko Yada <Ku...@ds-iq.com>.
It looks like there are recent changes in …\.m2\repository\com\microsoft\azure\azure-mgmt-datalake-store\1.0.0-SNAPSHOT\azure-mgmt-datalake-store-1.0.0-20160420.002135-2.jar!\com\microsoft\azure\management\datalake\store\DataLakeStoreAccountManagementClientImpl.class. I’m getting the same errors in my sample java program now ☹
-Kumiko
From: Kumiko Yada [mailto:Kumiko.Yada@ds-iq.com]
Sent: Friday, May 6, 2016 10:45 AM
To: users@nifi.apache.org
Cc: Ki Kang <Ki...@ds-iq.com>
Subject: RE: Cannot Authenticate the Azure Datalake Store
HI Bryan,
Here are the dependencies:
<dependencies>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-api</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-processor-utils</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-mock</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>azure-client-authentication</artifactId>
<version>1.0.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>azure-mgmt-datalake-store</artifactId>
<version>1.0.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.5</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.6.4</version>
</dependency>
</dependencies>
Thanks
Kumiko
From: Bryan Bende [mailto:bbende@gmail.com]
Sent: Friday, May 6, 2016 7:04 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Cc: Ki Kang <Ki...@ds-iq.com>>
Subject: Re: Cannot Authenticate the Azure Datalake Store
Hello,
It seems like maybe a wrong version of a library is being used, or maybe a JAR is missing that needs to be included in your NAR.
Can you share what dependencies you have in the pom.xml of your processors project?
You can check under NIFI_HOME/work/nar/extensions/<YOUR_NAR>/META-INF/bundled-dependencies/ to see the JARs that are being included in your NAR.
-Bryan
On Fri, May 6, 2016 at 12:52 AM, Kumiko Yada <Ku...@ds-iq.com>> wrote:
Hello,
I’m writing the custom processor to create a file in the Azure Datalake Store. I wrote the sample java program to do using this sample code, https://azure.microsoft.com/en-us/documentation/articles/data-lake-store-get-started-java-sdk/. This is working fine. However, when I wrote the code in the custom processor, it’s failing if the Authenticate code with this this error when this custom processor is running. I didn’t get any build error when I built this custom processor. Do you have any ideas why I’m getting this error? Is there any file I need to copy to the nifi folder other than nar file for the custom processor?
2016-05-05 20:37:37,338 ERROR [Timer-Driven Process Thread-4] d.p.custom.CreateFileAzureDatalakeStore CreateFileAzureDatalakeStore[id=f89e5860-1d99-4d20-be71-769f6bb775c7] CreateFileAzureDatalakeStore[id=f89e5860-1d99-4d20-be71-769f6bb775c7] failed to process due to java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method <init>()V not found; rolling back session: java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method <init>()V not found
2016-05-05 20:37:37,342 INFO [Flow Service Tasks Thread-2] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@3dcbc96c<ma...@3dcbc96c> // Another save pending = false
2016-05-05 20:37:37,342 ERROR [Timer-Driven Process Thread-4] d.p.custom.CreateFileAzureDatalakeStore
java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method <init>()V not found
at com.microsoft.azure.management.datalake.store.DataLakeStoreAccountManagementClientImpl.<init>(DataLakeStoreAccountManagementClientImpl.java:176) ~[na:na]
at com.microsoft.azure.management.datalake.store.DataLakeStoreAccountManagementClientImpl.<init>(DataLakeStoreAccountManagementClientImpl.java:166) ~[na:na]
at dsiq.processors.custom.CreateFileAzureDatalakeStore.SetupClients(CreateFileAzureDatalakeStore.java:286) ~[na:na]
at dsiq.processors.custom.CreateFileAzureDatalakeStore.onTrigger(CreateFileAzureDatalakeStore.java:269) ~[na:na]
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1057) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_77]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_77]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_77]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_77]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
2016-05-05 20:37:37,346 ERROR [Timer-Driven Process Thread-4] d.p.custom.CreateFileAzureDatalakeStore CreateFileAzureDatalakeStore[id=f89e5860-1d99-4d20-be71-769f6bb775c7] CreateFileAzureDatalakeStore[id=f89e5860-1d99-4d20-be71-769f6bb775c7] failed to process session due to java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method <init>()V not found: java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method <init>()V not found
2016-05-05 20:37:37,347 ERROR [Timer-Driven Process Thread-4] d.p.custom.CreateFileAzureDatalakeStore
java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method <init>()V not found
at com.microsoft.azure.management.datalake.store.DataLakeStoreAccountManagementClientImpl.<init>(DataLakeStoreAccountManagementClientImpl.java:176) ~[na:na]
at com.microsoft.azure.management.datalake.store.DataLakeStoreAccountManagementClientImpl.<init>(DataLakeStoreAccountManagementClientImpl.java:166) ~[na:na]
at dsiq.processors.custom.CreateFileAzureDatalakeStore.SetupClients(CreateFileAzureDatalakeStore.java:286) ~[na:na]
at dsiq.processors.custom.CreateFileAzureDatalakeStore.onTrigger(CreateFileAzureDatalakeStore.java:269) ~[na:na]
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1057) ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_77]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_77]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_77]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_77]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
ApplicationTokenCredentials creds = new ApplicationTokenCredentials(_clientId, _tenantId, _clientSecret, null);
_adlsClient = new DataLakeStoreAccountManagementClientImpl(creds); ---------- failing in this code
_adlsFileSystemClient = new DataLakeStoreFileSystemManagementClientImpl(creds);
_adlsClient.setSubscriptionId(_subId);
Thanks
Kumiko
RE: Cannot Authenticate the Azure Datalake Store
Posted by Kumiko Yada <Ku...@ds-iq.com>.
HI Bryan,
Here are the dependencies:
<dependencies>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-api</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-processor-utils</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-mock</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>azure-client-authentication</artifactId>
<version>1.0.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>azure-mgmt-datalake-store</artifactId>
<version>1.0.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.5</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.6.4</version>
</dependency>
</dependencies>
Thanks
Kumiko
From: Bryan Bende [mailto:bbende@gmail.com]
Sent: Friday, May 6, 2016 7:04 AM
To: users@nifi.apache.org
Cc: Ki Kang <Ki...@ds-iq.com>
Subject: Re: Cannot Authenticate the Azure Datalake Store
Hello,
It seems like maybe a wrong version of a library is being used, or maybe a JAR is missing that needs to be included in your NAR.
Can you share what dependencies you have in the pom.xml of your processors project?
You can check under NIFI_HOME/work/nar/extensions/<YOUR_NAR>/META-INF/bundled-dependencies/ to see the JARs that are being included in your NAR.
-Bryan
On Fri, May 6, 2016 at 12:52 AM, Kumiko Yada <Ku...@ds-iq.com>> wrote:
Hello,
I’m writing the custom processor to create a file in the Azure Datalake Store. I wrote the sample java program to do using this sample code, https://azure.microsoft.com/en-us/documentation/articles/data-lake-store-get-started-java-sdk/. This is working fine. However, when I wrote the code in the custom processor, it’s failing if the Authenticate code with this this error when this custom processor is running. I didn’t get any build error when I built this custom processor. Do you have any ideas why I’m getting this error? Is there any file I need to copy to the nifi folder other than nar file for the custom processor?
2016-05-05 20:37:37,338 ERROR [Timer-Driven Process Thread-4] d.p.custom.CreateFileAzureDatalakeStore CreateFileAzureDatalakeStore[id=f89e5860-1d99-4d20-be71-769f6bb775c7] CreateFileAzureDatalakeStore[id=f89e5860-1d99-4d20-be71-769f6bb775c7] failed to process due to java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method <init>()V not found; rolling back session: java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method <init>()V not found
2016-05-05 20:37:37,342 INFO [Flow Service Tasks Thread-2] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@3dcbc96c<ma...@3dcbc96c> // Another save pending = false
2016-05-05 20:37:37,342 ERROR [Timer-Driven Process Thread-4] d.p.custom.CreateFileAzureDatalakeStore
java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method <init>()V not found
at com.microsoft.azure.management.datalake.store.DataLakeStoreAccountManagementClientImpl.<init>(DataLakeStoreAccountManagementClientImpl.java:176) ~[na:na]
at com.microsoft.azure.management.datalake.store.DataLakeStoreAccountManagementClientImpl.<init>(DataLakeStoreAccountManagementClientImpl.java:166) ~[na:na]
at dsiq.processors.custom.CreateFileAzureDatalakeStore.SetupClients(CreateFileAzureDatalakeStore.java:286) ~[na:na]
at dsiq.processors.custom.CreateFileAzureDatalakeStore.onTrigger(CreateFileAzureDatalakeStore.java:269) ~[na:na]
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1057) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_77]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_77]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_77]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_77]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
2016-05-05 20:37:37,346 ERROR [Timer-Driven Process Thread-4] d.p.custom.CreateFileAzureDatalakeStore CreateFileAzureDatalakeStore[id=f89e5860-1d99-4d20-be71-769f6bb775c7] CreateFileAzureDatalakeStore[id=f89e5860-1d99-4d20-be71-769f6bb775c7] failed to process session due to java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method <init>()V not found: java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method <init>()V not found
2016-05-05 20:37:37,347 ERROR [Timer-Driven Process Thread-4] d.p.custom.CreateFileAzureDatalakeStore
java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method <init>()V not found
at com.microsoft.azure.management.datalake.store.DataLakeStoreAccountManagementClientImpl.<init>(DataLakeStoreAccountManagementClientImpl.java:176) ~[na:na]
at com.microsoft.azure.management.datalake.store.DataLakeStoreAccountManagementClientImpl.<init>(DataLakeStoreAccountManagementClientImpl.java:166) ~[na:na]
at dsiq.processors.custom.CreateFileAzureDatalakeStore.SetupClients(CreateFileAzureDatalakeStore.java:286) ~[na:na]
at dsiq.processors.custom.CreateFileAzureDatalakeStore.onTrigger(CreateFileAzureDatalakeStore.java:269) ~[na:na]
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1057) ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_77]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_77]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_77]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_77]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
ApplicationTokenCredentials creds = new ApplicationTokenCredentials(_clientId, _tenantId, _clientSecret, null);
_adlsClient = new DataLakeStoreAccountManagementClientImpl(creds); ---------- failing in this code
_adlsFileSystemClient = new DataLakeStoreFileSystemManagementClientImpl(creds);
_adlsClient.setSubscriptionId(_subId);
Thanks
Kumiko
Re: Cannot Authenticate the Azure Datalake Store
Posted by Bryan Bende <bb...@gmail.com>.
Hello,
It seems like maybe a wrong version of a library is being used, or maybe a
JAR is missing that needs to be included in your NAR.
Can you share what dependencies you have in the pom.xml of your processors
project?
You can check under
NIFI_HOME/work/nar/extensions/<YOUR_NAR>/META-INF/bundled-dependencies/ to
see the JARs that are being included in your NAR.
-Bryan
On Fri, May 6, 2016 at 12:52 AM, Kumiko Yada <Ku...@ds-iq.com> wrote:
> Hello,
>
>
>
> I’m writing the custom processor to create a file in the Azure Datalake
> Store. I wrote the sample java program to do using this sample code,
> https://azure.microsoft.com/en-us/documentation/articles/data-lake-store-get-started-java-sdk/.
> This is working fine. However, when I wrote the code in the custom
> processor, it’s failing if the Authenticate code with this this error when
> this custom processor is running. I didn’t get any build error when I
> built this custom processor. Do you have any ideas why I’m getting this
> error? Is there any file I need to copy to the nifi folder other than nar
> file for the custom processor?
>
>
>
> *2016-05-05 20:37:37,338 ERROR [Timer-Driven Process Thread-4]
> d.p.custom.CreateFileAzureDatalakeStore
> CreateFileAzureDatalakeStore[id=f89e5860-1d99-4d20-be71-769f6bb775c7]
> CreateFileAzureDatalakeStore[id=f89e5860-1d99-4d20-be71-769f6bb775c7]
> failed to process due to java.lang.NoSuchMethodError:
> com.microsoft.azure.AzureServiceClient: method <init>()V not found; rolling
> back session: java.lang.NoSuchMethodError:
> com.microsoft.azure.AzureServiceClient: method <init>()V not found*
>
> *2016-05-05 20:37:37,342 INFO [Flow Service Tasks Thread-2]
> o.a.nifi.controller.StandardFlowService Saved flow controller
> org.apache.nifi.controller.FlowController@3dcbc96c // Another save pending
> = false*
>
> *2016-05-05 20:37:37,342 ERROR [Timer-Driven Process Thread-4]
> d.p.custom.CreateFileAzureDatalakeStore *
>
> *java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient:
> method <init>()V not found*
>
> * at
> com.microsoft.azure.management.datalake.store.DataLakeStoreAccountManagementClientImpl.<init>(DataLakeStoreAccountManagementClientImpl.java:176)
> ~[na:na]*
>
> * at
> com.microsoft.azure.management.datalake.store.DataLakeStoreAccountManagementClientImpl.<init>(DataLakeStoreAccountManagementClientImpl.java:166)
> ~[na:na]*
>
> * at
> dsiq.processors.custom.CreateFileAzureDatalakeStore.SetupClients(CreateFileAzureDatalakeStore.java:286)
> ~[na:na]*
>
> * at
> dsiq.processors.custom.CreateFileAzureDatalakeStore.onTrigger(CreateFileAzureDatalakeStore.java:269)
> ~[na:na]*
>
> * at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]*
>
> * at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1057)
> [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]*
>
> * at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]*
>
> * at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]*
>
> * at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123)
> [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]*
>
> * at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_77]*
>
> * at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_77]*
>
> * at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_77]*
>
> * at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_77]*
>
> * at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_77]*
>
> * at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_77]*
>
> * at java.lang.Thread.run(Thread.java:745)
> [na:1.8.0_77]*
>
> *2016-05-05 20:37:37,346 ERROR [Timer-Driven Process Thread-4]
> d.p.custom.CreateFileAzureDatalakeStore
> CreateFileAzureDatalakeStore[id=f89e5860-1d99-4d20-be71-769f6bb775c7]
> CreateFileAzureDatalakeStore[id=f89e5860-1d99-4d20-be71-769f6bb775c7]
> failed to process session due to java.lang.NoSuchMethodError:
> com.microsoft.azure.AzureServiceClient: method <init>()V not found:
> java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method
> <init>()V not found*
>
> *2016-05-05 20:37:37,347 ERROR [Timer-Driven Process Thread-4]
> d.p.custom.CreateFileAzureDatalakeStore *
>
> *java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient:
> method <init>()V not found*
>
> * at
> com.microsoft.azure.management.datalake.store.DataLakeStoreAccountManagementClientImpl.<init>(DataLakeStoreAccountManagementClientImpl.java:176)
> ~[na:na]*
>
> * at
> com.microsoft.azure.management.datalake.store.DataLakeStoreAccountManagementClientImpl.<init>(DataLakeStoreAccountManagementClientImpl.java:166)
> ~[na:na]*
>
> * at
> dsiq.processors.custom.CreateFileAzureDatalakeStore.SetupClients(CreateFileAzureDatalakeStore.java:286)
> ~[na:na]*
>
> * at
> dsiq.processors.custom.CreateFileAzureDatalakeStore.onTrigger(CreateFileAzureDatalakeStore.java:269)
> ~[na:na]*
>
> * at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]*
>
> * at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1057)
> ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]*
>
> * at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]*
>
> * at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]*
>
> * at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123)
> [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]*
>
> * at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_77]*
>
> * at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_77]*
>
> * at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_77]*
>
> * at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_77]*
>
> * at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_77]*
>
> * at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_77]*
>
> * at java.lang.Thread.run(Thread.java:745)
> [na:1.8.0_77]*
>
>
>
>
>
>
>
> ApplicationTokenCredentials creds = *new* ApplicationTokenCredentials(
> *_clientId*, *_tenantId*, *_clientSecret*, *null*);
>
>
>
> *_adlsClient* = *new* DataLakeStoreAccountManagementClientImpl(creds);
> ---------- failing in this code
>
> _adlsFileSystemClient = new DataLakeStoreFileSystemManagementClientImpl(
> *creds*);
>
> _adlsClient.setSubscriptionId(_subId);
>
>
>
> Thanks
>
> Kumiko
>
>
>