You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Porta Léonard <le...@kudelskisecurity.com> on 2016/09/08 13:00:42 UTC
NiFI 1.0.0 errors with DetectDuplicate processor
Hello,
Under NiFI 1.0.0 I am trying to configure a DetectDuplicate processor.
I have configured and enabled
DistributedSetCacheServer, with by default configuration (Port=4557)
and
DistributedMapCacheClientService (Server Hostname=localhost, Server Port=4557, SSL Context Service=<empty>, Comm. Timeout=30 secs)
DetectDuplicate configuration is
Cache Entry Identifier=${my-attribute-key}
FlowFile Description=my text here
Age Off Duration=24 hrs
Distributed Cache Service= DistributedMapCacheClientService
Cache The Entry Identifier=true
May someone help.
Thanks
Leo
Here is the log from nifi-user.log :
2016-09-08 12:12:26,157 ERROR [Distributed Cache Server Communications Thread: 09a381b1-0157-1000-902d-067af19e4c19] o.a.n.d.cache.server.AbstractCacheServer org.apache.nifi.distributed.cache.server.AbstractCacheServer$1$1@759fdc8a unable to communicate with remote peer localhost due to java.io.IOException: IllegalRequest
2016-09-08 12:12:26,166 ERROR [Timer-Driven Process Thread-1] o.a.n.p.standard.DetectDuplicate DetectDuplicate[id=09981881-0157-1000-ac0f-43f500adcf1a] Unable to communicate with cache when processing StandardFlowFileRecord[uuid=35de4947-92b3-46fb-ada7-0a195600e4f1,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1473336740239-1293, container=default, section=269], offset=22578, length=1044],offset=0,name=689694820062267,size=1044] due to java.io.EOFException: java.io.EOFException
2016-09-08 12:12:26,169 ERROR [Timer-Driven Process Thread-1] o.a.n.p.standard.DetectDuplicate
java.io.EOFException: null
at java.io.DataInputStream.readInt(DataInputStream.java:392) ~[na:1.8.0_101]
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.readLengthDelimitedResponse(DistributedMapCacheClientService.java:220) ~[na:na]
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.access$100(DistributedMapCacheClientService.java:51) ~[na:na]
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService$4.execute(DistributedMapCacheClientService.java:176) ~[na:na]
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.withCommsSession(DistributedMapCacheClientService.java:305) ~[na:na]
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.getAndPutIfAbsent(DistributedMapCacheClientService.java:164) ~[na:na]
at sun.reflect.GeneratedMethodAccessor882.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_101]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_101]
at org.apache.nifi.controller.service.StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:177) ~[na:na]
at com.sun.proxy.$Proxy163.getAndPutIfAbsent(Unknown Source) ~[na:na]
at org.apache.nifi.processors.standard.DetectDuplicate.onTrigger(DetectDuplicate.java:182) ~[nifi-standard-processors-1.0.0.jar:1.0.0]
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-1.0.0.jar:1.0.0]
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064) [nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.0.0.jar:1.0.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_101]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_101]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_101]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
RE: NiFI 1.0.0 errors with DetectDuplicate processor
Posted by Porta Léonard <le...@kudelskisecurity.com>.
Thanks Bryan,
Everything worked well using the map cache server.
4 eyes are better than two !
Regards,
Léo
From: Bryan Bende [mailto:bbende@gmail.com]
Sent: jeudi 8 septembre 2016 15:28
To: users@nifi.apache.org
Subject: Re: NiFI 1.0.0 errors with DetectDuplicate processor
Hello,
I'm not sure if this is the problem, but I noticed you are using the map cache client with the set cache server.
I think the map cache client needs to be used with the map cache server. Can you let us know if that works.
-Bryan
On Thu, Sep 8, 2016 at 9:00 AM, Porta Léonard <le...@kudelskisecurity.com>> wrote:
Hello,
Under NiFI 1.0.0 I am trying to configure a DetectDuplicate processor.
I have configured and enabled
DistributedSetCacheServer, with by default configuration (Port=4557)
and
DistributedMapCacheClientService (Server Hostname=localhost, Server Port=4557, SSL Context Service=<empty>, Comm. Timeout=30 secs)
DetectDuplicate configuration is
Cache Entry Identifier=${my-attribute-key}
FlowFile Description=my text here
Age Off Duration=24 hrs
Distributed Cache Service= DistributedMapCacheClientService
Cache The Entry Identifier=true
May someone help.
Thanks
Leo
Here is the log from nifi-user.log :
2016-09-08 12:12:26,157 ERROR [Distributed Cache Server Communications Thread: 09a381b1-0157-1000-902d-067af19e4c19] o.a.n.d.cache.server.AbstractCacheServer org.apache.nifi.distributed.cache.server.AbstractCacheServer$1$1@759fdc8a<ma...@759fdc8a> unable to communicate with remote peer localhost due to java.io.IOException: IllegalRequest
2016-09-08 12:12:26,166 ERROR [Timer-Driven Process Thread-1] o.a.n.p.standard.DetectDuplicate DetectDuplicate[id=09981881-0157-1000-ac0f-43f500adcf1a] Unable to communicate with cache when processing StandardFlowFileRecord[uuid=35de4947-92b3-46fb-ada7-0a195600e4f1,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1473336740239-1293, container=default, section=269], offset=22578, length=1044],offset=0,name=689694820062267,size=1044] due to java.io.EOFException: java.io.EOFException
2016-09-08 12:12:26,169 ERROR [Timer-Driven Process Thread-1] o.a.n.p.standard.DetectDuplicate
java.io.EOFException: null
at java.io.DataInputStream.readInt(DataInputStream.java:392) ~[na:1.8.0_101]
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.readLengthDelimitedResponse(DistributedMapCacheClientService.java:220) ~[na:na]
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.access$100(DistributedMapCacheClientService.java:51) ~[na:na]
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService$4.execute(DistributedMapCacheClientService.java:176) ~[na:na]
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.withCommsSession(DistributedMapCacheClientService.java:305) ~[na:na]
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.getAndPutIfAbsent(DistributedMapCacheClientService.java:164) ~[na:na]
at sun.reflect.GeneratedMethodAccessor882.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_101]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_101]
at org.apache.nifi.controller.service.StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:177) ~[na:na]
at com.sun.proxy.$Proxy163.getAndPutIfAbsent(Unknown Source) ~[na:na]
at org.apache.nifi.processors.standard.DetectDuplicate.onTrigger(DetectDuplicate.java:182) ~[nifi-standard-processors-1.0.0.jar:1.0.0]
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-1.0.0.jar:1.0.0]
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064) [nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.0.0.jar:1.0.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_101]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_101]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_101]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
Re: NiFI 1.0.0 errors with DetectDuplicate processor
Posted by Bryan Bende <bb...@gmail.com>.
Hello,
I'm not sure if this is the problem, but I noticed you are using the map
cache client with the set cache server.
I think the map cache client needs to be used with the map cache server.
Can you let us know if that works.
-Bryan
On Thu, Sep 8, 2016 at 9:00 AM, Porta Léonard <
leonard.porta@kudelskisecurity.com> wrote:
> Hello,
>
>
>
> Under NiFI 1.0.0 I am trying to configure a DetectDuplicate processor.
>
>
>
> I have configured and enabled
>
> DistributedSetCacheServer, with by default configuration (Port=4557)
>
> and
>
> DistributedMapCacheClientService (Server Hostname=localhost, Server
> Port=4557, SSL Context Service=<empty>, Comm. Timeout=30 secs)
>
>
>
>
>
> DetectDuplicate configuration is
>
> Cache Entry Identifier=${my-attribute-key}
>
> FlowFile Description=my text here
>
> Age Off Duration=24 hrs
>
> Distributed Cache Service= DistributedMapCacheClientService
>
> Cache The Entry Identifier=true
>
>
>
>
>
> May someone help.
>
>
>
> Thanks
>
> Leo
>
>
>
> Here is the log from nifi-user.log :
>
> 2016-09-08 12:12:26,157 ERROR [Distributed Cache Server Communications
> Thread: 09a381b1-0157-1000-902d-067af19e4c19] o.a.n.d.cache.server.AbstractCacheServer
> org.apache.nifi.distributed.cache.server.AbstractCacheServer$1$1@759fdc8a
> unable to communicate with remote peer localhost due to
> java.io.IOException: IllegalRequest
>
> 2016-09-08 12:12:26,166 ERROR [Timer-Driven Process Thread-1]
> o.a.n.p.standard.DetectDuplicate DetectDuplicate[id=09981881-0157-1000-ac0f-43f500adcf1a]
> Unable to communicate with cache when processing
> StandardFlowFileRecord[uuid=35de4947-92b3-46fb-ada7-0a195600e4f1,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1473336740239-1293,
> container=default, section=269], offset=22578, length=1044],offset=0,name=689694820062267,size=1044]
> due to java.io.EOFException: java.io.EOFException
>
> 2016-09-08 12:12:26,169 ERROR [Timer-Driven Process Thread-1]
> o.a.n.p.standard.DetectDuplicate
>
> java.io.EOFException: null
>
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> ~[na:1.8.0_101]
>
> at org.apache.nifi.distributed.cache.client.
> DistributedMapCacheClientService.readLengthDelimitedResponse(
> DistributedMapCacheClientService.java:220) ~[na:na]
>
> at org.apache.nifi.distributed.cache.client.
> DistributedMapCacheClientService.access$100(DistributedMapCacheClientService.java:51)
> ~[na:na]
>
> at org.apache.nifi.distributed.cache.client.
> DistributedMapCacheClientService$4.execute(DistributedMapCacheClientService.java:176)
> ~[na:na]
>
> at org.apache.nifi.distributed.cache.client.
> DistributedMapCacheClientService.withCommsSession(
> DistributedMapCacheClientService.java:305) ~[na:na]
>
> at org.apache.nifi.distributed.cache.client.
> DistributedMapCacheClientService.getAndPutIfAbsent(
> DistributedMapCacheClientService.java:164) ~[na:na]
>
> at sun.reflect.GeneratedMethodAccessor882.invoke(Unknown Source)
> ~[na:na]
>
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_101]
>
> at java.lang.reflect.Method.invoke(Method.java:498)
> ~[na:1.8.0_101]
>
> at org.apache.nifi.controller.service.
> StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:177)
> ~[na:na]
>
> at com.sun.proxy.$Proxy163.getAndPutIfAbsent(Unknown Source)
> ~[na:na]
>
> at org.apache.nifi.processors.standard.DetectDuplicate.
> onTrigger(DetectDuplicate.java:182) ~[nifi-standard-processors-1.
> 0.0.jar:1.0.0]
>
> at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> [nifi-api-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.controller.StandardProcessorNode.onTrigger(
> StandardProcessorNode.java:1064) [nifi-framework-core-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.
> call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.0.0.
> jar:1.0.0]
>
> at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.
> call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.0.0.
> jar:1.0.0]
>
> at org.apache.nifi.controller.scheduling.
> TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> [nifi-framework-core-1.0.0.jar:1.0.0]
>
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_101]
>
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_101]
>
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_101]
>
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_101]
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_101]
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_101]
>
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
>
>
>
>
>