You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by William Gosse <Wi...@aifoundry.com> on 2019/12/24 18:16:16 UTC
Need help with DetectDuplicate
I’m trying to use DetectDuplicate processor but not having much luck. Here the config:
Cache Entry Identifier
${resourceId}
FlowFile Description
Ingestion
Age Off Duration
60 sec
Cache The Entry Identifier
true
Distributed Cache Service
DistributedMapCacheClientService
I created abd enabled a DistributedMapCacheClientService. Here’s its config:
Server Hostname
localhost
Server Port
4557
SSL Context ServiceNo value setCommunications Timeout
30 secs
When I run it I get the following error:
2019-12-24 13:14:05,355 ERROR [Timer-Driven Process Thread-9] o.a.n.p.standard.DetectDuplicate DetectDuplicate[id=38cb8a64-016f-1000-b55b-f6c4e0f69f61] Unable to communicate with cache when processing StandardFlowFileRecord[uuid=7f049d8e-1d04-4fee-9f04-320d6980bc55,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1577202782598-43, container=default, section=43], offset=74111, length=4528],offset=0,name=84068ffb-69b1-4471-abbd-29243d3be39e,size=4528] due to java.net.ConnectException: Connection refused: no further information: java.net.ConnectException: Connection refused: no further information
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111)
at org.apache.nifi.distributed.cache.client.StandardCommsSession.<init>(StandardCommsSession.java:52)
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.createCommsSession(DistributedMapCacheClientService.java:410)
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.leaseCommsSession(DistributedMapCacheClientService.java:425)
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.withCommsSession(DistributedMapCacheClientService.java:491)
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.getAndPutIfAbsent(DistributedMapCacheClientService.java:174)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87)
at com.sun.proxy.$Proxy142.getAndPutIfAbsent(Unknown Source)
at org.apache.nifi.processors.standard.DetectDuplicate.onTrigger(DetectDuplicate.java:183)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Not sure whats missing?
Re: Need help with DetectDuplicate
Posted by Matt Burgess <ma...@apache.org>.
William,
DistributedMapCacheClientService works in a standalone NiFi. Do you
have a DistributedMapCacheServer configured for localhost:4557 and
enabled?
On Thu, Jan 2, 2020 at 4:39 PM William Gosse
<Wi...@aifoundry.com> wrote:
>
> Will the DetectDuplicate with DistributedMapCacheClientService only work in a cluster? I’m trying to use it in a standalone NiFi.
>
>
>
> From: Emanuel Oliveira <em...@gmail.com>
> Sent: Tuesday, December 24, 2019 4:29 PM
> To: users@nifi.apache.org
> Subject: Re: Need help with DetectDuplicate
>
>
>
> [CAUTION: This email originated from outside of Kodak Alaris. Do not click links or open attachments unless you recognize the sender and know the content is safe.]
>
> ________________________________
>
> Hi,
>
>
>
> Depending on how your cluster setup you may need to add/setup ssl controller service?
>
>
>
> Emanuel
>
>
>
> On Tue 24 Dec 2019, 18:16 William Gosse, <Wi...@aifoundry.com> wrote:
>
> I’m trying to use DetectDuplicate processor but not having much luck. Here the config:
>
> Cache Entry Identifier
>
> ${resourceId}
>
> FlowFile Description
>
> Ingestion
>
> Age Off Duration
>
> 60 sec
>
> Cache The Entry Identifier
>
> true
>
> Distributed Cache Service
>
> DistributedMapCacheClientService
>
>
>
> I created abd enabled a DistributedMapCacheClientService. Here’s its config:
>
> Server Hostname
>
> localhost
>
> Server Port
>
> 4557
>
> SSL Context ServiceNo value setCommunications Timeout
>
> 30 secs
>
>
>
> When I run it I get the following error:
>
> 2019-12-24 13:14:05,355 ERROR [Timer-Driven Process Thread-9] o.a.n.p.standard.DetectDuplicate DetectDuplicate[id=38cb8a64-016f-1000-b55b-f6c4e0f69f61] Unable to communicate with cache when processing StandardFlowFileRecord[uuid=7f049d8e-1d04-4fee-9f04-320d6980bc55,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1577202782598-43, container=default, section=43], offset=74111, length=4528],offset=0,name=84068ffb-69b1-4471-abbd-29243d3be39e,size=4528] due to java.net.ConnectException: Connection refused: no further information: java.net.ConnectException: Connection refused: no further information
>
> java.net.ConnectException: Connection refused: no further information
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
>
> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111)
>
> at org.apache.nifi.distributed.cache.client.StandardCommsSession.<init>(StandardCommsSession.java:52)
>
> at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.createCommsSession(DistributedMapCacheClientService.java:410)
>
> at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.leaseCommsSession(DistributedMapCacheClientService.java:425)
>
> at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.withCommsSession(DistributedMapCacheClientService.java:491)
>
> at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.getAndPutIfAbsent(DistributedMapCacheClientService.java:174)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
>
> at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87)
>
> at com.sun.proxy.$Proxy142.getAndPutIfAbsent(Unknown Source)
>
> at org.apache.nifi.processors.standard.DetectDuplicate.onTrigger(DetectDuplicate.java:183)
>
> at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>
> at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
>
> at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
>
> at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> Not sure whats missing?
RE: Need help with DetectDuplicate
Posted by William Gosse <Wi...@aifoundry.com>.
Will the DetectDuplicate with DistributedMapCacheClientService only work in a cluster? I’m trying to use it in a standalone NiFi.
From: Emanuel Oliveira <em...@gmail.com>
Sent: Tuesday, December 24, 2019 4:29 PM
To: users@nifi.apache.org
Subject: Re: Need help with DetectDuplicate
[CAUTION: This email originated from outside of Kodak Alaris. Do not click links or open attachments unless you recognize the sender and know the content is safe.]
________________________________
Hi,
Depending on how your cluster setup you may need to add/setup ssl controller service?
Emanuel
On Tue 24 Dec 2019, 18:16 William Gosse, <Wi...@aifoundry.com>> wrote:
I’m trying to use DetectDuplicate processor but not having much luck. Here the config:
Cache Entry Identifier
${resourceId}
FlowFile Description
Ingestion
Age Off Duration
60 sec
Cache The Entry Identifier
true
Distributed Cache Service
DistributedMapCacheClientService
I created abd enabled a DistributedMapCacheClientService. Here’s its config:
Server Hostname
localhost
Server Port
4557
SSL Context ServiceNo value setCommunications Timeout
30 secs
When I run it I get the following error:
2019-12-24 13:14:05,355 ERROR [Timer-Driven Process Thread-9] o.a.n.p.standard.DetectDuplicate DetectDuplicate[id=38cb8a64-016f-1000-b55b-f6c4e0f69f61] Unable to communicate with cache when processing StandardFlowFileRecord[uuid=7f049d8e-1d04-4fee-9f04-320d6980bc55,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1577202782598-43, container=default, section=43], offset=74111, length=4528],offset=0,name=84068ffb-69b1-4471-abbd-29243d3be39e,size=4528] due to java.net.ConnectException: Connection refused: no further information: java.net.ConnectException: Connection refused: no further information
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111)
at org.apache.nifi.distributed.cache.client.StandardCommsSession.<init>(StandardCommsSession.java:52)
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.createCommsSession(DistributedMapCacheClientService.java:410)
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.leaseCommsSession(DistributedMapCacheClientService.java:425)
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.withCommsSession(DistributedMapCacheClientService.java:491)
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.getAndPutIfAbsent(DistributedMapCacheClientService.java:174)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87)
at com.sun.proxy.$Proxy142.getAndPutIfAbsent(Unknown Source)
at org.apache.nifi.processors.standard.DetectDuplicate.onTrigger(DetectDuplicate.java:183)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Not sure whats missing?
Re: Need help with DetectDuplicate
Posted by Emanuel Oliveira <em...@gmail.com>.
Hi,
Depending on how your cluster setup you may need to add/setup ssl
controller service?
Emanuel
On Tue 24 Dec 2019, 18:16 William Gosse, <Wi...@aifoundry.com>
wrote:
> I’m trying to use DetectDuplicate processor but not having much luck. Here
> the config:
>
> Cache Entry Identifier
>
> ${resourceId}
>
> FlowFile Description
>
> Ingestion
>
> Age Off Duration
>
> 60 sec
>
> Cache The Entry Identifier
>
> true
>
> Distributed Cache Service
>
> DistributedMapCacheClientService
>
>
>
> I created abd enabled a DistributedMapCacheClientService. Here’s its
> config:
>
> Server Hostname
>
> localhost
>
> Server Port
>
> 4557
>
> SSL Context ServiceNo value setCommunications Timeout
>
> 30 secs
>
>
>
> When I run it I get the following error:
>
> 2019-12-24 13:14:05,355 ERROR [Timer-Driven Process Thread-9]
> o.a.n.p.standard.DetectDuplicate
> DetectDuplicate[id=38cb8a64-016f-1000-b55b-f6c4e0f69f61] Unable to
> communicate with cache when processing
> StandardFlowFileRecord[uuid=7f049d8e-1d04-4fee-9f04-320d6980bc55,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1577202782598-43,
> container=default, section=43], offset=74111,
> length=4528],offset=0,name=84068ffb-69b1-4471-abbd-29243d3be39e,size=4528]
> due to java.net.ConnectException: Connection refused: no further
> information: java.net.ConnectException: Connection refused: no further
> information
>
> java.net.ConnectException: Connection refused: no further information
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
>
> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111)
>
> at
> org.apache.nifi.distributed.cache.client.StandardCommsSession.<init>(StandardCommsSession.java:52)
>
> at
> org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.createCommsSession(DistributedMapCacheClientService.java:410)
>
> at
> org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.leaseCommsSession(DistributedMapCacheClientService.java:425)
>
> at
> org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.withCommsSession(DistributedMapCacheClientService.java:491)
>
> at
> org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.getAndPutIfAbsent(DistributedMapCacheClientService.java:174)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
>
> at
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87)
>
> at com.sun.proxy.$Proxy142.getAndPutIfAbsent(Unknown
> Source)
>
> at
> org.apache.nifi.processors.standard.DetectDuplicate.onTrigger(DetectDuplicate.java:183)
>
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
>
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
>
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>
> at
> org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
> at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> Not sure whats missing?
>
Re: Need help with DetectDuplicate
Posted by Shawn Weeks <sw...@weeksconsulting.us>.
Did you create the actual map cache server in controller services? I couldn’t tell. All I saw was the client service.
Thanks
Shawn
Sent from my iPhone
On Dec 24, 2019, at 12:16 PM, William Gosse <Wi...@aifoundry.com> wrote:
I’m trying to use DetectDuplicate processor but not having much luck. Here the config:
Cache Entry Identifier
${resourceId}
FlowFile Description
Ingestion
Age Off Duration
60 sec
Cache The Entry Identifier
true
Distributed Cache Service
DistributedMapCacheClientService
I created abd enabled a DistributedMapCacheClientService. Here’s its config:
Server Hostname
localhost
Server Port
4557
SSL Context ServiceNo value setCommunications Timeout
30 secs
When I run it I get the following error:
2019-12-24 13:14:05,355 ERROR [Timer-Driven Process Thread-9] o.a.n.p.standard.DetectDuplicate DetectDuplicate[id=38cb8a64-016f-1000-b55b-f6c4e0f69f61] Unable to communicate with cache when processing StandardFlowFileRecord[uuid=7f049d8e-1d04-4fee-9f04-320d6980bc55,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1577202782598-43, container=default, section=43], offset=74111, length=4528],offset=0,name=84068ffb-69b1-4471-abbd-29243d3be39e,size=4528] due to java.net.ConnectException: Connection refused: no further information: java.net.ConnectException: Connection refused: no further information
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111)
at org.apache.nifi.distributed.cache.client.StandardCommsSession.<init>(StandardCommsSession.java:52)
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.createCommsSession(DistributedMapCacheClientService.java:410)
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.leaseCommsSession(DistributedMapCacheClientService.java:425)
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.withCommsSession(DistributedMapCacheClientService.java:491)
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.getAndPutIfAbsent(DistributedMapCacheClientService.java:174)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87)
at com.sun.proxy.$Proxy142.getAndPutIfAbsent(Unknown Source)
at org.apache.nifi.processors.standard.DetectDuplicate.onTrigger(DetectDuplicate.java:183)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Not sure whats missing?