You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by "Adams, Jeremiah" <je...@pearson.com> on 2015/07/08 18:14:02 UTC
Problem with Custom Processor Incorrectly Routing FlowFile
I am implementing a custom processor based on JSON messages.
I have unit tests around the processor. These verify the expected
functionality and are similar to:
ClassLoader loader = Thread.currentThread().getContextClassLoader();
InputStream inputStream =
loader.getResourceAsStream("matched.json");
testRunner.enqueue(inputStream);
testRunner.run();
testRunner.assertTransferCount(JsonRouteProcessor.REL_MATCHED, 1);
testRunner.assertQueueEmpty();
Once I drop my processor into the NiFi server and put it in my flow, this
processor does not route messages as expected and as verified in the unit
tests. I can't see into the logic in the logs, just that this processor
routed the message to the wrong downstream processor. I've even taken the
messages the processor rejects in my NiFi flow and ran them through my unit
tests - they behave as expected.
Do I have a misunderstanding of the correct way to unit test the onTrigger
method for my processor?
*Jeremiah Adams*
Senior Software Developer
Pearson
2154 East Commons Ave.
Suite 400
Centennial, CO 80122
Always Learning
Learn more at www.pearson.com
RE: Problem with Custom Processor Incorrectly Routing FlowFile
Posted by Mark Payne <ma...@hotmail.com>.
OK great. The nice thing about the existing codebase is that the nifi-standard-processors module has a lot of
really great examples that should help bring people up to speed, and they are generally pretty well
unit tested. Certainly that's a good place to go for examples.
But if you run into any issues, let us know and I'm sure we'll get everything straightened out!
Thanks
-Mark
----------------------------------------
> Date: Wed, 8 Jul 2015 13:12:19 -0600
> Subject: Re: Problem with Custom Processor Incorrectly Routing FlowFile
> From: jeremiah.adams@pearson.com
> To: dev@nifi.incubator.apache.org
>
> I was glad to see the mocks. When I first saw the stubs for the processor I
> was wondering how I was going to test it. After I get this first POC out
> I'll slowdown and back up through documentation and pass along any
> suggestions, etc. I tend to just dive into things and see what shakes out.
> Ease of use is big win for me. It is a good sign if I can generally "just
> figure things out".
>
> *Jeremiah Adams*
>
> Senior Software Developer
> Pearson
>
> 2154 East Commons Ave.
> Suite 400
> Centennial, CO 80122
>
>
> Always Learning
> Learn more at www.pearson.com
>
> On Wed, Jul 8, 2015 at 11:05 AM, Mark Payne <ma...@hotmail.com> wrote:
>
>> Great, glad I could help.
>>
>> I've found the mock framework pretty invaluable to make sure that
>> processors work
>> right before trying to test in a real flow. That being said, we are often
>> finding ways to improve
>> the mock framework, especially things like detecting "bad practices". So
>> if you hit something that you think
>> we could add to the mock framework to help catching these things early on,
>> please share so we
>> can make it better!
>>
>> Thanks
>> -Mark
>>
>> ----------------------------------------
>>> Date: Wed, 8 Jul 2015 10:46:53 -0600
>>> Subject: Re: Problem with Custom Processor Incorrectly Routing FlowFile
>>> From: jeremiah.adams@pearson.com
>>> To: dev@nifi.incubator.apache.org
>>>
>>> Thanks Mark, that got me in the right direction. Silly error on my part.
>>>
>>> *Jeremiah Adams*
>>>
>>> Senior Software Developer
>>> Pearson
>>>
>>> 2154 East Commons Ave.
>>> Suite 400
>>> Centennial, CO 80122
>>>
>>>
>>> Always Learning
>>> Learn more at www.pearson.com
>>>
>>> On Wed, Jul 8, 2015 at 10:24 AM, Mark Payne <ma...@hotmail.com>
>> wrote:
>>>
>>>> Jeremiah,
>>>>
>>>> That should be fine. I would recommend you modify the bootstrap.conf
>> file
>>>> in $NIFI_HOME/conf and uncomment this line:
>>>>
>>>>
>>>>
>> #java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000
>>>>
>>>> That will then allow you to attach a remote debugger (i.e., eclipse or
>>>> whatever IDE you use) and then you can set a breakpoint and understand
>>>> what's going on.
>>>>
>>>> Thanks
>>>> -Mark
>>>>
>>>> ----------------------------------------
>>>>> Date: Wed, 8 Jul 2015 10:14:02 -0600
>>>>> Subject: Problem with Custom Processor Incorrectly Routing FlowFile
>>>>> From: jeremiah.adams@pearson.com
>>>>> To: dev@nifi.incubator.apache.org
>>>>>
>>>>> I am implementing a custom processor based on JSON messages.
>>>>>
>>>>> I have unit tests around the processor. These verify the expected
>>>>> functionality and are similar to:
>>>>>
>>>>> ClassLoader loader = Thread.currentThread().getContextClassLoader();
>>>>> InputStream inputStream =
>>>>> loader.getResourceAsStream("matched.json");
>>>>> testRunner.enqueue(inputStream);
>>>>> testRunner.run();
>>>>> testRunner.assertTransferCount(JsonRouteProcessor.REL_MATCHED, 1);
>>>>> testRunner.assertQueueEmpty();
>>>>>
>>>>>
>>>>> Once I drop my processor into the NiFi server and put it in my flow,
>> this
>>>>> processor does not route messages as expected and as verified in the
>> unit
>>>>> tests. I can't see into the logic in the logs, just that this processor
>>>>> routed the message to the wrong downstream processor. I've even taken
>> the
>>>>> messages the processor rejects in my NiFi flow and ran them through my
>>>> unit
>>>>> tests - they behave as expected.
>>>>>
>>>>> Do I have a misunderstanding of the correct way to unit test the
>>>> onTrigger
>>>>> method for my processor?
>>>>>
>>>>>
>>>>> *Jeremiah Adams*
>>>>>
>>>>> Senior Software Developer
>>>>> Pearson
>>>>>
>>>>> 2154 East Commons Ave.
>>>>> Suite 400
>>>>> Centennial, CO 80122
>>>>>
>>>>>
>>>>> Always Learning
>>>>> Learn more at www.pearson.com
>>>>
>>
Re: Problem with Custom Processor Incorrectly Routing FlowFile
Posted by "Adams, Jeremiah" <je...@pearson.com>.
I was glad to see the mocks. When I first saw the stubs for the processor I
was wondering how I was going to test it. After I get this first POC out
I'll slowdown and back up through documentation and pass along any
suggestions, etc. I tend to just dive into things and see what shakes out.
Ease of use is big win for me. It is a good sign if I can generally "just
figure things out".
*Jeremiah Adams*
Senior Software Developer
Pearson
2154 East Commons Ave.
Suite 400
Centennial, CO 80122
Always Learning
Learn more at www.pearson.com
On Wed, Jul 8, 2015 at 11:05 AM, Mark Payne <ma...@hotmail.com> wrote:
> Great, glad I could help.
>
> I've found the mock framework pretty invaluable to make sure that
> processors work
> right before trying to test in a real flow. That being said, we are often
> finding ways to improve
> the mock framework, especially things like detecting "bad practices". So
> if you hit something that you think
> we could add to the mock framework to help catching these things early on,
> please share so we
> can make it better!
>
> Thanks
> -Mark
>
> ----------------------------------------
> > Date: Wed, 8 Jul 2015 10:46:53 -0600
> > Subject: Re: Problem with Custom Processor Incorrectly Routing FlowFile
> > From: jeremiah.adams@pearson.com
> > To: dev@nifi.incubator.apache.org
> >
> > Thanks Mark, that got me in the right direction. Silly error on my part.
> >
> > *Jeremiah Adams*
> >
> > Senior Software Developer
> > Pearson
> >
> > 2154 East Commons Ave.
> > Suite 400
> > Centennial, CO 80122
> >
> >
> > Always Learning
> > Learn more at www.pearson.com
> >
> > On Wed, Jul 8, 2015 at 10:24 AM, Mark Payne <ma...@hotmail.com>
> wrote:
> >
> >> Jeremiah,
> >>
> >> That should be fine. I would recommend you modify the bootstrap.conf
> file
> >> in $NIFI_HOME/conf and uncomment this line:
> >>
> >>
> >>
> #java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000
> >>
> >> That will then allow you to attach a remote debugger (i.e., eclipse or
> >> whatever IDE you use) and then you can set a breakpoint and understand
> >> what's going on.
> >>
> >> Thanks
> >> -Mark
> >>
> >> ----------------------------------------
> >>> Date: Wed, 8 Jul 2015 10:14:02 -0600
> >>> Subject: Problem with Custom Processor Incorrectly Routing FlowFile
> >>> From: jeremiah.adams@pearson.com
> >>> To: dev@nifi.incubator.apache.org
> >>>
> >>> I am implementing a custom processor based on JSON messages.
> >>>
> >>> I have unit tests around the processor. These verify the expected
> >>> functionality and are similar to:
> >>>
> >>> ClassLoader loader = Thread.currentThread().getContextClassLoader();
> >>> InputStream inputStream =
> >>> loader.getResourceAsStream("matched.json");
> >>> testRunner.enqueue(inputStream);
> >>> testRunner.run();
> >>> testRunner.assertTransferCount(JsonRouteProcessor.REL_MATCHED, 1);
> >>> testRunner.assertQueueEmpty();
> >>>
> >>>
> >>> Once I drop my processor into the NiFi server and put it in my flow,
> this
> >>> processor does not route messages as expected and as verified in the
> unit
> >>> tests. I can't see into the logic in the logs, just that this processor
> >>> routed the message to the wrong downstream processor. I've even taken
> the
> >>> messages the processor rejects in my NiFi flow and ran them through my
> >> unit
> >>> tests - they behave as expected.
> >>>
> >>> Do I have a misunderstanding of the correct way to unit test the
> >> onTrigger
> >>> method for my processor?
> >>>
> >>>
> >>> *Jeremiah Adams*
> >>>
> >>> Senior Software Developer
> >>> Pearson
> >>>
> >>> 2154 East Commons Ave.
> >>> Suite 400
> >>> Centennial, CO 80122
> >>>
> >>>
> >>> Always Learning
> >>> Learn more at www.pearson.com
> >>
>
RE: Problem with Custom Processor Incorrectly Routing FlowFile
Posted by Mark Payne <ma...@hotmail.com>.
Great, glad I could help.
I've found the mock framework pretty invaluable to make sure that processors work
right before trying to test in a real flow. That being said, we are often finding ways to improve
the mock framework, especially things like detecting "bad practices". So if you hit something that you think
we could add to the mock framework to help catching these things early on, please share so we
can make it better!
Thanks
-Mark
----------------------------------------
> Date: Wed, 8 Jul 2015 10:46:53 -0600
> Subject: Re: Problem with Custom Processor Incorrectly Routing FlowFile
> From: jeremiah.adams@pearson.com
> To: dev@nifi.incubator.apache.org
>
> Thanks Mark, that got me in the right direction. Silly error on my part.
>
> *Jeremiah Adams*
>
> Senior Software Developer
> Pearson
>
> 2154 East Commons Ave.
> Suite 400
> Centennial, CO 80122
>
>
> Always Learning
> Learn more at www.pearson.com
>
> On Wed, Jul 8, 2015 at 10:24 AM, Mark Payne <ma...@hotmail.com> wrote:
>
>> Jeremiah,
>>
>> That should be fine. I would recommend you modify the bootstrap.conf file
>> in $NIFI_HOME/conf and uncomment this line:
>>
>>
>> #java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000
>>
>> That will then allow you to attach a remote debugger (i.e., eclipse or
>> whatever IDE you use) and then you can set a breakpoint and understand
>> what's going on.
>>
>> Thanks
>> -Mark
>>
>> ----------------------------------------
>>> Date: Wed, 8 Jul 2015 10:14:02 -0600
>>> Subject: Problem with Custom Processor Incorrectly Routing FlowFile
>>> From: jeremiah.adams@pearson.com
>>> To: dev@nifi.incubator.apache.org
>>>
>>> I am implementing a custom processor based on JSON messages.
>>>
>>> I have unit tests around the processor. These verify the expected
>>> functionality and are similar to:
>>>
>>> ClassLoader loader = Thread.currentThread().getContextClassLoader();
>>> InputStream inputStream =
>>> loader.getResourceAsStream("matched.json");
>>> testRunner.enqueue(inputStream);
>>> testRunner.run();
>>> testRunner.assertTransferCount(JsonRouteProcessor.REL_MATCHED, 1);
>>> testRunner.assertQueueEmpty();
>>>
>>>
>>> Once I drop my processor into the NiFi server and put it in my flow, this
>>> processor does not route messages as expected and as verified in the unit
>>> tests. I can't see into the logic in the logs, just that this processor
>>> routed the message to the wrong downstream processor. I've even taken the
>>> messages the processor rejects in my NiFi flow and ran them through my
>> unit
>>> tests - they behave as expected.
>>>
>>> Do I have a misunderstanding of the correct way to unit test the
>> onTrigger
>>> method for my processor?
>>>
>>>
>>> *Jeremiah Adams*
>>>
>>> Senior Software Developer
>>> Pearson
>>>
>>> 2154 East Commons Ave.
>>> Suite 400
>>> Centennial, CO 80122
>>>
>>>
>>> Always Learning
>>> Learn more at www.pearson.com
>>
Re: Problem with Custom Processor Incorrectly Routing FlowFile
Posted by "Adams, Jeremiah" <je...@pearson.com>.
Thanks Mark, that got me in the right direction. Silly error on my part.
*Jeremiah Adams*
Senior Software Developer
Pearson
2154 East Commons Ave.
Suite 400
Centennial, CO 80122
Always Learning
Learn more at www.pearson.com
On Wed, Jul 8, 2015 at 10:24 AM, Mark Payne <ma...@hotmail.com> wrote:
> Jeremiah,
>
> That should be fine. I would recommend you modify the bootstrap.conf file
> in $NIFI_HOME/conf and uncomment this line:
>
>
> #java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000
>
> That will then allow you to attach a remote debugger (i.e., eclipse or
> whatever IDE you use) and then you can set a breakpoint and understand
> what's going on.
>
> Thanks
> -Mark
>
> ----------------------------------------
> > Date: Wed, 8 Jul 2015 10:14:02 -0600
> > Subject: Problem with Custom Processor Incorrectly Routing FlowFile
> > From: jeremiah.adams@pearson.com
> > To: dev@nifi.incubator.apache.org
> >
> > I am implementing a custom processor based on JSON messages.
> >
> > I have unit tests around the processor. These verify the expected
> > functionality and are similar to:
> >
> > ClassLoader loader = Thread.currentThread().getContextClassLoader();
> > InputStream inputStream =
> > loader.getResourceAsStream("matched.json");
> > testRunner.enqueue(inputStream);
> > testRunner.run();
> > testRunner.assertTransferCount(JsonRouteProcessor.REL_MATCHED, 1);
> > testRunner.assertQueueEmpty();
> >
> >
> > Once I drop my processor into the NiFi server and put it in my flow, this
> > processor does not route messages as expected and as verified in the unit
> > tests. I can't see into the logic in the logs, just that this processor
> > routed the message to the wrong downstream processor. I've even taken the
> > messages the processor rejects in my NiFi flow and ran them through my
> unit
> > tests - they behave as expected.
> >
> > Do I have a misunderstanding of the correct way to unit test the
> onTrigger
> > method for my processor?
> >
> >
> > *Jeremiah Adams*
> >
> > Senior Software Developer
> > Pearson
> >
> > 2154 East Commons Ave.
> > Suite 400
> > Centennial, CO 80122
> >
> >
> > Always Learning
> > Learn more at www.pearson.com
>
RE: Problem with Custom Processor Incorrectly Routing FlowFile
Posted by Mark Payne <ma...@hotmail.com>.
Jeremiah,
That should be fine. I would recommend you modify the bootstrap.conf file in $NIFI_HOME/conf and uncomment this line:
#java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000
That will then allow you to attach a remote debugger (i.e., eclipse or whatever IDE you use) and then you can set a breakpoint and understand what's going on.
Thanks
-Mark
----------------------------------------
> Date: Wed, 8 Jul 2015 10:14:02 -0600
> Subject: Problem with Custom Processor Incorrectly Routing FlowFile
> From: jeremiah.adams@pearson.com
> To: dev@nifi.incubator.apache.org
>
> I am implementing a custom processor based on JSON messages.
>
> I have unit tests around the processor. These verify the expected
> functionality and are similar to:
>
> ClassLoader loader = Thread.currentThread().getContextClassLoader();
> InputStream inputStream =
> loader.getResourceAsStream("matched.json");
> testRunner.enqueue(inputStream);
> testRunner.run();
> testRunner.assertTransferCount(JsonRouteProcessor.REL_MATCHED, 1);
> testRunner.assertQueueEmpty();
>
>
> Once I drop my processor into the NiFi server and put it in my flow, this
> processor does not route messages as expected and as verified in the unit
> tests. I can't see into the logic in the logs, just that this processor
> routed the message to the wrong downstream processor. I've even taken the
> messages the processor rejects in my NiFi flow and ran them through my unit
> tests - they behave as expected.
>
> Do I have a misunderstanding of the correct way to unit test the onTrigger
> method for my processor?
>
>
> *Jeremiah Adams*
>
> Senior Software Developer
> Pearson
>
> 2154 East Commons Ave.
> Suite 400
> Centennial, CO 80122
>
>
> Always Learning
> Learn more at www.pearson.com