You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Andrew Hulbert <ah...@ccri.com> on 2015/06/05 20:23:59 UTC
Controller Service vs Processor Properties
Hi,
I was attempted to add web proxy support to the GetHttp and PostHttp
processors. Do you think its best to add a ControllerService similar to
the SSLContextService or just make two properties on each processor
named "proxyHost" and "proxyPort"
Here's the controller service version I was thinking about but it seems
like overkill...not sure what the bar was for services.
https://github.com/jahhulbert-ccri/incubator-nifi/tree/http_proxy
It's also going to be super-fun to test proxy support.
Thanks,
Andrew
Re: Controller Service vs Processor Properties
Posted by Andrew Hulbert <ah...@ccri.com>.
Dan,
Thanks for the feedback. I ended up refactoring to make proxy host and
proxy port properties of the GetHTTP and PostHTTP processors and put it
up for PR here:
https://github.com/apache/incubator-nifi/pull/67
It was certainly simpler than the controller service implementation that
I modeled after the SSLContextService. I actually have a Nifi Instance
where I have many Get and Post HTTP processors that are all sitting
behind a web proxy which is why I was originally thinking about the
service. All but three use the proxy.
Thanks again for the feedback,
Andrew
On 06/15/2015 08:25 PM, Dan Bress wrote:
> Andrew,
> I agree with Mark that these things would be better captured as Processor properties than as a ControllerService. I'll also try and elaborate on his comment "the tradeoff is more art than science" with my own thinking on the subject.
>
> Generally I think ControllerServices are useful when you want to put "stuff" in one place.
>
> For the ControllerServices contained in Apache Nifi, this "stuff" usually takes one of three forms.
>
> 1) You have several properties that all go together and get used in tons of places. SSLContextService is a great example of this. There are 6 properties that are required to configure an SSL Context. Rather than make every processor that wants to use SSL define these 6 properties, you put it in a controller service, configure it there(once), and let the processors refer to the service/SSLContext via one configuration parameter. This makes both Processor developers and NiFi users life simpler.
>
> 2) You have a large dataset you want to load into memory that would be used in several places. Lets say you are writing some kind of enrichment processor, where you have a huge table of key/value pairs. Putting these key/value pairs in one controller service lets them be loaded once and used by many processors.
>
> 3) You have state you want to share across multiple processors. Examples of this are the DistributedMapCache, and the HttpContextMap . DistributedMapCache lets you put arbitrary key/value pairs in one processor and reference them in another. HttpContextMap lets you store an HTTPContextRequest and Response in one part of the flow(when you are reading the request), and get them later on(when you are populating the response).
>
> Hope this clarifies when to use a ControllerService.
>
> Dan Bress
> Software Engineer
> ONYX Consulting Services
>
> ________________________________________
> From: Mark Payne <ma...@hotmail.com>
> Sent: Friday, June 5, 2015 3:10 PM
> To: dev@nifi.incubator.apache.org
> Subject: RE: Controller Service vs Processor Properties
>
> Andrew,
>
> That tradeoff is more art than science. Personally, I would go with adding them as optional properties.
>
> Thanks
> -Mark
>
> ----------------------------------------
>> Date: Fri, 5 Jun 2015 14:52:56 -0400
>> From: ahulbert@ccri.com
>> To: dev@nifi.incubator.apache.org
>> Subject: Re: Controller Service vs Processor Properties
>>
>> I also just noticed this ticket that is related:
>>
>> https://issues.apache.org/jira/browse/NIFI-332
>>
>> If the service is the way to go I can adapt that processor as well.
>>
>> On 06/05/2015 02:23 PM, Andrew Hulbert wrote:
>>> Hi,
>>>
>>> I was attempted to add web proxy support to the GetHttp and PostHttp
>>> processors. Do you think its best to add a ControllerService similar
>>> to the SSLContextService or just make two properties on each processor
>>> named "proxyHost" and "proxyPort"
>>>
>>> Here's the controller service version I was thinking about but it
>>> seems like overkill...not sure what the bar was for services.
>>>
>>> https://github.com/jahhulbert-ccri/incubator-nifi/tree/http_proxy
>>>
>>> It's also going to be super-fun to test proxy support.
>>>
>>> Thanks,
>>>
>>> Andrew
Re: Controller Service vs Processor Properties
Posted by Dan Bress <db...@onyxconsults.com>.
Andrew,
I agree with Mark that these things would be better captured as Processor properties than as a ControllerService. I'll also try and elaborate on his comment "the tradeoff is more art than science" with my own thinking on the subject.
Generally I think ControllerServices are useful when you want to put "stuff" in one place.
For the ControllerServices contained in Apache Nifi, this "stuff" usually takes one of three forms.
1) You have several properties that all go together and get used in tons of places. SSLContextService is a great example of this. There are 6 properties that are required to configure an SSL Context. Rather than make every processor that wants to use SSL define these 6 properties, you put it in a controller service, configure it there(once), and let the processors refer to the service/SSLContext via one configuration parameter. This makes both Processor developers and NiFi users life simpler.
2) You have a large dataset you want to load into memory that would be used in several places. Lets say you are writing some kind of enrichment processor, where you have a huge table of key/value pairs. Putting these key/value pairs in one controller service lets them be loaded once and used by many processors.
3) You have state you want to share across multiple processors. Examples of this are the DistributedMapCache, and the HttpContextMap . DistributedMapCache lets you put arbitrary key/value pairs in one processor and reference them in another. HttpContextMap lets you store an HTTPContextRequest and Response in one part of the flow(when you are reading the request), and get them later on(when you are populating the response).
Hope this clarifies when to use a ControllerService.
Dan Bress
Software Engineer
ONYX Consulting Services
________________________________________
From: Mark Payne <ma...@hotmail.com>
Sent: Friday, June 5, 2015 3:10 PM
To: dev@nifi.incubator.apache.org
Subject: RE: Controller Service vs Processor Properties
Andrew,
That tradeoff is more art than science. Personally, I would go with adding them as optional properties.
Thanks
-Mark
----------------------------------------
> Date: Fri, 5 Jun 2015 14:52:56 -0400
> From: ahulbert@ccri.com
> To: dev@nifi.incubator.apache.org
> Subject: Re: Controller Service vs Processor Properties
>
> I also just noticed this ticket that is related:
>
> https://issues.apache.org/jira/browse/NIFI-332
>
> If the service is the way to go I can adapt that processor as well.
>
> On 06/05/2015 02:23 PM, Andrew Hulbert wrote:
>> Hi,
>>
>> I was attempted to add web proxy support to the GetHttp and PostHttp
>> processors. Do you think its best to add a ControllerService similar
>> to the SSLContextService or just make two properties on each processor
>> named "proxyHost" and "proxyPort"
>>
>> Here's the controller service version I was thinking about but it
>> seems like overkill...not sure what the bar was for services.
>>
>> https://github.com/jahhulbert-ccri/incubator-nifi/tree/http_proxy
>>
>> It's also going to be super-fun to test proxy support.
>>
>> Thanks,
>>
>> Andrew
>
RE: Controller Service vs Processor Properties
Posted by Mark Payne <ma...@hotmail.com>.
Andrew,
That tradeoff is more art than science. Personally, I would go with adding them as optional properties.
Thanks
-Mark
----------------------------------------
> Date: Fri, 5 Jun 2015 14:52:56 -0400
> From: ahulbert@ccri.com
> To: dev@nifi.incubator.apache.org
> Subject: Re: Controller Service vs Processor Properties
>
> I also just noticed this ticket that is related:
>
> https://issues.apache.org/jira/browse/NIFI-332
>
> If the service is the way to go I can adapt that processor as well.
>
> On 06/05/2015 02:23 PM, Andrew Hulbert wrote:
>> Hi,
>>
>> I was attempted to add web proxy support to the GetHttp and PostHttp
>> processors. Do you think its best to add a ControllerService similar
>> to the SSLContextService or just make two properties on each processor
>> named "proxyHost" and "proxyPort"
>>
>> Here's the controller service version I was thinking about but it
>> seems like overkill...not sure what the bar was for services.
>>
>> https://github.com/jahhulbert-ccri/incubator-nifi/tree/http_proxy
>>
>> It's also going to be super-fun to test proxy support.
>>
>> Thanks,
>>
>> Andrew
>
Re: Controller Service vs Processor Properties
Posted by Andrew Hulbert <ah...@ccri.com>.
I also just noticed this ticket that is related:
https://issues.apache.org/jira/browse/NIFI-332
If the service is the way to go I can adapt that processor as well.
On 06/05/2015 02:23 PM, Andrew Hulbert wrote:
> Hi,
>
> I was attempted to add web proxy support to the GetHttp and PostHttp
> processors. Do you think its best to add a ControllerService similar
> to the SSLContextService or just make two properties on each processor
> named "proxyHost" and "proxyPort"
>
> Here's the controller service version I was thinking about but it
> seems like overkill...not sure what the bar was for services.
>
> https://github.com/jahhulbert-ccri/incubator-nifi/tree/http_proxy
>
> It's also going to be super-fun to test proxy support.
>
> Thanks,
>
> Andrew