You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Frank Martínez <mn...@gmail.com> on 2016/01/22 04:33:54 UTC
Processor Signaling
Hi Guys,
I want to develop a custom Router processor, But I want that my processor
logic depends on external parameters than can be hot changed. So I have to
send some signals to the processor while it is running, so it will change
its internal routing logic. Is it possible to do that with apache nifi?
Thanks,
Frank Martínez.
--
Frank D. Martínez M.
Re: Processor Signaling
Posted by Frank Martínez <mn...@gmail.com>.
Mark,
Thanks for your suggestions. I will do some experiments....
Cheers,
Frank.
On Fri, Jan 22, 2016 at 8:43 AM, Mark Payne <ma...@hotmail.com> wrote:
> Frank,
>
> The approach that you laid out here would certainly work. However, I feel
> that it may be a bit heavy-handed. If the idea is
> just to reload some information from a local file, then I would recommend
> that you take a look at the ScanContent / ScanAttribute
> Processors. They do something similar. They are configured with a file
> that is the Dictionary of Terms to route on. Then, they
> periodically check if the file has been updated. If so, they will reload
> the information, otherwise they will just keep what they
> already have loaded.
>
> They make use of SynchronousFileWatcher class that exists in the
> nifi-utils module, which helps to make this simpler.
>
> Please feel free to reach out with any additional questions, or if this
> doesn't seem like the right fit for you.
>
> Thanks
> -Mark
>
> On Jan 21, 2016, at 11:08 PM, Frank Martínez <mn...@gmail.com> wrote:
>
> Hi Joe,
>
> Thanks for your response.
>
> So the way is to get the instruction set from external source in onTrigger
> method each time? Or is there any way to push immediate commands or
> messages to the Procesor?
>
> Maybe I can create a custom ControllerService that provides and caches the
> current instruction set, then I can create a Custom processor that updates
> the cache of the ControllerService, then use GetFile to poll a directory
> where I can put files (commands) to be executed.
>
>
>
> get current config
> +--------------------> [config source endpoint]
> |
> +-------> ConfigLoaderProcessor
> | |
> GetFileProcessor | update config cache
> | \/
> Directory ConfigControllerService
> | ^
> CommandFile |
> .
> . get current config
> |
> .......> RouterProcessor .....>
>
>
> What do you think?
>
> Thanks
>
>
> On Thu, Jan 21, 2016 at 10:38 PM, Joe Witt <jo...@gmail.com> wrote:
>
>> Frank,
>>
>> Absolutely. Have seen several of these of various types of the years.
>> It usually works best if the processor itself calls out/polls some
>> endpoint to get the latest instruction set. This approach tends to
>> scale better, means clustered systems work smoothly as well, and it is
>> nice from a reliability perspective. You can still choose to push new
>> configuration information to the processor but keep in mind you cannot
>> change its properties while it is running.
>>
>> Feel free to fire more questions as you dig deeper into it.
>>
>> Thanks
>> Joe
>>
>> On Thu, Jan 21, 2016 at 10:33 PM, Frank Martínez <mn...@gmail.com>
>> wrote:
>> > Hi Guys,
>> >
>> > I want to develop a custom Router processor, But I want that my
>> processor
>> > logic depends on external parameters than can be hot changed. So I have
>> to
>> > send some signals to the processor while it is running, so it will
>> change
>> > its internal routing logic. Is it possible to do that with apache nifi?
>> >
>> > Thanks,
>> >
>> > Frank Martínez.
>> >
>> > --
>> > Frank D. Martínez M.
>>
>
>
>
> --
> Frank D. Martínez M.
>
>
>
--
Frank D. Martínez M.
Re: Processor Signaling
Posted by Mark Payne <ma...@hotmail.com>.
Frank,
The approach that you laid out here would certainly work. However, I feel that it may be a bit heavy-handed. If the idea is
just to reload some information from a local file, then I would recommend that you take a look at the ScanContent / ScanAttribute
Processors. They do something similar. They are configured with a file that is the Dictionary of Terms to route on. Then, they
periodically check if the file has been updated. If so, they will reload the information, otherwise they will just keep what they
already have loaded.
They make use of SynchronousFileWatcher class that exists in the nifi-utils module, which helps to make this simpler.
Please feel free to reach out with any additional questions, or if this doesn't seem like the right fit for you.
Thanks
-Mark
> On Jan 21, 2016, at 11:08 PM, Frank Martínez <mn...@gmail.com> wrote:
>
> Hi Joe,
>
> Thanks for your response.
>
> So the way is to get the instruction set from external source in onTrigger method each time? Or is there any way to push immediate commands or messages to the Procesor?
>
> Maybe I can create a custom ControllerService that provides and caches the current instruction set, then I can create a Custom processor that updates the cache of the ControllerService, then use GetFile to poll a directory where I can put files (commands) to be executed.
>
>
>
> get current config
> +--------------------> [config source endpoint]
> |
> +-------> ConfigLoaderProcessor
> | |
> GetFileProcessor | update config cache
> | \/
> Directory ConfigControllerService
> | ^
> CommandFile |
> .
> . get current config
> |
> .......> RouterProcessor .....>
>
>
> What do you think?
>
> Thanks
>
>
> On Thu, Jan 21, 2016 at 10:38 PM, Joe Witt <joe.witt@gmail.com <ma...@gmail.com>> wrote:
> Frank,
>
> Absolutely. Have seen several of these of various types of the years.
> It usually works best if the processor itself calls out/polls some
> endpoint to get the latest instruction set. This approach tends to
> scale better, means clustered systems work smoothly as well, and it is
> nice from a reliability perspective. You can still choose to push new
> configuration information to the processor but keep in mind you cannot
> change its properties while it is running.
>
> Feel free to fire more questions as you dig deeper into it.
>
> Thanks
> Joe
>
> On Thu, Jan 21, 2016 at 10:33 PM, Frank Martínez <mnesarco@gmail.com <ma...@gmail.com>> wrote:
> > Hi Guys,
> >
> > I want to develop a custom Router processor, But I want that my processor
> > logic depends on external parameters than can be hot changed. So I have to
> > send some signals to the processor while it is running, so it will change
> > its internal routing logic. Is it possible to do that with apache nifi?
> >
> > Thanks,
> >
> > Frank Martínez.
> >
> > --
> > Frank D. Martínez M.
>
>
>
> --
> Frank D. Martínez M.
Re: Processor Signaling
Posted by Frank Martínez <mn...@gmail.com>.
Hi Joe,
Thanks for your response.
So the way is to get the instruction set from external source in onTrigger
method each time? Or is there any way to push immediate commands or
messages to the Procesor?
Maybe I can create a custom ControllerService that provides and caches the
current instruction set, then I can create a Custom processor that updates
the cache of the ControllerService, then use GetFile to poll a directory
where I can put files (commands) to be executed.
get current config
+--------------------> [config source endpoint]
|
+-------> ConfigLoaderProcessor
| |
GetFileProcessor | update config cache
| \/
Directory ConfigControllerService
| ^
CommandFile |
.
. get current config
|
.......> RouterProcessor .....>
What do you think?
Thanks
On Thu, Jan 21, 2016 at 10:38 PM, Joe Witt <jo...@gmail.com> wrote:
> Frank,
>
> Absolutely. Have seen several of these of various types of the years.
> It usually works best if the processor itself calls out/polls some
> endpoint to get the latest instruction set. This approach tends to
> scale better, means clustered systems work smoothly as well, and it is
> nice from a reliability perspective. You can still choose to push new
> configuration information to the processor but keep in mind you cannot
> change its properties while it is running.
>
> Feel free to fire more questions as you dig deeper into it.
>
> Thanks
> Joe
>
> On Thu, Jan 21, 2016 at 10:33 PM, Frank Martínez <mn...@gmail.com>
> wrote:
> > Hi Guys,
> >
> > I want to develop a custom Router processor, But I want that my processor
> > logic depends on external parameters than can be hot changed. So I have
> to
> > send some signals to the processor while it is running, so it will change
> > its internal routing logic. Is it possible to do that with apache nifi?
> >
> > Thanks,
> >
> > Frank Martínez.
> >
> > --
> > Frank D. Martínez M.
>
--
Frank D. Martínez M.
Re: Processor Signaling
Posted by Joe Witt <jo...@gmail.com>.
Frank,
Absolutely. Have seen several of these of various types of the years.
It usually works best if the processor itself calls out/polls some
endpoint to get the latest instruction set. This approach tends to
scale better, means clustered systems work smoothly as well, and it is
nice from a reliability perspective. You can still choose to push new
configuration information to the processor but keep in mind you cannot
change its properties while it is running.
Feel free to fire more questions as you dig deeper into it.
Thanks
Joe
On Thu, Jan 21, 2016 at 10:33 PM, Frank Martínez <mn...@gmail.com> wrote:
> Hi Guys,
>
> I want to develop a custom Router processor, But I want that my processor
> logic depends on external parameters than can be hot changed. So I have to
> send some signals to the processor while it is running, so it will change
> its internal routing logic. Is it possible to do that with apache nifi?
>
> Thanks,
>
> Frank Martínez.
>
> --
> Frank D. Martínez M.