You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Eduardo Fontes <ed...@gmail.com> on 2020/03/01 12:00:15 UTC
Re: Queue Prioritization
Hi Midhun,
You can use DistributedMapCacheServer which starts a cache service within
NiFi's process, but I not recommend in production. Instead you could use
RedisConnectionPoolService.
To use solution Wait/Notify the NiFi must know the number of flowfiles to
process in one queue before start another queue. If it is a batch/on demand
flow I think it's easier. If it is a continuos/streamming flow you must
define a window, by number of flowfiles or by time, to switch between
queues.
Please let me know what is your case.
On Fri, Feb 28, 2020 at 2:39 PM Midhun Mohan <mi...@esginc.us> wrote:
> I think wait and notify can solve my problem. I checked the documents ,
> which is bit unclear to me. Can anyone help me?
>
> Just to know few things like whether we need to configure a cache server,
> if so which one is best
> And also getting confused on how to configure wait and notify signals
>
> On Fri, 28 Feb 2020, 10:59 pm Eduardo Fontes, <ed...@gmail.com>
> wrote:
>
> > Hi! Maybe you should try another approach, like Wait/Notify or configure
> > prioritizer just before the funnel, depending on your demand.
> >
> > On Fri, Feb 28, 2020 at 11:40 AM Midhun Mohan <mi...@esginc.us>
> > wrote:
> >
> > > Hi all,
> > > How to prioritize one queue before a funnel?
> > >
> > > I need to make sure one queue is emptied before another queue is taken
> > > over for executing, I tried with a priority attribute but it fails
> when a
> > > queue with less priority has messages before the queue with high
> > priority.
> > >
> > > --
> > >
> > >
> > > Regards,
> > > Midhun Mohan
> > >
> >
>
Re: Queue Prioritization
Posted by Endre Kovács <ek...@cloudera.com.INVALID>.
Please upload it to somewhere and link it here. ASF mailing lists are striping the attachments/images.
> On 2 Mar 2020, at 13:35, Midhun Mohan <mi...@esginc.us> wrote:
>
> Hi Eduardo,
> Can you please send the Image again, I am not able to open it
>
> On Mon, 2 Mar 2020 at 18:02, Eduardo Fontes <eduardo.fontes@gmail.com <ma...@gmail.com>>
> wrote:
>
>> Hi Midhun,
>>
>> I think this flow will do what you want.
>>
>> [image: image.png]
>>
>> Be advised that use of Wait/Notify will slow down the average throughput
>> of your flow because the aditional steps with the Cache.
>>
>>
>>
>>
>> On Sun, Mar 1, 2020 at 12:57 PM Midhun Mohan <mi...@esginc.us>
>> wrote:
>>
>>> So I would like to know your suggestions on how to implement wait and
>>> notify.
>>>
>>> In my case messages will flow one after another
>>>
>>> If a failure occurs I need to retry the failure before the orginal ones,
>>> and notify users if it fails again.
>>>
>>>
>>> On Sun, 1 Mar 2020, 6:31 pm Emanuel Oliveira, <em...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Invokehttp have 4 exits
>>>> - success
>>>> - error retry -> loop/count retries (updateattribite) -> if retries >
>>> max
>>>> then logmessage(error) flow stops here ff discarded Otherwise loop
>>> back to
>>>> invokehttp.
>>>> - error type 2 (bad password etc, )
>>>> - error type 3 (connectivity problems) you may want also to loop this
>>> one
>>>> with some delay , count retries and exit /log/stop after max.
>>>>
>>>> I just highly advise prior to invokehttp to have an updateattribute
>>>> deleting all attributes that don't start with some prefix.. to be sure
>>> your
>>>> ff looping doesnt keep adfing different errors, attributes wriiten by
>>>> invoke http.
>>>>
>>>> As good practice we prefix name all control attributes with
>>> flow_xxxxxxx so
>>>> our "cleanup," Updateattribute before Invokehttp have special regex
>>> sayng
>>>> "delete all attributes that don't start with flow_
>>>> (If you interested tomorrow when im back in office i csn share the
>>> regex a
>>>> bit tricky).
>>>>
>>>> Hope this helps, i just would love all processors to be as good as
>>>> invokehttp providing hard multipke exit points for specifics + all the
>>> nice
>>>> write attributes invokehttp adds that ptovide high details on errors or
>>>> success (atttibutes created/added by invokehttp to ff are prefixed with
>>>> invoke.xxxxx
>>>>
>>>> Cheers,
>>>> Emanuel O.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Sun, 1 Mar 2020, 12:42 Midhun Mohan, <mi...@esginc.us> wrote:
>>>>
>>>>> Hi my use case is
>>>>>
>>>>> When a invokehttp processor fails I need to stop the incoming message
>>> and
>>>>> retry the failed message 2 times and if it is success start the
>>> execution
>>>>> normally or if it fails again need to notify admins and stop the main
>>>>> invoke http processor.
>>>>>
>>>>>
>>>>>
>>>>> Hope you can get the scenario from this
>>>>>
>>>>> On Sun, 1 Mar 2020, 5:30 pm Eduardo Fontes, <eduardo.fontes@gmail.com
>>>>
>>>>> wrote:
>>>>>
>>>>>> Hi Midhun,
>>>>>>
>>>>>> You can use DistributedMapCacheServer which starts a cache service
>>>> within
>>>>>> NiFi's process, but I not recommend in production. Instead you could
>>>> use
>>>>>> RedisConnectionPoolService.
>>>>>>
>>>>>> To use solution Wait/Notify the NiFi must know the number of
>>> flowfiles
>>>> to
>>>>>> process in one queue before start another queue. If it is a batch/on
>>>>> demand
>>>>>> flow I think it's easier. If it is a continuos/streamming flow you
>>> must
>>>>>> define a window, by number of flowfiles or by time, to switch
>>> between
>>>>>> queues.
>>>>>>
>>>>>> Please let me know what is your case.
>>>>>>
>>>>>> On Fri, Feb 28, 2020 at 2:39 PM Midhun Mohan <
>>> midhun.mohan@esginc.us>
>>>>>> wrote:
>>>>>>
>>>>>>> I think wait and notify can solve my problem. I checked the
>>>> documents ,
>>>>>>> which is bit unclear to me. Can anyone help me?
>>>>>>>
>>>>>>> Just to know few things like whether we need to configure a cache
>>>>> server,
>>>>>>> if so which one is best
>>>>>>> And also getting confused on how to configure wait and notify
>>> signals
>>>>>>>
>>>>>>> On Fri, 28 Feb 2020, 10:59 pm Eduardo Fontes, <
>>>>> eduardo.fontes@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi! Maybe you should try another approach, like Wait/Notify or
>>>>>> configure
>>>>>>>> prioritizer just before the funnel, depending on your demand.
>>>>>>>>
>>>>>>>> On Fri, Feb 28, 2020 at 11:40 AM Midhun Mohan <
>>>>> midhun.mohan@esginc.us>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>> How to prioritize one queue before a funnel?
>>>>>>>>>
>>>>>>>>> I need to make sure one queue is emptied before another
>>> queue is
>>>>>> taken
>>>>>>>>> over for executing, I tried with a priority attribute but it
>>>> fails
>>>>>>> when a
>>>>>>>>> queue with less priority has messages before the queue with
>>> high
>>>>>>>> priority.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Midhun Mohan
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
> --
>
>
> Regards,
> Midhun Mohan
Re: Queue Prioritization
Posted by Eduardo Fontes <ed...@gmail.com>.
OK.
The "GenerateFlowFile (on demand)" is only to "open the door" for the first
time, next flowfile will release the next and so on, automatically.
You can define any name for the release signal. I use "test.release" in
this example. You have to use the same signal name in both Wait and Notify
processors and define a very long expiration duration in Wait to guarantee
flowfiles will be released only after all tries of failed flowfile.
Wait: https://pasteboard.co/IXf873U.png
Notify: https://pasteboard.co/IXf8svS.png
On Mon, Mar 2, 2020 at 10:09 AM Midhun Mohan <mi...@esginc.us> wrote:
> Hi Eduardo,
>
> I have created the same flow in NIFI, the only thing I am not having is
> GenerateONDemand.
> and also am confused about how to set up the release signal for
> notifying and wait. Can you help me with that?
>
> On Mon, 2 Mar 2020 at 18:26, Eduardo Fontes <ed...@gmail.com>
> wrote:
>
> > Sorry,
> >
> > Try this link: https://pasteboard.co/IXejthc.png
> >
> > On Mon, Mar 2, 2020 at 9:35 AM Midhun Mohan <mi...@esginc.us>
> > wrote:
> >
> > > Hi Eduardo,
> > > Can you please send the Image again, I am not able to open it
> > >
> > > On Mon, 2 Mar 2020 at 18:02, Eduardo Fontes <ed...@gmail.com>
> > > wrote:
> > >
> > > > Hi Midhun,
> > > >
> > > > I think this flow will do what you want.
> > > >
> > > > [image: image.png]
> > > >
> > > > Be advised that use of Wait/Notify will slow down the average
> > throughput
> > > > of your flow because the aditional steps with the Cache.
> > > >
> > > >
> > > >
> > > >
> > > > On Sun, Mar 1, 2020 at 12:57 PM Midhun Mohan <midhun.mohan@esginc.us
> >
> > > > wrote:
> > > >
> > > >> So I would like to know your suggestions on how to implement wait
> and
> > > >> notify.
> > > >>
> > > >> In my case messages will flow one after another
> > > >>
> > > >> If a failure occurs I need to retry the failure before the orginal
> > ones,
> > > >> and notify users if it fails again.
> > > >>
> > > >>
> > > >> On Sun, 1 Mar 2020, 6:31 pm Emanuel Oliveira, <em...@gmail.com>
> > > wrote:
> > > >>
> > > >> > Hi,
> > > >> >
> > > >> > Invokehttp have 4 exits
> > > >> > - success
> > > >> > - error retry -> loop/count retries (updateattribite) -> if
> retries
> > >
> > > >> max
> > > >> > then logmessage(error) flow stops here ff discarded Otherwise
> loop
> > > >> back to
> > > >> > invokehttp.
> > > >> > - error type 2 (bad password etc, )
> > > >> > - error type 3 (connectivity problems) you may want also to loop
> > this
> > > >> one
> > > >> > with some delay , count retries and exit /log/stop after max.
> > > >> >
> > > >> > I just highly advise prior to invokehttp to have an
> updateattribute
> > > >> > deleting all attributes that don't start with some prefix.. to be
> > sure
> > > >> your
> > > >> > ff looping doesnt keep adfing different errors, attributes wriiten
> > by
> > > >> > invoke http.
> > > >> >
> > > >> > As good practice we prefix name all control attributes with
> > > >> flow_xxxxxxx so
> > > >> > our "cleanup," Updateattribute before Invokehttp have special
> regex
> > > >> sayng
> > > >> > "delete all attributes that don't start with flow_
> > > >> > (If you interested tomorrow when im back in office i csn share the
> > > >> regex a
> > > >> > bit tricky).
> > > >> >
> > > >> > Hope this helps, i just would love all processors to be as good as
> > > >> > invokehttp providing hard multipke exit points for specifics + all
> > the
> > > >> nice
> > > >> > write attributes invokehttp adds that ptovide high details on
> errors
> > > or
> > > >> > success (atttibutes created/added by invokehttp to ff are prefixed
> > > with
> > > >> > invoke.xxxxx
> > > >> >
> > > >> > Cheers,
> > > >> > Emanuel O.
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Sun, 1 Mar 2020, 12:42 Midhun Mohan, <mi...@esginc.us>
> > > wrote:
> > > >> >
> > > >> > > Hi my use case is
> > > >> > >
> > > >> > > When a invokehttp processor fails I need to stop the incoming
> > > message
> > > >> and
> > > >> > > retry the failed message 2 times and if it is success start the
> > > >> execution
> > > >> > > normally or if it fails again need to notify admins and stop the
> > > main
> > > >> > > invoke http processor.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > Hope you can get the scenario from this
> > > >> > >
> > > >> > > On Sun, 1 Mar 2020, 5:30 pm Eduardo Fontes, <
> > > eduardo.fontes@gmail.com
> > > >> >
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Hi Midhun,
> > > >> > > >
> > > >> > > > You can use DistributedMapCacheServer which starts a cache
> > service
> > > >> > within
> > > >> > > > NiFi's process, but I not recommend in production. Instead you
> > > could
> > > >> > use
> > > >> > > > RedisConnectionPoolService.
> > > >> > > >
> > > >> > > > To use solution Wait/Notify the NiFi must know the number of
> > > >> flowfiles
> > > >> > to
> > > >> > > > process in one queue before start another queue. If it is a
> > > batch/on
> > > >> > > demand
> > > >> > > > flow I think it's easier. If it is a continuos/streamming flow
> > you
> > > >> must
> > > >> > > > define a window, by number of flowfiles or by time, to switch
> > > >> between
> > > >> > > > queues.
> > > >> > > >
> > > >> > > > Please let me know what is your case.
> > > >> > > >
> > > >> > > > On Fri, Feb 28, 2020 at 2:39 PM Midhun Mohan <
> > > >> midhun.mohan@esginc.us>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > I think wait and notify can solve my problem. I checked the
> > > >> > documents ,
> > > >> > > > > which is bit unclear to me. Can anyone help me?
> > > >> > > > >
> > > >> > > > > Just to know few things like whether we need to configure a
> > > cache
> > > >> > > server,
> > > >> > > > > if so which one is best
> > > >> > > > > And also getting confused on how to configure wait and
> notify
> > > >> signals
> > > >> > > > >
> > > >> > > > > On Fri, 28 Feb 2020, 10:59 pm Eduardo Fontes, <
> > > >> > > eduardo.fontes@gmail.com>
> > > >> > > > > wrote:
> > > >> > > > >
> > > >> > > > > > Hi! Maybe you should try another approach, like
> Wait/Notify
> > or
> > > >> > > > configure
> > > >> > > > > > prioritizer just before the funnel, depending on your
> > demand.
> > > >> > > > > >
> > > >> > > > > > On Fri, Feb 28, 2020 at 11:40 AM Midhun Mohan <
> > > >> > > midhun.mohan@esginc.us>
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > Hi all,
> > > >> > > > > > > How to prioritize one queue before a funnel?
> > > >> > > > > > >
> > > >> > > > > > > I need to make sure one queue is emptied before another
> > > >> queue is
> > > >> > > > taken
> > > >> > > > > > > over for executing, I tried with a priority attribute
> but
> > it
> > > >> > fails
> > > >> > > > > when a
> > > >> > > > > > > queue with less priority has messages before the queue
> > with
> > > >> high
> > > >> > > > > > priority.
> > > >> > > > > > >
> > > >> > > > > > > --
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > Regards,
> > > >> > > > > > > Midhun Mohan
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> > > --
> > >
> > >
> > > Regards,
> > > Midhun Mohan
> > >
> >
>
>
> --
>
>
> Regards,
> Midhun Mohan
>
Re: Queue Prioritization
Posted by Midhun Mohan <mi...@esginc.us>.
Hi Eduardo,
I have created the same flow in NIFI, the only thing I am not having is
GenerateONDemand.
and also am confused about how to set up the release signal for
notifying and wait. Can you help me with that?
On Mon, 2 Mar 2020 at 18:26, Eduardo Fontes <ed...@gmail.com>
wrote:
> Sorry,
>
> Try this link: https://pasteboard.co/IXejthc.png
>
> On Mon, Mar 2, 2020 at 9:35 AM Midhun Mohan <mi...@esginc.us>
> wrote:
>
> > Hi Eduardo,
> > Can you please send the Image again, I am not able to open it
> >
> > On Mon, 2 Mar 2020 at 18:02, Eduardo Fontes <ed...@gmail.com>
> > wrote:
> >
> > > Hi Midhun,
> > >
> > > I think this flow will do what you want.
> > >
> > > [image: image.png]
> > >
> > > Be advised that use of Wait/Notify will slow down the average
> throughput
> > > of your flow because the aditional steps with the Cache.
> > >
> > >
> > >
> > >
> > > On Sun, Mar 1, 2020 at 12:57 PM Midhun Mohan <mi...@esginc.us>
> > > wrote:
> > >
> > >> So I would like to know your suggestions on how to implement wait and
> > >> notify.
> > >>
> > >> In my case messages will flow one after another
> > >>
> > >> If a failure occurs I need to retry the failure before the orginal
> ones,
> > >> and notify users if it fails again.
> > >>
> > >>
> > >> On Sun, 1 Mar 2020, 6:31 pm Emanuel Oliveira, <em...@gmail.com>
> > wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > Invokehttp have 4 exits
> > >> > - success
> > >> > - error retry -> loop/count retries (updateattribite) -> if retries
> >
> > >> max
> > >> > then logmessage(error) flow stops here ff discarded Otherwise loop
> > >> back to
> > >> > invokehttp.
> > >> > - error type 2 (bad password etc, )
> > >> > - error type 3 (connectivity problems) you may want also to loop
> this
> > >> one
> > >> > with some delay , count retries and exit /log/stop after max.
> > >> >
> > >> > I just highly advise prior to invokehttp to have an updateattribute
> > >> > deleting all attributes that don't start with some prefix.. to be
> sure
> > >> your
> > >> > ff looping doesnt keep adfing different errors, attributes wriiten
> by
> > >> > invoke http.
> > >> >
> > >> > As good practice we prefix name all control attributes with
> > >> flow_xxxxxxx so
> > >> > our "cleanup," Updateattribute before Invokehttp have special regex
> > >> sayng
> > >> > "delete all attributes that don't start with flow_
> > >> > (If you interested tomorrow when im back in office i csn share the
> > >> regex a
> > >> > bit tricky).
> > >> >
> > >> > Hope this helps, i just would love all processors to be as good as
> > >> > invokehttp providing hard multipke exit points for specifics + all
> the
> > >> nice
> > >> > write attributes invokehttp adds that ptovide high details on errors
> > or
> > >> > success (atttibutes created/added by invokehttp to ff are prefixed
> > with
> > >> > invoke.xxxxx
> > >> >
> > >> > Cheers,
> > >> > Emanuel O.
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On Sun, 1 Mar 2020, 12:42 Midhun Mohan, <mi...@esginc.us>
> > wrote:
> > >> >
> > >> > > Hi my use case is
> > >> > >
> > >> > > When a invokehttp processor fails I need to stop the incoming
> > message
> > >> and
> > >> > > retry the failed message 2 times and if it is success start the
> > >> execution
> > >> > > normally or if it fails again need to notify admins and stop the
> > main
> > >> > > invoke http processor.
> > >> > >
> > >> > >
> > >> > >
> > >> > > Hope you can get the scenario from this
> > >> > >
> > >> > > On Sun, 1 Mar 2020, 5:30 pm Eduardo Fontes, <
> > eduardo.fontes@gmail.com
> > >> >
> > >> > > wrote:
> > >> > >
> > >> > > > Hi Midhun,
> > >> > > >
> > >> > > > You can use DistributedMapCacheServer which starts a cache
> service
> > >> > within
> > >> > > > NiFi's process, but I not recommend in production. Instead you
> > could
> > >> > use
> > >> > > > RedisConnectionPoolService.
> > >> > > >
> > >> > > > To use solution Wait/Notify the NiFi must know the number of
> > >> flowfiles
> > >> > to
> > >> > > > process in one queue before start another queue. If it is a
> > batch/on
> > >> > > demand
> > >> > > > flow I think it's easier. If it is a continuos/streamming flow
> you
> > >> must
> > >> > > > define a window, by number of flowfiles or by time, to switch
> > >> between
> > >> > > > queues.
> > >> > > >
> > >> > > > Please let me know what is your case.
> > >> > > >
> > >> > > > On Fri, Feb 28, 2020 at 2:39 PM Midhun Mohan <
> > >> midhun.mohan@esginc.us>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > I think wait and notify can solve my problem. I checked the
> > >> > documents ,
> > >> > > > > which is bit unclear to me. Can anyone help me?
> > >> > > > >
> > >> > > > > Just to know few things like whether we need to configure a
> > cache
> > >> > > server,
> > >> > > > > if so which one is best
> > >> > > > > And also getting confused on how to configure wait and notify
> > >> signals
> > >> > > > >
> > >> > > > > On Fri, 28 Feb 2020, 10:59 pm Eduardo Fontes, <
> > >> > > eduardo.fontes@gmail.com>
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > Hi! Maybe you should try another approach, like Wait/Notify
> or
> > >> > > > configure
> > >> > > > > > prioritizer just before the funnel, depending on your
> demand.
> > >> > > > > >
> > >> > > > > > On Fri, Feb 28, 2020 at 11:40 AM Midhun Mohan <
> > >> > > midhun.mohan@esginc.us>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Hi all,
> > >> > > > > > > How to prioritize one queue before a funnel?
> > >> > > > > > >
> > >> > > > > > > I need to make sure one queue is emptied before another
> > >> queue is
> > >> > > > taken
> > >> > > > > > > over for executing, I tried with a priority attribute but
> it
> > >> > fails
> > >> > > > > when a
> > >> > > > > > > queue with less priority has messages before the queue
> with
> > >> high
> > >> > > > > > priority.
> > >> > > > > > >
> > >> > > > > > > --
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > Regards,
> > >> > > > > > > Midhun Mohan
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
> > --
> >
> >
> > Regards,
> > Midhun Mohan
> >
>
--
Regards,
Midhun Mohan
Re: Queue Prioritization
Posted by Eduardo Fontes <ed...@gmail.com>.
Sorry,
Try this link: https://pasteboard.co/IXejthc.png
On Mon, Mar 2, 2020 at 9:35 AM Midhun Mohan <mi...@esginc.us> wrote:
> Hi Eduardo,
> Can you please send the Image again, I am not able to open it
>
> On Mon, 2 Mar 2020 at 18:02, Eduardo Fontes <ed...@gmail.com>
> wrote:
>
> > Hi Midhun,
> >
> > I think this flow will do what you want.
> >
> > [image: image.png]
> >
> > Be advised that use of Wait/Notify will slow down the average throughput
> > of your flow because the aditional steps with the Cache.
> >
> >
> >
> >
> > On Sun, Mar 1, 2020 at 12:57 PM Midhun Mohan <mi...@esginc.us>
> > wrote:
> >
> >> So I would like to know your suggestions on how to implement wait and
> >> notify.
> >>
> >> In my case messages will flow one after another
> >>
> >> If a failure occurs I need to retry the failure before the orginal ones,
> >> and notify users if it fails again.
> >>
> >>
> >> On Sun, 1 Mar 2020, 6:31 pm Emanuel Oliveira, <em...@gmail.com>
> wrote:
> >>
> >> > Hi,
> >> >
> >> > Invokehttp have 4 exits
> >> > - success
> >> > - error retry -> loop/count retries (updateattribite) -> if retries >
> >> max
> >> > then logmessage(error) flow stops here ff discarded Otherwise loop
> >> back to
> >> > invokehttp.
> >> > - error type 2 (bad password etc, )
> >> > - error type 3 (connectivity problems) you may want also to loop this
> >> one
> >> > with some delay , count retries and exit /log/stop after max.
> >> >
> >> > I just highly advise prior to invokehttp to have an updateattribute
> >> > deleting all attributes that don't start with some prefix.. to be sure
> >> your
> >> > ff looping doesnt keep adfing different errors, attributes wriiten by
> >> > invoke http.
> >> >
> >> > As good practice we prefix name all control attributes with
> >> flow_xxxxxxx so
> >> > our "cleanup," Updateattribute before Invokehttp have special regex
> >> sayng
> >> > "delete all attributes that don't start with flow_
> >> > (If you interested tomorrow when im back in office i csn share the
> >> regex a
> >> > bit tricky).
> >> >
> >> > Hope this helps, i just would love all processors to be as good as
> >> > invokehttp providing hard multipke exit points for specifics + all the
> >> nice
> >> > write attributes invokehttp adds that ptovide high details on errors
> or
> >> > success (atttibutes created/added by invokehttp to ff are prefixed
> with
> >> > invoke.xxxxx
> >> >
> >> > Cheers,
> >> > Emanuel O.
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Sun, 1 Mar 2020, 12:42 Midhun Mohan, <mi...@esginc.us>
> wrote:
> >> >
> >> > > Hi my use case is
> >> > >
> >> > > When a invokehttp processor fails I need to stop the incoming
> message
> >> and
> >> > > retry the failed message 2 times and if it is success start the
> >> execution
> >> > > normally or if it fails again need to notify admins and stop the
> main
> >> > > invoke http processor.
> >> > >
> >> > >
> >> > >
> >> > > Hope you can get the scenario from this
> >> > >
> >> > > On Sun, 1 Mar 2020, 5:30 pm Eduardo Fontes, <
> eduardo.fontes@gmail.com
> >> >
> >> > > wrote:
> >> > >
> >> > > > Hi Midhun,
> >> > > >
> >> > > > You can use DistributedMapCacheServer which starts a cache service
> >> > within
> >> > > > NiFi's process, but I not recommend in production. Instead you
> could
> >> > use
> >> > > > RedisConnectionPoolService.
> >> > > >
> >> > > > To use solution Wait/Notify the NiFi must know the number of
> >> flowfiles
> >> > to
> >> > > > process in one queue before start another queue. If it is a
> batch/on
> >> > > demand
> >> > > > flow I think it's easier. If it is a continuos/streamming flow you
> >> must
> >> > > > define a window, by number of flowfiles or by time, to switch
> >> between
> >> > > > queues.
> >> > > >
> >> > > > Please let me know what is your case.
> >> > > >
> >> > > > On Fri, Feb 28, 2020 at 2:39 PM Midhun Mohan <
> >> midhun.mohan@esginc.us>
> >> > > > wrote:
> >> > > >
> >> > > > > I think wait and notify can solve my problem. I checked the
> >> > documents ,
> >> > > > > which is bit unclear to me. Can anyone help me?
> >> > > > >
> >> > > > > Just to know few things like whether we need to configure a
> cache
> >> > > server,
> >> > > > > if so which one is best
> >> > > > > And also getting confused on how to configure wait and notify
> >> signals
> >> > > > >
> >> > > > > On Fri, 28 Feb 2020, 10:59 pm Eduardo Fontes, <
> >> > > eduardo.fontes@gmail.com>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hi! Maybe you should try another approach, like Wait/Notify or
> >> > > > configure
> >> > > > > > prioritizer just before the funnel, depending on your demand.
> >> > > > > >
> >> > > > > > On Fri, Feb 28, 2020 at 11:40 AM Midhun Mohan <
> >> > > midhun.mohan@esginc.us>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Hi all,
> >> > > > > > > How to prioritize one queue before a funnel?
> >> > > > > > >
> >> > > > > > > I need to make sure one queue is emptied before another
> >> queue is
> >> > > > taken
> >> > > > > > > over for executing, I tried with a priority attribute but it
> >> > fails
> >> > > > > when a
> >> > > > > > > queue with less priority has messages before the queue with
> >> high
> >> > > > > > priority.
> >> > > > > > >
> >> > > > > > > --
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > Regards,
> >> > > > > > > Midhun Mohan
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>
> --
>
>
> Regards,
> Midhun Mohan
>
Re: Queue Prioritization
Posted by Midhun Mohan <mi...@esginc.us>.
Hi Eduardo,
Can you please send the Image again, I am not able to open it
On Mon, 2 Mar 2020 at 18:02, Eduardo Fontes <ed...@gmail.com>
wrote:
> Hi Midhun,
>
> I think this flow will do what you want.
>
> [image: image.png]
>
> Be advised that use of Wait/Notify will slow down the average throughput
> of your flow because the aditional steps with the Cache.
>
>
>
>
> On Sun, Mar 1, 2020 at 12:57 PM Midhun Mohan <mi...@esginc.us>
> wrote:
>
>> So I would like to know your suggestions on how to implement wait and
>> notify.
>>
>> In my case messages will flow one after another
>>
>> If a failure occurs I need to retry the failure before the orginal ones,
>> and notify users if it fails again.
>>
>>
>> On Sun, 1 Mar 2020, 6:31 pm Emanuel Oliveira, <em...@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > Invokehttp have 4 exits
>> > - success
>> > - error retry -> loop/count retries (updateattribite) -> if retries >
>> max
>> > then logmessage(error) flow stops here ff discarded Otherwise loop
>> back to
>> > invokehttp.
>> > - error type 2 (bad password etc, )
>> > - error type 3 (connectivity problems) you may want also to loop this
>> one
>> > with some delay , count retries and exit /log/stop after max.
>> >
>> > I just highly advise prior to invokehttp to have an updateattribute
>> > deleting all attributes that don't start with some prefix.. to be sure
>> your
>> > ff looping doesnt keep adfing different errors, attributes wriiten by
>> > invoke http.
>> >
>> > As good practice we prefix name all control attributes with
>> flow_xxxxxxx so
>> > our "cleanup," Updateattribute before Invokehttp have special regex
>> sayng
>> > "delete all attributes that don't start with flow_
>> > (If you interested tomorrow when im back in office i csn share the
>> regex a
>> > bit tricky).
>> >
>> > Hope this helps, i just would love all processors to be as good as
>> > invokehttp providing hard multipke exit points for specifics + all the
>> nice
>> > write attributes invokehttp adds that ptovide high details on errors or
>> > success (atttibutes created/added by invokehttp to ff are prefixed with
>> > invoke.xxxxx
>> >
>> > Cheers,
>> > Emanuel O.
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Sun, 1 Mar 2020, 12:42 Midhun Mohan, <mi...@esginc.us> wrote:
>> >
>> > > Hi my use case is
>> > >
>> > > When a invokehttp processor fails I need to stop the incoming message
>> and
>> > > retry the failed message 2 times and if it is success start the
>> execution
>> > > normally or if it fails again need to notify admins and stop the main
>> > > invoke http processor.
>> > >
>> > >
>> > >
>> > > Hope you can get the scenario from this
>> > >
>> > > On Sun, 1 Mar 2020, 5:30 pm Eduardo Fontes, <eduardo.fontes@gmail.com
>> >
>> > > wrote:
>> > >
>> > > > Hi Midhun,
>> > > >
>> > > > You can use DistributedMapCacheServer which starts a cache service
>> > within
>> > > > NiFi's process, but I not recommend in production. Instead you could
>> > use
>> > > > RedisConnectionPoolService.
>> > > >
>> > > > To use solution Wait/Notify the NiFi must know the number of
>> flowfiles
>> > to
>> > > > process in one queue before start another queue. If it is a batch/on
>> > > demand
>> > > > flow I think it's easier. If it is a continuos/streamming flow you
>> must
>> > > > define a window, by number of flowfiles or by time, to switch
>> between
>> > > > queues.
>> > > >
>> > > > Please let me know what is your case.
>> > > >
>> > > > On Fri, Feb 28, 2020 at 2:39 PM Midhun Mohan <
>> midhun.mohan@esginc.us>
>> > > > wrote:
>> > > >
>> > > > > I think wait and notify can solve my problem. I checked the
>> > documents ,
>> > > > > which is bit unclear to me. Can anyone help me?
>> > > > >
>> > > > > Just to know few things like whether we need to configure a cache
>> > > server,
>> > > > > if so which one is best
>> > > > > And also getting confused on how to configure wait and notify
>> signals
>> > > > >
>> > > > > On Fri, 28 Feb 2020, 10:59 pm Eduardo Fontes, <
>> > > eduardo.fontes@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi! Maybe you should try another approach, like Wait/Notify or
>> > > > configure
>> > > > > > prioritizer just before the funnel, depending on your demand.
>> > > > > >
>> > > > > > On Fri, Feb 28, 2020 at 11:40 AM Midhun Mohan <
>> > > midhun.mohan@esginc.us>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi all,
>> > > > > > > How to prioritize one queue before a funnel?
>> > > > > > >
>> > > > > > > I need to make sure one queue is emptied before another
>> queue is
>> > > > taken
>> > > > > > > over for executing, I tried with a priority attribute but it
>> > fails
>> > > > > when a
>> > > > > > > queue with less priority has messages before the queue with
>> high
>> > > > > > priority.
>> > > > > > >
>> > > > > > > --
>> > > > > > >
>> > > > > > >
>> > > > > > > Regards,
>> > > > > > > Midhun Mohan
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
--
Regards,
Midhun Mohan
Re: Queue Prioritization
Posted by Eduardo Fontes <ed...@gmail.com>.
Hi Midhun,
I think this flow will do what you want.
[image: image.png]
Be advised that use of Wait/Notify will slow down the average throughput of
your flow because the aditional steps with the Cache.
On Sun, Mar 1, 2020 at 12:57 PM Midhun Mohan <mi...@esginc.us> wrote:
> So I would like to know your suggestions on how to implement wait and
> notify.
>
> In my case messages will flow one after another
>
> If a failure occurs I need to retry the failure before the orginal ones,
> and notify users if it fails again.
>
>
> On Sun, 1 Mar 2020, 6:31 pm Emanuel Oliveira, <em...@gmail.com> wrote:
>
> > Hi,
> >
> > Invokehttp have 4 exits
> > - success
> > - error retry -> loop/count retries (updateattribite) -> if retries > max
> > then logmessage(error) flow stops here ff discarded Otherwise loop back
> to
> > invokehttp.
> > - error type 2 (bad password etc, )
> > - error type 3 (connectivity problems) you may want also to loop this one
> > with some delay , count retries and exit /log/stop after max.
> >
> > I just highly advise prior to invokehttp to have an updateattribute
> > deleting all attributes that don't start with some prefix.. to be sure
> your
> > ff looping doesnt keep adfing different errors, attributes wriiten by
> > invoke http.
> >
> > As good practice we prefix name all control attributes with flow_xxxxxxx
> so
> > our "cleanup," Updateattribute before Invokehttp have special regex sayng
> > "delete all attributes that don't start with flow_
> > (If you interested tomorrow when im back in office i csn share the regex
> a
> > bit tricky).
> >
> > Hope this helps, i just would love all processors to be as good as
> > invokehttp providing hard multipke exit points for specifics + all the
> nice
> > write attributes invokehttp adds that ptovide high details on errors or
> > success (atttibutes created/added by invokehttp to ff are prefixed with
> > invoke.xxxxx
> >
> > Cheers,
> > Emanuel O.
> >
> >
> >
> >
> >
> >
> > On Sun, 1 Mar 2020, 12:42 Midhun Mohan, <mi...@esginc.us> wrote:
> >
> > > Hi my use case is
> > >
> > > When a invokehttp processor fails I need to stop the incoming message
> and
> > > retry the failed message 2 times and if it is success start the
> execution
> > > normally or if it fails again need to notify admins and stop the main
> > > invoke http processor.
> > >
> > >
> > >
> > > Hope you can get the scenario from this
> > >
> > > On Sun, 1 Mar 2020, 5:30 pm Eduardo Fontes, <ed...@gmail.com>
> > > wrote:
> > >
> > > > Hi Midhun,
> > > >
> > > > You can use DistributedMapCacheServer which starts a cache service
> > within
> > > > NiFi's process, but I not recommend in production. Instead you could
> > use
> > > > RedisConnectionPoolService.
> > > >
> > > > To use solution Wait/Notify the NiFi must know the number of
> flowfiles
> > to
> > > > process in one queue before start another queue. If it is a batch/on
> > > demand
> > > > flow I think it's easier. If it is a continuos/streamming flow you
> must
> > > > define a window, by number of flowfiles or by time, to switch between
> > > > queues.
> > > >
> > > > Please let me know what is your case.
> > > >
> > > > On Fri, Feb 28, 2020 at 2:39 PM Midhun Mohan <midhun.mohan@esginc.us
> >
> > > > wrote:
> > > >
> > > > > I think wait and notify can solve my problem. I checked the
> > documents ,
> > > > > which is bit unclear to me. Can anyone help me?
> > > > >
> > > > > Just to know few things like whether we need to configure a cache
> > > server,
> > > > > if so which one is best
> > > > > And also getting confused on how to configure wait and notify
> signals
> > > > >
> > > > > On Fri, 28 Feb 2020, 10:59 pm Eduardo Fontes, <
> > > eduardo.fontes@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi! Maybe you should try another approach, like Wait/Notify or
> > > > configure
> > > > > > prioritizer just before the funnel, depending on your demand.
> > > > > >
> > > > > > On Fri, Feb 28, 2020 at 11:40 AM Midhun Mohan <
> > > midhun.mohan@esginc.us>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > > How to prioritize one queue before a funnel?
> > > > > > >
> > > > > > > I need to make sure one queue is emptied before another queue
> is
> > > > taken
> > > > > > > over for executing, I tried with a priority attribute but it
> > fails
> > > > > when a
> > > > > > > queue with less priority has messages before the queue with
> high
> > > > > > priority.
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > >
> > > > > > > Regards,
> > > > > > > Midhun Mohan
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Queue Prioritization
Posted by Midhun Mohan <mi...@esginc.us>.
So I would like to know your suggestions on how to implement wait and
notify.
In my case messages will flow one after another
If a failure occurs I need to retry the failure before the orginal ones,
and notify users if it fails again.
On Sun, 1 Mar 2020, 6:31 pm Emanuel Oliveira, <em...@gmail.com> wrote:
> Hi,
>
> Invokehttp have 4 exits
> - success
> - error retry -> loop/count retries (updateattribite) -> if retries > max
> then logmessage(error) flow stops here ff discarded Otherwise loop back to
> invokehttp.
> - error type 2 (bad password etc, )
> - error type 3 (connectivity problems) you may want also to loop this one
> with some delay , count retries and exit /log/stop after max.
>
> I just highly advise prior to invokehttp to have an updateattribute
> deleting all attributes that don't start with some prefix.. to be sure your
> ff looping doesnt keep adfing different errors, attributes wriiten by
> invoke http.
>
> As good practice we prefix name all control attributes with flow_xxxxxxx so
> our "cleanup," Updateattribute before Invokehttp have special regex sayng
> "delete all attributes that don't start with flow_
> (If you interested tomorrow when im back in office i csn share the regex a
> bit tricky).
>
> Hope this helps, i just would love all processors to be as good as
> invokehttp providing hard multipke exit points for specifics + all the nice
> write attributes invokehttp adds that ptovide high details on errors or
> success (atttibutes created/added by invokehttp to ff are prefixed with
> invoke.xxxxx
>
> Cheers,
> Emanuel O.
>
>
>
>
>
>
> On Sun, 1 Mar 2020, 12:42 Midhun Mohan, <mi...@esginc.us> wrote:
>
> > Hi my use case is
> >
> > When a invokehttp processor fails I need to stop the incoming message and
> > retry the failed message 2 times and if it is success start the execution
> > normally or if it fails again need to notify admins and stop the main
> > invoke http processor.
> >
> >
> >
> > Hope you can get the scenario from this
> >
> > On Sun, 1 Mar 2020, 5:30 pm Eduardo Fontes, <ed...@gmail.com>
> > wrote:
> >
> > > Hi Midhun,
> > >
> > > You can use DistributedMapCacheServer which starts a cache service
> within
> > > NiFi's process, but I not recommend in production. Instead you could
> use
> > > RedisConnectionPoolService.
> > >
> > > To use solution Wait/Notify the NiFi must know the number of flowfiles
> to
> > > process in one queue before start another queue. If it is a batch/on
> > demand
> > > flow I think it's easier. If it is a continuos/streamming flow you must
> > > define a window, by number of flowfiles or by time, to switch between
> > > queues.
> > >
> > > Please let me know what is your case.
> > >
> > > On Fri, Feb 28, 2020 at 2:39 PM Midhun Mohan <mi...@esginc.us>
> > > wrote:
> > >
> > > > I think wait and notify can solve my problem. I checked the
> documents ,
> > > > which is bit unclear to me. Can anyone help me?
> > > >
> > > > Just to know few things like whether we need to configure a cache
> > server,
> > > > if so which one is best
> > > > And also getting confused on how to configure wait and notify signals
> > > >
> > > > On Fri, 28 Feb 2020, 10:59 pm Eduardo Fontes, <
> > eduardo.fontes@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi! Maybe you should try another approach, like Wait/Notify or
> > > configure
> > > > > prioritizer just before the funnel, depending on your demand.
> > > > >
> > > > > On Fri, Feb 28, 2020 at 11:40 AM Midhun Mohan <
> > midhun.mohan@esginc.us>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > > How to prioritize one queue before a funnel?
> > > > > >
> > > > > > I need to make sure one queue is emptied before another queue is
> > > taken
> > > > > > over for executing, I tried with a priority attribute but it
> fails
> > > > when a
> > > > > > queue with less priority has messages before the queue with high
> > > > > priority.
> > > > > >
> > > > > > --
> > > > > >
> > > > > >
> > > > > > Regards,
> > > > > > Midhun Mohan
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Queue Prioritization
Posted by Emanuel Oliveira <em...@gmail.com>.
Hi,
Invokehttp have 4 exits
- success
- error retry -> loop/count retries (updateattribite) -> if retries > max
then logmessage(error) flow stops here ff discarded Otherwise loop back to
invokehttp.
- error type 2 (bad password etc, )
- error type 3 (connectivity problems) you may want also to loop this one
with some delay , count retries and exit /log/stop after max.
I just highly advise prior to invokehttp to have an updateattribute
deleting all attributes that don't start with some prefix.. to be sure your
ff looping doesnt keep adfing different errors, attributes wriiten by
invoke http.
As good practice we prefix name all control attributes with flow_xxxxxxx so
our "cleanup," Updateattribute before Invokehttp have special regex sayng
"delete all attributes that don't start with flow_
(If you interested tomorrow when im back in office i csn share the regex a
bit tricky).
Hope this helps, i just would love all processors to be as good as
invokehttp providing hard multipke exit points for specifics + all the nice
write attributes invokehttp adds that ptovide high details on errors or
success (atttibutes created/added by invokehttp to ff are prefixed with
invoke.xxxxx
Cheers,
Emanuel O.
On Sun, 1 Mar 2020, 12:42 Midhun Mohan, <mi...@esginc.us> wrote:
> Hi my use case is
>
> When a invokehttp processor fails I need to stop the incoming message and
> retry the failed message 2 times and if it is success start the execution
> normally or if it fails again need to notify admins and stop the main
> invoke http processor.
>
>
>
> Hope you can get the scenario from this
>
> On Sun, 1 Mar 2020, 5:30 pm Eduardo Fontes, <ed...@gmail.com>
> wrote:
>
> > Hi Midhun,
> >
> > You can use DistributedMapCacheServer which starts a cache service within
> > NiFi's process, but I not recommend in production. Instead you could use
> > RedisConnectionPoolService.
> >
> > To use solution Wait/Notify the NiFi must know the number of flowfiles to
> > process in one queue before start another queue. If it is a batch/on
> demand
> > flow I think it's easier. If it is a continuos/streamming flow you must
> > define a window, by number of flowfiles or by time, to switch between
> > queues.
> >
> > Please let me know what is your case.
> >
> > On Fri, Feb 28, 2020 at 2:39 PM Midhun Mohan <mi...@esginc.us>
> > wrote:
> >
> > > I think wait and notify can solve my problem. I checked the documents ,
> > > which is bit unclear to me. Can anyone help me?
> > >
> > > Just to know few things like whether we need to configure a cache
> server,
> > > if so which one is best
> > > And also getting confused on how to configure wait and notify signals
> > >
> > > On Fri, 28 Feb 2020, 10:59 pm Eduardo Fontes, <
> eduardo.fontes@gmail.com>
> > > wrote:
> > >
> > > > Hi! Maybe you should try another approach, like Wait/Notify or
> > configure
> > > > prioritizer just before the funnel, depending on your demand.
> > > >
> > > > On Fri, Feb 28, 2020 at 11:40 AM Midhun Mohan <
> midhun.mohan@esginc.us>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > > How to prioritize one queue before a funnel?
> > > > >
> > > > > I need to make sure one queue is emptied before another queue is
> > taken
> > > > > over for executing, I tried with a priority attribute but it fails
> > > when a
> > > > > queue with less priority has messages before the queue with high
> > > > priority.
> > > > >
> > > > > --
> > > > >
> > > > >
> > > > > Regards,
> > > > > Midhun Mohan
> > > > >
> > > >
> > >
> >
>
Re: Queue Prioritization
Posted by Midhun Mohan <mi...@esginc.us>.
Hi my use case is
When a invokehttp processor fails I need to stop the incoming message and
retry the failed message 2 times and if it is success start the execution
normally or if it fails again need to notify admins and stop the main
invoke http processor.
Hope you can get the scenario from this
On Sun, 1 Mar 2020, 5:30 pm Eduardo Fontes, <ed...@gmail.com>
wrote:
> Hi Midhun,
>
> You can use DistributedMapCacheServer which starts a cache service within
> NiFi's process, but I not recommend in production. Instead you could use
> RedisConnectionPoolService.
>
> To use solution Wait/Notify the NiFi must know the number of flowfiles to
> process in one queue before start another queue. If it is a batch/on demand
> flow I think it's easier. If it is a continuos/streamming flow you must
> define a window, by number of flowfiles or by time, to switch between
> queues.
>
> Please let me know what is your case.
>
> On Fri, Feb 28, 2020 at 2:39 PM Midhun Mohan <mi...@esginc.us>
> wrote:
>
> > I think wait and notify can solve my problem. I checked the documents ,
> > which is bit unclear to me. Can anyone help me?
> >
> > Just to know few things like whether we need to configure a cache server,
> > if so which one is best
> > And also getting confused on how to configure wait and notify signals
> >
> > On Fri, 28 Feb 2020, 10:59 pm Eduardo Fontes, <ed...@gmail.com>
> > wrote:
> >
> > > Hi! Maybe you should try another approach, like Wait/Notify or
> configure
> > > prioritizer just before the funnel, depending on your demand.
> > >
> > > On Fri, Feb 28, 2020 at 11:40 AM Midhun Mohan <mi...@esginc.us>
> > > wrote:
> > >
> > > > Hi all,
> > > > How to prioritize one queue before a funnel?
> > > >
> > > > I need to make sure one queue is emptied before another queue is
> taken
> > > > over for executing, I tried with a priority attribute but it fails
> > when a
> > > > queue with less priority has messages before the queue with high
> > > priority.
> > > >
> > > > --
> > > >
> > > >
> > > > Regards,
> > > > Midhun Mohan
> > > >
> > >
> >
>