You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Midhun Mohan <mi...@esginc.us> on 2020/06/30 11:57:47 UTC
Retry on HTTP processor preserving order of flow file
Hi all,
I was trying to figure out a way to do a retry on http processor on
following scenario
1. The flow file should preserve the order , since the order of data is
critical for the API
I am using the retry processor to do the 3 times default retry with 20 sec
delay. All the In and Out queue is first in first out priority.
But on my realtime load test it seems the Order is not preserved. I am
getting a mix of responses.
Please help me to figure it out
Thanks in advance
--
Regards,
Midhun Mohan
Re: Retry on HTTP processor preserving order of flow file
Posted by Midhun Mohan <mi...@esginc.us>.
Thank you for your thoughts on it. Eduardo
On Wed, 29 Jul 2020 at 06:00, Eduardo Fontes <ed...@gmail.com>
wrote:
> Hi Midhun,
>
> Good idea! This way the processor will not consume from the main input
> connection because at least one of its output connection's queue is full,
> but it can consume from the "failure" because it will release the queue.
>
> Good job!
>
> On Mon, Jul 20, 2020 at 12:57 PM Midhun Mohan <mi...@esginc.us>
> wrote:
>
> > Hi Eduardo,
> > Thanks for the suggestions, I finally ended up testing it in a
> > different way.
> >
> > 1. Having the failure/retry relationship connected back to invoke http
> > processor with a backpressure value as 1.
> > 2. So on any failure all messages will get queued up before the
> > invokehttp processor.
> > 3. When we fix the issue with failing endpoints, the flow
> > miraculously keeps it's Order.
> > 4. I tested the API by failing it for more than 2 hours, thay I want to
> > make sure if it fails for days. It should keep the order.
> > 5. I am also planning for sending a mail if the queue is having a single
> > flowfile
> >
> > Do you see any issues with this method ?
> >
> >
> > On Sun, 5 Jul 2020 at 19:57, Eduardo Fontes <ed...@gmail.com>
> > wrote:
> >
> > > Hi Midhu,
> > >
> > > I can't think in other solution that guarantees the order using only
> > NiFi.
> > > Even if you put a funnel before the InvokeHTTP, receiving the main
> input
> > > and the retries with prioritizers in the queue, the order could not be
> > > guaranteed.Using other tools, you can do it by, for exemple, using an
> > > external queue (like Kafka), you could consume from it every time the
> > > InvokeHTTP succeed or failed with retries.
> > >
> > > Eduardo Fontes
> > > Data Engineer
> > >
> > >
> > > On Fri, Jul 3, 2020 at 12:53 AM Midhun Mohan <mi...@esginc.us>
> > > wrote:
> > >
> > > > Hi Eduardo,
> > > >
> > > > It seems success on 100% success scenario, but 50 % failure scenario
> it
> > > > seems to be mixing up the sequence.. So do you have any other
> > suggestion
> > > > than using wait/notify .
> > > >
> > > > On Wed, 1 Jul 2020 at 05:53, Eduardo Fontes <
> eduardo.fontes@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Mithun,
> > > > >
> > > > > First of all, did you verify if the sequence is ok in a 100%
> success
> > > > > scenario? If not, you could use the prioritizers to correct this.
> > > > > If you need to confirm every HTTP call (including retries) before
> > > process
> > > > > the next in the queue, you could use a Wait/Notify. But be advised
> > that
> > > > > this approach will slow down the throughput of your flow.
> > > > >
> > > > > Eduardo Fontes
> > > > > Data Engineer
> > > > >
> > > > >
> > > > > On Tue, Jun 30, 2020 at 8:58 AM Midhun Mohan <
> midhun.mohan@esginc.us
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I was trying to figure out a way to do a retry on http processor
> on
> > > > > > following scenario
> > > > > >
> > > > > > 1. The flow file should preserve the order , since the order of
> > data
> > > is
> > > > > > critical for the API
> > > > > >
> > > > > >
> > > > > > I am using the retry processor to do the 3 times default retry
> with
> > > 20
> > > > > sec
> > > > > > delay. All the In and Out queue is first in first out priority.
> > > > > >
> > > > > >
> > > > > > But on my realtime load test it seems the Order is not
> preserved. I
> > > am
> > > > > > getting a mix of responses.
> > > > > >
> > > > > > Please help me to figure it out
> > > > > >
> > > > > >
> > > > > > Thanks in advance
> > > > > >
> > > > > > --
> > > > > >
> > > > > >
> > > > > > Regards,
> > > > > > Midhun Mohan
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > >
> > > > Regards,
> > > > Midhun Mohan
> > > >
> > >
> >
> >
> > --
> >
> >
> > Regards,
> > Midhun Mohan
> >
>
--
Regards,
Midhun Mohan
Re: Retry on HTTP processor preserving order of flow file
Posted by Eduardo Fontes <ed...@gmail.com>.
Hi Midhun,
Good idea! This way the processor will not consume from the main input
connection because at least one of its output connection's queue is full,
but it can consume from the "failure" because it will release the queue.
Good job!
On Mon, Jul 20, 2020 at 12:57 PM Midhun Mohan <mi...@esginc.us>
wrote:
> Hi Eduardo,
> Thanks for the suggestions, I finally ended up testing it in a
> different way.
>
> 1. Having the failure/retry relationship connected back to invoke http
> processor with a backpressure value as 1.
> 2. So on any failure all messages will get queued up before the
> invokehttp processor.
> 3. When we fix the issue with failing endpoints, the flow
> miraculously keeps it's Order.
> 4. I tested the API by failing it for more than 2 hours, thay I want to
> make sure if it fails for days. It should keep the order.
> 5. I am also planning for sending a mail if the queue is having a single
> flowfile
>
> Do you see any issues with this method ?
>
>
> On Sun, 5 Jul 2020 at 19:57, Eduardo Fontes <ed...@gmail.com>
> wrote:
>
> > Hi Midhu,
> >
> > I can't think in other solution that guarantees the order using only
> NiFi.
> > Even if you put a funnel before the InvokeHTTP, receiving the main input
> > and the retries with prioritizers in the queue, the order could not be
> > guaranteed.Using other tools, you can do it by, for exemple, using an
> > external queue (like Kafka), you could consume from it every time the
> > InvokeHTTP succeed or failed with retries.
> >
> > Eduardo Fontes
> > Data Engineer
> >
> >
> > On Fri, Jul 3, 2020 at 12:53 AM Midhun Mohan <mi...@esginc.us>
> > wrote:
> >
> > > Hi Eduardo,
> > >
> > > It seems success on 100% success scenario, but 50 % failure scenario it
> > > seems to be mixing up the sequence.. So do you have any other
> suggestion
> > > than using wait/notify .
> > >
> > > On Wed, 1 Jul 2020 at 05:53, Eduardo Fontes <ed...@gmail.com>
> > > wrote:
> > >
> > > > Hi Mithun,
> > > >
> > > > First of all, did you verify if the sequence is ok in a 100% success
> > > > scenario? If not, you could use the prioritizers to correct this.
> > > > If you need to confirm every HTTP call (including retries) before
> > process
> > > > the next in the queue, you could use a Wait/Notify. But be advised
> that
> > > > this approach will slow down the throughput of your flow.
> > > >
> > > > Eduardo Fontes
> > > > Data Engineer
> > > >
> > > >
> > > > On Tue, Jun 30, 2020 at 8:58 AM Midhun Mohan <midhun.mohan@esginc.us
> >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I was trying to figure out a way to do a retry on http processor on
> > > > > following scenario
> > > > >
> > > > > 1. The flow file should preserve the order , since the order of
> data
> > is
> > > > > critical for the API
> > > > >
> > > > >
> > > > > I am using the retry processor to do the 3 times default retry with
> > 20
> > > > sec
> > > > > delay. All the In and Out queue is first in first out priority.
> > > > >
> > > > >
> > > > > But on my realtime load test it seems the Order is not preserved. I
> > am
> > > > > getting a mix of responses.
> > > > >
> > > > > Please help me to figure it out
> > > > >
> > > > >
> > > > > Thanks in advance
> > > > >
> > > > > --
> > > > >
> > > > >
> > > > > Regards,
> > > > > Midhun Mohan
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> > >
> > > Regards,
> > > Midhun Mohan
> > >
> >
>
>
> --
>
>
> Regards,
> Midhun Mohan
>
Re: Retry on HTTP processor preserving order of flow file
Posted by Midhun Mohan <mi...@esginc.us>.
Hi Eduardo,
Thanks for the suggestions, I finally ended up testing it in a
different way.
1. Having the failure/retry relationship connected back to invoke http
processor with a backpressure value as 1.
2. So on any failure all messages will get queued up before the
invokehttp processor.
3. When we fix the issue with failing endpoints, the flow
miraculously keeps it's Order.
4. I tested the API by failing it for more than 2 hours, thay I want to
make sure if it fails for days. It should keep the order.
5. I am also planning for sending a mail if the queue is having a single
flowfile
Do you see any issues with this method ?
On Sun, 5 Jul 2020 at 19:57, Eduardo Fontes <ed...@gmail.com>
wrote:
> Hi Midhu,
>
> I can't think in other solution that guarantees the order using only NiFi.
> Even if you put a funnel before the InvokeHTTP, receiving the main input
> and the retries with prioritizers in the queue, the order could not be
> guaranteed.Using other tools, you can do it by, for exemple, using an
> external queue (like Kafka), you could consume from it every time the
> InvokeHTTP succeed or failed with retries.
>
> Eduardo Fontes
> Data Engineer
>
>
> On Fri, Jul 3, 2020 at 12:53 AM Midhun Mohan <mi...@esginc.us>
> wrote:
>
> > Hi Eduardo,
> >
> > It seems success on 100% success scenario, but 50 % failure scenario it
> > seems to be mixing up the sequence.. So do you have any other suggestion
> > than using wait/notify .
> >
> > On Wed, 1 Jul 2020 at 05:53, Eduardo Fontes <ed...@gmail.com>
> > wrote:
> >
> > > Hi Mithun,
> > >
> > > First of all, did you verify if the sequence is ok in a 100% success
> > > scenario? If not, you could use the prioritizers to correct this.
> > > If you need to confirm every HTTP call (including retries) before
> process
> > > the next in the queue, you could use a Wait/Notify. But be advised that
> > > this approach will slow down the throughput of your flow.
> > >
> > > Eduardo Fontes
> > > Data Engineer
> > >
> > >
> > > On Tue, Jun 30, 2020 at 8:58 AM Midhun Mohan <mi...@esginc.us>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I was trying to figure out a way to do a retry on http processor on
> > > > following scenario
> > > >
> > > > 1. The flow file should preserve the order , since the order of data
> is
> > > > critical for the API
> > > >
> > > >
> > > > I am using the retry processor to do the 3 times default retry with
> 20
> > > sec
> > > > delay. All the In and Out queue is first in first out priority.
> > > >
> > > >
> > > > But on my realtime load test it seems the Order is not preserved. I
> am
> > > > getting a mix of responses.
> > > >
> > > > Please help me to figure it out
> > > >
> > > >
> > > > Thanks in advance
> > > >
> > > > --
> > > >
> > > >
> > > > Regards,
> > > > Midhun Mohan
> > > >
> > >
> >
> >
> > --
> >
> >
> > Regards,
> > Midhun Mohan
> >
>
--
Regards,
Midhun Mohan
Re: Retry on HTTP processor preserving order of flow file
Posted by Eduardo Fontes <ed...@gmail.com>.
Hi Midhu,
I can't think in other solution that guarantees the order using only NiFi.
Even if you put a funnel before the InvokeHTTP, receiving the main input
and the retries with prioritizers in the queue, the order could not be
guaranteed.Using other tools, you can do it by, for exemple, using an
external queue (like Kafka), you could consume from it every time the
InvokeHTTP succeed or failed with retries.
Eduardo Fontes
Data Engineer
On Fri, Jul 3, 2020 at 12:53 AM Midhun Mohan <mi...@esginc.us> wrote:
> Hi Eduardo,
>
> It seems success on 100% success scenario, but 50 % failure scenario it
> seems to be mixing up the sequence.. So do you have any other suggestion
> than using wait/notify .
>
> On Wed, 1 Jul 2020 at 05:53, Eduardo Fontes <ed...@gmail.com>
> wrote:
>
> > Hi Mithun,
> >
> > First of all, did you verify if the sequence is ok in a 100% success
> > scenario? If not, you could use the prioritizers to correct this.
> > If you need to confirm every HTTP call (including retries) before process
> > the next in the queue, you could use a Wait/Notify. But be advised that
> > this approach will slow down the throughput of your flow.
> >
> > Eduardo Fontes
> > Data Engineer
> >
> >
> > On Tue, Jun 30, 2020 at 8:58 AM Midhun Mohan <mi...@esginc.us>
> > wrote:
> >
> > > Hi all,
> > >
> > > I was trying to figure out a way to do a retry on http processor on
> > > following scenario
> > >
> > > 1. The flow file should preserve the order , since the order of data is
> > > critical for the API
> > >
> > >
> > > I am using the retry processor to do the 3 times default retry with 20
> > sec
> > > delay. All the In and Out queue is first in first out priority.
> > >
> > >
> > > But on my realtime load test it seems the Order is not preserved. I am
> > > getting a mix of responses.
> > >
> > > Please help me to figure it out
> > >
> > >
> > > Thanks in advance
> > >
> > > --
> > >
> > >
> > > Regards,
> > > Midhun Mohan
> > >
> >
>
>
> --
>
>
> Regards,
> Midhun Mohan
>
Re: Retry on HTTP processor preserving order of flow file
Posted by Midhun Mohan <mi...@esginc.us>.
Hi Eduardo,
It seems success on 100% success scenario, but 50 % failure scenario it
seems to be mixing up the sequence.. So do you have any other suggestion
than using wait/notify .
On Wed, 1 Jul 2020 at 05:53, Eduardo Fontes <ed...@gmail.com>
wrote:
> Hi Mithun,
>
> First of all, did you verify if the sequence is ok in a 100% success
> scenario? If not, you could use the prioritizers to correct this.
> If you need to confirm every HTTP call (including retries) before process
> the next in the queue, you could use a Wait/Notify. But be advised that
> this approach will slow down the throughput of your flow.
>
> Eduardo Fontes
> Data Engineer
>
>
> On Tue, Jun 30, 2020 at 8:58 AM Midhun Mohan <mi...@esginc.us>
> wrote:
>
> > Hi all,
> >
> > I was trying to figure out a way to do a retry on http processor on
> > following scenario
> >
> > 1. The flow file should preserve the order , since the order of data is
> > critical for the API
> >
> >
> > I am using the retry processor to do the 3 times default retry with 20
> sec
> > delay. All the In and Out queue is first in first out priority.
> >
> >
> > But on my realtime load test it seems the Order is not preserved. I am
> > getting a mix of responses.
> >
> > Please help me to figure it out
> >
> >
> > Thanks in advance
> >
> > --
> >
> >
> > Regards,
> > Midhun Mohan
> >
>
--
Regards,
Midhun Mohan
Re: Retry on HTTP processor preserving order of flow file
Posted by Eduardo Fontes <ed...@gmail.com>.
Hi Mithun,
First of all, did you verify if the sequence is ok in a 100% success
scenario? If not, you could use the prioritizers to correct this.
If you need to confirm every HTTP call (including retries) before process
the next in the queue, you could use a Wait/Notify. But be advised that
this approach will slow down the throughput of your flow.
Eduardo Fontes
Data Engineer
On Tue, Jun 30, 2020 at 8:58 AM Midhun Mohan <mi...@esginc.us> wrote:
> Hi all,
>
> I was trying to figure out a way to do a retry on http processor on
> following scenario
>
> 1. The flow file should preserve the order , since the order of data is
> critical for the API
>
>
> I am using the retry processor to do the 3 times default retry with 20 sec
> delay. All the In and Out queue is first in first out priority.
>
>
> But on my realtime load test it seems the Order is not preserved. I am
> getting a mix of responses.
>
> Please help me to figure it out
>
>
> Thanks in advance
>
> --
>
>
> Regards,
> Midhun Mohan
>