http.context.identifier not found when flowfile does not come back on the same node

[oliver twix <ol...@gmail.com>]

Hello,

Here is the problem I am facing:

I have two NIFI clusters (3 members each) connected through S2S connection.
I am creating a web service using handleHttpRequest and  handleHttpResponse
on cluster 1 and implement web service backend logic on cluster 2. When I
want to give back the answer to the requester, the flowfile will go back to
cluster 1 randomly over cluster 1 members. The problem is “http context
cache” not being distributed, I got 2 chances over 3 to get
“http.context.identifier” not found in the cache. I can apply a workaround
like adding a round robin load balancing strategy on failure queue but it
is not clean.



I identify two possible solutions:

-        Change the cache to a distribute one

-        Add the possibility to route a flowfile to a given node



How could you help?



Best regards

Olivier

Re: http.context.identifier not found when flowfile does not come back on the same node

[Bryan Bende <bb...@gmail.com>]

Hello,

It wasn't really intended that S2S would be used as a mechanism to handle
web service request. The best option would be to just use a single cluster
for the web service so you don't need S2S.

If for some reason you need two clusters, then you need a different way of
calling to the second cluster. You could have a second set of
HandleHttpRequest/Response on the second cluster. The first cluster could
have an InvokeHttp processor for each node of the second cluster, and then
a DistributeLoad processor in front of them to round robin the requests. It
is not as flexible as S2S since you would be hard coding the hostnames of
the second cluster into the processors in the first cluster, but it would
ensure you get the response back on the node that made the request.

Thanks,

Bryan

On Wed, Apr 1, 2020 at 3:25 AM oliver twix <ol...@gmail.com>
wrote:

> Hello,
>
> Here is the problem I am facing:
>
> I have two NIFI clusters (3 members each) connected through S2S
> connection. I am creating a web service using handleHttpRequest and
>  handleHttpResponse on cluster 1 and implement web service backend logic on
> cluster 2. When I want to give back the answer to the requester, the
> flowfile will go back to cluster 1 randomly over cluster 1 members. The
> problem is “http context cache” not being distributed, I got 2 chances over
> 3 to get “http.context.identifier” not found in the cache. I can apply a
> workaround like adding a round robin load balancing strategy on failure
> queue but it is not clean.
>
>
>
> I identify two possible solutions:
>
> -        Change the cache to a distribute one
>
> -        Add the possibility to route a flowfile to a given node
>
>
>
> How could you help?
>
>
>
> Best regards
>
> Olivier
>