Camel consul leader election

[Amin Mohammed-Coleman <am...@digitalpatterns.io>]

Hi there

I have integrated camel and consult to perform leader election when it come
to polling. So far everything is working as expected. If I kill on of my
pods then another pod becomes the leader. One thing I have noticed is that
when I invalidate the session via Consul UI the camel service throws the
following exception:



com.orbitz.consul.ConsulException: Consul request failed with status
[404]: Session id 'da77a3a5-dbf8-49ab-9a46-f759b325895b' not
foundcom.orbitz.consul.ConsulException: Consul request failed with
status [404]: Session id 'da77a3a5-dbf8-49ab-9a46-f759b325895b' not
found at com.orbitz.consul.util.Http.ensureResponseSuccessful(Http.java:61)
at com.orbitz.consul.util.Http.extract(Http.java:33) at
com.orbitz.consul.SessionClient.renewSession(SessionClient.java:80) at
com.orbitz.consul.SessionClient.renewSession(SessionClient.java:69) at
org.apache.camel.component.consul.cluster.ConsulClusterView$Watcher.watch(ConsulClusterView.java:279)
at org.apache.camel.component.consul.cluster.ConsulClusterView$Watcher.onComplete(ConsulClusterView.java:252)
at com.orbitz.consul.KeyValueClient$1.onComplete(KeyValueClient.java:167)
at com.orbitz.consul.util.Http$1.onResponse(Http.java:80) at
retrofit2.OkHttpCall$1.onResponse(OkHttpCall.java:129) at
okhttp3.RealCall$AsyncCall.execute(RealCall.java:174) at
okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

Which I would expect to happen. But this continues until I have to manually
kill the pod and restart the camel service.

Is there anyway to perform a retry X times and then acquire a new session?
I couldn't see any configuration that would enable this, unless this is the
expected behaviour.



Any help would be appreciated.

Re: Camel consul leader election

[Claus Ibsen <cl...@gmail.com>]

Hi

What version of Camel do you use?

And when you kill the session, its the Camel master pod that throws
this exception? The slaves are running okay and standing by?

And yeah maybe there is something that can be improved in camel-consul
in that cluster logic to detect this session invalidation problem, and
then maybe loose its master status, and make it re-elect master and
build up a new session etc.

Wonder if you could dive into the camel-consul source code, and try to
work on a fix, and if so provide a PR with the fix
https://camel.apache.org/community/support/

And you are welcome to create a JIRA about this issue

On Thu, Aug 20, 2020 at 12:31 AM Amin Mohammed-Coleman
<am...@digitalpatterns.io> wrote:
>
> Hi there
>
> I have integrated camel and consult to perform leader election when it come
> to polling. So far everything is working as expected. If I kill on of my
> pods then another pod becomes the leader. One thing I have noticed is that
> when I invalidate the session via Consul UI the camel service throws the
> following exception:
>
>
>
> com.orbitz.consul.ConsulException: Consul request failed with status
> [404]: Session id 'da77a3a5-dbf8-49ab-9a46-f759b325895b' not
> foundcom.orbitz.consul.ConsulException: Consul request failed with
> status [404]: Session id 'da77a3a5-dbf8-49ab-9a46-f759b325895b' not
> found at com.orbitz.consul.util.Http.ensureResponseSuccessful(Http.java:61)
> at com.orbitz.consul.util.Http.extract(Http.java:33) at
> com.orbitz.consul.SessionClient.renewSession(SessionClient.java:80) at
> com.orbitz.consul.SessionClient.renewSession(SessionClient.java:69) at
> org.apache.camel.component.consul.cluster.ConsulClusterView$Watcher.watch(ConsulClusterView.java:279)
> at org.apache.camel.component.consul.cluster.ConsulClusterView$Watcher.onComplete(ConsulClusterView.java:252)
> at com.orbitz.consul.KeyValueClient$1.onComplete(KeyValueClient.java:167)
> at com.orbitz.consul.util.Http$1.onResponse(Http.java:80) at
> retrofit2.OkHttpCall$1.onResponse(OkHttpCall.java:129) at
> okhttp3.RealCall$AsyncCall.execute(RealCall.java:174) at
> okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
>
> Which I would expect to happen. But this continues until I have to manually
> kill the pod and restart the camel service.
>
> Is there anyway to perform a retry X times and then acquire a new session?
> I couldn't see any configuration that would enable this, unless this is the
> expected behaviour.
>
>
>
> Any help would be appreciated.



-- 
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2