UIMA-AS on Kubernetes

["Huy, Patrick" <pa...@sap.com>]

Hi,

we are running UIMA-AS on Kubernetes and we are facing some issues: when the endpoint a BaseUIMAAsynchronousEngine_impl is connected to goes missing (because it's being restarted by Kubernetes) while a CAS is being processed it seems like the AsynchronousEngine never recovers and becomes unusable from that point (it seems that the CAS is "gone"), currently we are using the "failover" activemq protocol but it does not seem to always help with the situation.

When the components processing a CAS go down while a CAS is being processed we are currently unable to detect whether the process is just taking long or whether the component failed - we are currently waiting on processing Timeouts to catch this state but something better/faster (like healthiness pings or something) would be more desirable.

Also it seems like our component using the  BaseUIMAAsynchronousEngines requires a very high amount of RAM to hold the CASes for each CAS Pool (around 5MB per CAS) is there anything that can be done to optimize this?

Thanks in advance!
Patrick

Re: UIMA-AS on Kubernetes

[Jaroslaw Cwiklik <cw...@apache.org>]

Patrick, service monitoring is left to applications. I think Kubernetes has
some basic monitoring of pods and
if more is needed I am sure there are third party monitoring tools
available.

The UIMA-AS does provide you with info about which service instance is
processing each CAS.
If you register a callback listener, you get node IP, pid and CAS via

Callback.onBeforeProcessCAS(status, nodeIP, pid);

You can use this to check if the service is running. Recent UIMA-AS
supports targeting specific service instance.
https://uima.apache.org/d/uima-as-2.10.3/uima_async_scaleout.html#ugr.ref.async.api.usage_targetservice
You can send a small test CAS to the specific instance at regular intervals
to see if it is viable.

The UIMA-AS holds on to each CAS which is in-process because it needs it to
deserialize a reply and
to support retry. The only controls you have are the size of the pool and
-Xmx

Jerry

On Thu, Oct 11, 2018 at 5:26 AM Huy, Patrick <pa...@sap.com> wrote:

> Hi,
>
> we are running UIMA-AS on Kubernetes and we are facing some issues: when
> the endpoint a BaseUIMAAsynchronousEngine_impl is connected to goes missing
> (because it's being restarted by Kubernetes) while a CAS is being processed
> it seems like the AsynchronousEngine never recovers and becomes unusable
> from that point (it seems that the CAS is "gone"), currently we are using
> the "failover" activemq protocol but it does not seem to always help with
> the situation.
>
> When the components processing a CAS go down while a CAS is being
> processed we are currently unable to detect whether the process is just
> taking long or whether the component failed - we are currently waiting on
> processing Timeouts to catch this state but something better/faster (like
> healthiness pings or something) would be more desirable.
>
> Also it seems like our component using the  BaseUIMAAsynchronousEngines
> requires a very high amount of RAM to hold the CASes for each CAS Pool
> (around 5MB per CAS) is there anything that can be done to optimize this?
>
> Thanks in advance!
> Patrick
>