UIMA-AS worker node failure

[Jörn Kottmann <ko...@gmail.com>]

Hi,

in our setup a service is deployed on multiple machines,
now one of these worker nodes fails (e.g. power supply failed).

What does UIMA-AS in this case?

Is it possible that CASes sent from the client are not processed?

Thanks,
Jörn

Re: UIMA-AS worker node failure

[Eddie Epstein <ea...@gmail.com>]

On Fri, Oct 16, 2009 at 10:37 AM, Jörn Kottmann <ko...@gmail.com> wrote:
> Jaroslaw Cwiklik wrote:
>>
>> In your particular case ( node dying), getting a message in a
>> transactional
>> context could have helped. Transactions are not "free" and I certainly
>> would
>> not want to use them by default. Transactions have impact on performance.
>> We
>> could enable transactional JMS session using a configuration setting.
>> Perhaps via -D.
>>
>
> Sure, do you think transactions are really expensive and slow the
> system down ?
>
> Otherwise it might be better to have a cluster master which just keeps
> track of everything.
>
> I also noticed that client node failure is difficult to handle, because then
> you loose
> track of the of the pending CASes, which are processed right now.
>
And if the cluster master fails?


> Our system reads from a work queue which is feed by a different system,
> then the work item is wrapped in a CAS and sent to the uima-as cluster, when
> the client
> gets the work back from the cluster the result is sent to another queue.
> If now the client crashes (dies) we do not get back the CASes already sent
> to the cluster
> but have not returned yet. To improve that situation it would be cool if
> UIMA AS could
> guarantee that every CAS which was sent to the cluster can also be retrieved
> from it,
> even in case the client crashes (or a worker node dies). Our use case would
> be really simple if UIMA AS just defines
> two queues, one to feed it and one to read results form it (I think it is
> nearly done like that right now).
UIMA AS uses temporary queues to create unique reply
queues for any number of clients running on the same or
multiple nodes, as well as cleaning up broker resources
created by clients.

> The part which reads the result from uima-as must not necessary run in the
> same process (or machine) as the
> part which feeds it.
>
How about creating a service that all worker nodes call at the
of their processing pipelines to save the results and or status?
Or using a shared filesystem for final results?

Eddie

Re: UIMA-AS worker node failure

[Jörn Kottmann <ko...@gmail.com>]

Jaroslaw Cwiklik wrote:
> In your particular case ( node dying), getting a message in a transactional
> context could have helped. Transactions are not "free" and I certainly would
> not want to use them by default. Transactions have impact on performance. We
> could enable transactional JMS session using a configuration setting.
> Perhaps via -D.
>   
Sure, do you think transactions are really expensive and slow the
system down ?

Otherwise it might be better to have a cluster master which just keeps
track of everything.

I also noticed that client node failure is difficult to handle, because 
then you loose
track of the of the pending CASes, which are processed right now.

Our system reads from a work queue which is feed by a different system,
then the work item is wrapped in a CAS and sent to the uima-as cluster, 
when the client
gets the work back from the cluster the result is sent to another queue.
If now the client crashes (dies) we do not get back the CASes already 
sent to the cluster
but have not returned yet. To improve that situation it would be cool if 
UIMA AS could
guarantee that every CAS which was sent to the cluster can also be 
retrieved from it,
even in case the client crashes (or a worker node dies). Our use case 
would be really simple if UIMA AS just defines
two queues, one to feed it and one to read results form it (I think it 
is nearly done like that right now).
The part which reads the result from uima-as must not necessary run in 
the same process (or machine) as the
part which feeds it.

Jörn

Re: UIMA-AS worker node failure

[Jaroslaw Cwiklik <ui...@gmail.com>]

In your particular case ( node dying), getting a message in a transactional
context could have helped. Transactions are not "free" and I certainly would
not want to use them by default. Transactions have impact on performance. We
could enable transactional JMS session using a configuration setting.
Perhaps via -D.
jerry


On Fri, Oct 16, 2009 at 7:16 AM, Jörn Kottmann <ko...@gmail.com> wrote:

> Jaroslaw Cwiklik wrote:
>
>> Jorn, if a node dies while a service processes a CAS your client may not
>> see
>> a response. What happens next really depends if the client is configured
>> to
>> timeout. If you use
>> timeouts, UIMA AS client notifies an application and it may resend the CAS
>> for processing
>> by remaining services
>>
>>
> Ok, thanks for your answer.
>
> Would it be possible to use JMS transactions
> or the JMS acknowledge feature to make node failure safe
> on the cluster side (and not handle it the client) ?
>
> Jörn
>
>

Re: UIMA-AS worker node failure

[Jörn Kottmann <ko...@gmail.com>]

Jaroslaw Cwiklik wrote:
> Jorn, if a node dies while a service processes a CAS your client may not see
> a response. What happens next really depends if the client is configured to
> timeout. If you use
> timeouts, UIMA AS client notifies an application and it may resend the CAS
> for processing
> by remaining services
>   
Ok, thanks for your answer.

Would it be possible to use JMS transactions
or the JMS acknowledge feature to make node failure safe
on the cluster side (and not handle it the client) ?

Jörn

Re: UIMA-AS worker node failure

[Jaroslaw Cwiklik <ui...@gmail.com>]

Jorn, if a node dies while a service processes a CAS your client may not see
a response. What happens next really depends if the client is configured to
timeout. If you use
timeouts, UIMA AS client notifies an application and it may resend the CAS
for processing
by remaining services

Jerrry


On Thu, Oct 15, 2009 at 1:18 PM, Jörn Kottmann <ko...@gmail.com> wrote:

> Hi,
>
> in our setup a service is deployed on multiple machines,
> now one of these worker nodes fails (e.g. power supply failed).
>
> What does UIMA-AS in this case?
>
> Is it possible that CASes sent from the client are not processed?
>
> Thanks,
> Jörn
>