[jira] [Commented] (OOZIE-3612) CompositeCallable became empty after requeue and cause IndexOutOfBoundException in getType() method, which cause counters for this type in activeCallables never be descreased and exceed concurrency

["yuanyimeng (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OOZIE-3612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17216586#comment-17216586 ] 

yuanyimeng commented on OOZIE-3612:
-----------------------------------

I  use a workaround to fix this by modify getType method. fix the CompositeCallable's the when create and return in getType(). Not dynamically get type form the first element. This would have some problems like the compositeCallable is action_check type first, but the element is filtered and the first element became coord action check. We will get a CompositeCallable only have coord action check command but the type indicate it is an action check.  But I this it's not a big deal.

> CompositeCallable became empty after requeue and cause IndexOutOfBoundException in getType() method, which cause counters for this type in  activeCallables never be descreased and exceed concurrency 
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: OOZIE-3612
>                 URL: https://issues.apache.org/jira/browse/OOZIE-3612
>             Project: Oozie
>          Issue Type: Bug
>          Components: action, workflow
>    Affects Versions: 4.2.0
>         Environment: We use oozie 4.2.0. but in the newest release this problems seems also exists
>            Reporter: yuanyimeng
>            Priority: Minor
>         Attachments: callableRun.png, counternotdescreaedifgettypeexceptionhappen.png, eception_in_composite_callable.png, exceptionlog.png
>
>
> This should rarely happens but it just happened in out environment twice. The process is listed below.
>  # we have long hang actions which last serveral hours (This is and error sitution, task execute quickly in normal time). the action type is developed by ourself by extend the Executor.
>  # These actions's ActionCheckXCommand is put into the callable queue by composited into an array with max ten element
>  # Before really put into the thread pool for execute, they will be filtered if element is already existed ,  the existence is identified by the uniqueCallables. So the real CompositeCallable in queue may actually have element less then 10.
>  # When this CompositeCallable is poll form queue. Before execute it will check the concurreny for the action_check type. if the concurrency is reached, it will be requeued. 
>  # In the requeue procedure ,  the ActionCheckService happens to already put these ActionCheckXCommand in queue, so the CompositeCallable is filtered with 0 element.
>  # In the finally procedure, the CallableEnd method will need to descrease the counter of this type . But IndexOutOfBounds happens when called on the empty CompositeCallable's getType method, which cause the counter will never be descreased
>  # If it happens for the maxComcurrency time, the counter itself will exceed the maxConcurrency and this type can never be taked outside of the queue,  they will lived in queue forever, which cause the workflow hang. 
>  
> The pic  where the exception happened is attached. Hopes it describe it clearly. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)