[jira] [Updated] (TOMEE-4699) ThreadContext constructor is not thread-safe, causing loss of submitted tasks.

["Richard Zowalla (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/TOMEE-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Richard Zowalla updated TOMEE-4699:
-----------------------------------
    Fix Version/s: 11.0.0
                   10.3.0

> ThreadContext constructor is not thread-safe, causing loss of submitted tasks.
> ------------------------------------------------------------------------------
>
>                 Key: TOMEE-4699
>                 URL: https://issues.apache.org/jira/browse/TOMEE-4699
>             Project: TomEE
>          Issue Type: Bug
>          Components: TomEE Core Server
>    Affects Versions: 10.2.0
>            Reporter: Jorren Hendriks
>            Priority: Major
>             Fix For: 11.0.0, 10.3.0
>
>         Attachments: stacktrace.txt
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> Some of our tomee applications have experienced issues losing tasks submitted to a managed executor. 
> The root cause has been identified as a `{_}ConcurrentModificationException{_}` in the ThreadContext constructor (stacktrace attached).
> {code:java}
> public ThreadContext(final ThreadContext that) {
>     this.beanContext = that.beanContext;
>     this.primaryKey = that.primaryKey;
>     this.data.putAll(that.data); // throws ConcurrentModificationException when that.data is modified on another thread
>     this.oldClassLoader = that.oldClassLoader;
> }{code}
> `{_}HashMap.putAll{_}` will iterate the entrySet, which is not thread-safe on the `{_}Collections.synchronizedMap{_}` wrapped HashMap currently in use.
> A possible solution would be to use a thread-safe Map implementation.
> This constructor is called when entering the snapshot in the
> [{_}ApplicationThreadContextProvider{_}|[https://github.com/apache/tomee/blob/main/container/openejb-core/src/main/java/org/apache/openejb/threads/impl/ApplicationThreadContextProvider.java#L76|https://github.com/apache/tomee/blob/main/container/openejb-core/src/main/java/org/apache/openejb/threads/impl/ApplicationThreadContextProvider.java#L76].]]
> This issue also revealed possible other issues for us:
> 1. `{_}CUTask{_}` does not handle exceptions before starting the task. Any exceptions in `{_}contextService.enter(snapshot){_}` cancel invocation, but do not abort the task.
> 2. Exceptions in a `{_}CUTask{_}`/`{_}CURunnable{_}` are propagated to the `{_}ThreadPoolExecutor{_}`. The current implementation does not expose these exceptions which makes this a silent failure. 
> If you'd prefer that I can file these as separate issues, but they are related to this problem.
> h2. Reproduction
> This is a race condition which very rarely occurs. We were able to reproduce it (under high load) within an hour on one of our applications with a debugger attached. So far we were not able to create a minimal reproduction outside our application. I will update the issue when we do.
> The pattern we use in the application is as follows, the comments indicate my interpretation of what happens:
> {code:java}
> // MyTaskExecutor.java
> @Stateless
> public class MyTaskExecutor {
>   @Resource
>   private ManagedExecutorService managedExecutorService; // default TomEE managed executor service
>   @Override
>   public void execute(@NotNull Runnable runnable) {
>     // 1. runnable is wrapped in CUTask. ThreadContext is copied on the executor thread.
>     managedExecutorService.submit(runnable);
>   } // 2. method returns immediately, causing an update to the current ThreadContext.
> } {code}
> When the exception occurs, 1 & 2 both execute at the same time, accessing the current ThreadContext concurrently.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)