[jira] [Commented] (CAMEL-21671) camel-core - Issue with Camel Splitter and Asynchronous Processing in Parallel Mode can use many threads

["Claus Ibsen (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CAMEL-21671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17931917#comment-17931917 ] 

Claus Ibsen commented on CAMEL-21671:
-------------------------------------

Okay we have the synchronous option already, so I am improving this to run like Camel v2 when you set synchronous(true)

> camel-core - Issue with Camel Splitter and Asynchronous Processing in Parallel Mode can use many threads
> --------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-21671
>                 URL: https://issues.apache.org/jira/browse/CAMEL-21671
>             Project: Camel
>          Issue Type: Improvement
>          Components: came-core
>    Affects Versions: 4.8.0
>            Reporter: Soheila Esmaeili
>            Assignee: Claus Ibsen
>            Priority: Major
>             Fix For: 4.11.0
>
>         Attachments: SplitterParallelAsyncProcessorTest.java
>
>
> We are currently facing an issue with a Camel route that uses a splitter with parallelProcessing. The route includes an asynchronous processor and a delay within the splitter. However, the problem is that the split threads do not wait for the asynchronous process to complete before proceeding further. 
> Although the max. number of threads for split processing is set to 2, we have no control over thread usage due to the async processor. In *Camel 3.14* and {*}Camel 4.8{*}, the 2 threads will forward the message to the async processor, then continue with the next 2 split parts, which are also handed over to another thread pool and so on. This can lead to hundreds of parallel threads performing the operations defined within the split-block. If these operations are long-running and memory-intensive, this can lead to further problems. {*}In the older (unmaintained – 2.x) Camel version{*}, this was not the case. Although the threads changed due to the async processor, the “split-related threads” were in a blocking/waiting state (I assume). At least there was no “unbounded” upscaling in the number of threads.
> Could you please confirm if this is a known issue or if there are specific configurations needed in newer Camel versions to achieve the expected behavior? 
> A simplified unit test can be found in the appendix.
> *Result in Camel 2.24:*
> Before AsyncProcessor - Thread: Camel (camel-1) thread #2 - Split
> Before AsyncProcessor - Thread: Camel (camel-1) thread #1 - Split
> After AsyncProcessor - Thread: ForkJoinPool.commonPool-worker-1
> After AsyncProcessor - Thread: ForkJoinPool.commonPool-worker-2
> Before AsyncProcessor - Thread: Camel (camel-1) thread #2 - Split
> Before AsyncProcessor - Thread: Camel (camel-1) thread #1 - Split
> After AsyncProcessor - Thread: ForkJoinPool.commonPool-worker-1
> After AsyncProcessor - Thread: ForkJoinPool.commonPool-worker-2
> Before AsyncProcessor - Thread: Camel (camel-1) thread #2 - Split
> Before AsyncProcessor - Thread: Camel (camel-1) thread #1 - Split
> After AsyncProcessor - Thread: ForkJoinPool.commonPool-worker-2
> After AsyncProcessor - Thread: ForkJoinPool.commonPool-worker-1
> Before AsyncProcessor - Thread: Camel (camel-1) thread #1 - Split
> Before AsyncProcessor - Thread: Camel (camel-1) thread #2 - Split
> After AsyncProcessor - Thread: ForkJoinPool.commonPool-worker-2
> After AsyncProcessor - Thread: ForkJoinPool.commonPool-worker-1
> Before AsyncProcessor - Thread: Camel (camel-1) thread #1 - Split
> Before AsyncProcessor - Thread: Camel (camel-1) thread #2 - Split
> After AsyncProcessor - Thread: ForkJoinPool.commonPool-worker-2
> After AsyncProcessor - Thread: ForkJoinPool.commonPool-worker-1
>  
> *Result in 3.14 and 4.8:*
> Before AsyncProcessor - Thread: Camel (camel-1) thread #2 - Split
> Before AsyncProcessor - Thread: Camel (camel-1) thread #3 - Split
> Before AsyncProcessor - Thread: Camel (camel-1) thread #2 - Split
> Before AsyncProcessor - Thread: Camel (camel-1) thread #2 - Split
> Before AsyncProcessor - Thread: Camel (camel-1) thread #3 - Split
> Before AsyncProcessor - Thread: Camel (camel-1) thread #2 - Split
> Before AsyncProcessor - Thread: Camel (camel-1) thread #2 - Split
> Before AsyncProcessor - Thread: Camel (camel-1) thread #2 - Split
> Before AsyncProcessor - Thread: Camel (camel-1) thread #2 - Split
> Before AsyncProcessor - Thread: Camel (camel-1) thread #3 - Split
> After AsyncProcessor - Thread: ForkJoinPool.commonPool-worker-6
> After AsyncProcessor - Thread: ForkJoinPool.commonPool-worker-3
> After AsyncProcessor - Thread: ForkJoinPool.commonPool-worker-5
> After AsyncProcessor - Thread: ForkJoinPool.commonPool-worker-10
> After AsyncProcessor - Thread: ForkJoinPool.commonPool-worker-8
> After AsyncProcessor - Thread: ForkJoinPool.commonPool-worker-2
> After AsyncProcessor - Thread: ForkJoinPool.commonPool-worker-9
> After AsyncProcessor - Thread: ForkJoinPool.commonPool-worker-7
> After AsyncProcessor - Thread: ForkJoinPool.commonPool-worker-1
> After AsyncProcessor - Thread: ForkJoinPool.commonPool-worker-4



--
This message was sent by Atlassian Jira
(v8.20.10#820010)