You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@camel.apache.org by "Alexander Anpilov (Jira)" <ji...@apache.org> on 2024/03/13 10:48:00 UTC
[jira] [Closed] (CAMEL-20556) FileConsumer OutOfMemory for big files in Unix
[ https://issues.apache.org/jira/browse/CAMEL-20556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexander Anpilov closed CAMEL-20556.
-------------------------------------
Resolution: Fixed
BacklogTracing was tried to log file content.
You should disable tracing or disable tracing file content with properties
> FileConsumer OutOfMemory for big files in Unix
> ----------------------------------------------
>
> Key: CAMEL-20556
> URL: https://issues.apache.org/jira/browse/CAMEL-20556
> Project: Camel
> Issue Type: Bug
> Components: camel-file
> Affects Versions: 3.20.9, 3.22.1
> Reporter: Alexander Anpilov
> Priority: Minor
>
> For clarity, I have tested and described issue with simple routes:
>
> 1. *FileConsumer*
> **
> {code:java}
> from("file:/path")
> .log("Received file: ${file:name}"); {code}
>
>
> 2. *File pollEnrich*
> {code:java}
> from("direct:start")
> .setProperty("SOURCE_URI", simple("{{path_to_folder}}"))
> .pollEnrich().exchangeProperty("SOURCE_URI").timeout(60000)
> .log("Received file: ${file:name}"); {code}
> My process receive big files (more than 4x of Java XMX) and running as spring-boot app on Kubernetes.
> After upgrading from camel-3.20.2 to camel-3.20.9, the OutOfMemory error occurs on consume. I have rolled back and checked with camel-3.20.2 again on the same files - everything OK.
> It seems, that FileConsumer publish file content into memory, instead of using streams.
> The problem starts from version camel-3.20.3 and reproduced up to 3.20.9.
> *BUT:* I couldn't reproduce problem on Windows with Camel Test, only in Unix prod env. Maybe, there are some camel-test effects...
> *UPD:* I have tested camel-3.22.1 and received OutOfMemory
> *UPD*
> With some route modifications:
> {code:java}
> from("amq:start")
> .pollEnrich().simple("{{path_to_folder}}").timeout(5000)
> .log("Received file: ${file:name}"); {code}
> I have collected HeapDump and as I can see, Camel tries to convert file content to String
> {code:java}
> at java.lang.OutOfMemoryError.<init>(<unresolved string 0x0>)
> at java.util.Arrays.copyOf(<unresolved string 0x0>)
> local variable: byte[]#206926
> at java.lang.AbstractStringBuilder.ensureCapacityInternal(<unresolved string 0x0>)
> local variable: java.lang.StringBuilder#70
> at java.lang.AbstractStringBuilder.append(<unresolved string 0x0>)
> local variable: java.lang.StringBuilder#70
> local variable: char[]#3191
> at java.lang.StringBuilder.append(<unresolved string 0x0>)
> local variable: java.lang.StringBuilder#70
> at org.apache.camel.util.IOHelper.toString(IOHelper.java:142)
> local variable: java.io.BufferedReader#1
> local variable: java.lang.StringBuilder#70
> local variable: char[]#3191
> at org.apache.camel.util.IOHelper.toString(IOHelper.java:125)
> local variable: java.io.BufferedReader#1
> at org.apache.camel.util.IOHelper.toString(IOHelper.java:121)
> local variable: java.io.BufferedReader#1
> at org.apache.camel.component.file.GenericFileConverter.genericFileToString(GenericFileConverter.java:149)
> local variable: org.apache.camel.component.file.GenericFile#1
> local variable: org.apache.camel.support.DefaultExchange#1
> local variable: java.io.BufferedReader#1
> at org.apache.camel.component.file.GenericFileConverterLoader.lambda$registerConverters$3(GenericFileConverterLoader.java:52)
> local variable: class java.lang.String
> local variable: org.apache.camel.support.DefaultExchange#1
> local variable: org.apache.camel.component.file.GenericFile#1
> at org.apache.camel.component.file.GenericFileConverterLoader$$Lambda$1150+0x00007f36829dbc80.doConvert(<unresolved string 0x0>)
> local variable: org.apache.camel.component.file.GenericFileConverterLoader$$Lambda$1150+0x00007f36829dbc80#1
> local variable: class java.lang.String
> local variable: org.apache.camel.support.DefaultExchange#1
> local variable: org.apache.camel.component.file.GenericFile#1
> at org.apache.camel.support.SimpleTypeConverter.convertTo(SimpleTypeConverter.java:101)
> local variable: class java.lang.String
> local variable: org.apache.camel.component.file.GenericFile#1
> at org.apache.camel.support.SimpleTypeConverter.tryConvertTo(SimpleTypeConverter.java:90)
> at org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvertTo(CoreTypeConverterRegistry.java:477)
> local variable: org.apache.camel.impl.converter.DefaultTypeConverter#1
> local variable: class java.lang.String
> local variable: org.apache.camel.support.DefaultExchange#1
> local variable: org.apache.camel.component.file.GenericFile#1
> local variable: org.apache.camel.support.SimpleTypeConverter#26
> at org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvertTo(CoreTypeConverterRegistry.java:358)
> local variable: org.apache.camel.impl.converter.DefaultTypeConverter#1
> local variable: class java.lang.String
> local variable: org.apache.camel.support.DefaultExchange#1
> local variable: org.apache.camel.component.file.GenericFile#1
> at org.apache.camel.impl.converter.CoreTypeConverterRegistry.tryConvertTo(CoreTypeConverterRegistry.java:347)
> at org.apache.camel.support.MessageHelper.extractValueForLogging(MessageHelper.java:386)
> local variable: org.apache.camel.component.file.GenericFile#1
> local variable: org.apache.camel.component.jms.JmsMessage#1
> at org.apache.camel.support.MessageHelper.extractValueForLogging(MessageHelper.java:316)
> local variable: org.apache.camel.component.file.GenericFile#1
> local variable: org.apache.camel.component.jms.JmsMessage#1
> at org.apache.camel.support.MessageHelper.extractBodyForLogging(MessageHelper.java:265)
> local variable: org.apache.camel.component.jms.JmsMessage#1
> at org.apache.camel.support.MessageHelper.dumpAsXml(MessageHelper.java:528)
> local variable: org.apache.camel.component.jms.JmsMessage#1
> local variable: java.lang.StringBuilder#71
> local variable: java.lang.StringBuilder#72
> local variable: java.lang.String#70214
> at org.apache.camel.impl.engine.CamelInternalProcessor$BacklogTracerAdvice.before(CamelInternalProcessor.java:587)
> local variable: org.apache.camel.impl.engine.CamelInternalProcessor$BacklogTracerAdvice#71
> local variable: org.apache.camel.support.DefaultExchange#1
> local variable: java.lang.String#45325
> local variable: java.lang.String#189314
> at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:335)
> local variable: org.apache.camel.impl.engine.DefaultChannel#70
> local variable: org.apache.camel.support.DefaultExchange#1
> local variable: org.apache.camel.processor.Pipeline$PipelineTask#1
> local variable: java.lang.Object[]#28303
> local variable: org.apache.camel.impl.engine.CamelInternalProcessor$AsyncAfterTask#1
> local variable: org.apache.camel.impl.engine.CamelInternalProcessor$BacklogTracerAdvice#71
> at org.apache.camel.processor.Pipeline$PipelineTask.run(Pipeline.java:104)
> local variable: org.apache.camel.processor.Pipeline$PipelineTask#1
> local variable: org.apache.camel.impl.engine.DefaultChannel#70
> at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:181)
> local variable: org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker#7
> at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:59)
> local variable: org.apache.camel.impl.engine.DefaultReactiveExecutor#1
> local variable: org.apache.camel.processor.Pipeline$PipelineTask#1
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:165)
> local variable: org.apache.camel.processor.RoutePipeline#14
> local variable: org.apache.camel.support.DefaultExchange#1
> local variable: org.apache.camel.impl.engine.CamelInternalProcessor$AsyncAfterTask#2
> local variable: org.apache.camel.processor.Pipeline$PipelineTask#1
> at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:392)
> local variable: org.apache.camel.impl.engine.CamelInternalProcessor#19
> local variable: org.apache.camel.support.DefaultExchange#1
> local variable: org.apache.camel.impl.engine.DefaultAsyncProcessorAwaitManager$$Lambda$1773+0x00007f3683025fc8#1
> local variable: java.lang.Object[]#28304
> local variable: org.apache.camel.impl.engine.CamelInternalProcessor$AsyncAfterTask#2
> local variable: org.apache.camel.impl.engine.DefaultUnitOfWork#1
> local variable: org.apache.camel.impl.engine.CamelInternalProcessor$AsyncAfterTask#2
> at org.apache.camel.impl.engine.DefaultAsyncProcessorAwaitManager.process(DefaultAsyncProcessorAwaitManager.java:83)
> local variable: org.apache.camel.impl.engine.DefaultAsyncProcessorAwaitManager#1
> local variable: org.apache.camel.impl.engine.CamelInternalProcessor#19
> local variable: org.apache.camel.support.DefaultExchange#1
> local variable: java.util.concurrent.CountDownLatch#29
> at org.apache.camel.support.AsyncProcessorSupport.process(AsyncProcessorSupport.java:41)
> local variable: org.apache.camel.impl.engine.CamelInternalProcessor#19
> local variable: org.apache.camel.support.DefaultExchange#1
> local variable: org.apache.camel.impl.engine.DefaultAsyncProcessorAwaitManager#1
> at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:132)
> local variable: org.apache.camel.component.jms.EndpointMessageListener#6
> local variable: org.apache.activemq.command.ActiveMQTextMessage#2
> local variable: org.apache.activemq.jms.pool.PooledSession#79
> local variable: org.apache.activemq.command.ActiveMQQueue#83
> local variable: org.apache.camel.support.DefaultExchange#1
> local variable: org.apache.camel.component.jms.EndpointMessageListener$EndpointMessageListenerAsyncCallback#1
> at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:736)
> local variable: org.apache.camel.component.jms.DefaultJmsMessageListenerContainer#6
> local variable: org.apache.camel.component.jms.EndpointMessageListener#6
> local variable: org.apache.activemq.jms.pool.PooledSession#79
> local variable: org.apache.activemq.command.ActiveMQTextMessage#2
> local variable: org.apache.activemq.jms.pool.PooledSession#79
> at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:696)
> local variable: org.apache.camel.component.jms.DefaultJmsMessageListenerContainer#6
> local variable: org.apache.activemq.jms.pool.PooledSession#79
> local variable: org.apache.activemq.command.ActiveMQTextMessage#2
> local variable: org.apache.camel.component.jms.EndpointMessageListener#6
> at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:674)
> local variable: org.apache.camel.component.jms.DefaultJmsMessageListenerContainer#6
> local variable: org.apache.activemq.jms.pool.PooledSession#79
> local variable: org.apache.activemq.command.ActiveMQTextMessage#2
> at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:331)
> local variable: org.apache.camel.component.jms.DefaultJmsMessageListenerContainer#6
> at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:270)
> at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1237)
> local variable: org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker#79
> at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1227)
> local variable: org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker#79
> local variable: java.lang.Object#11095
> at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1120)
> local variable: org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker#79
> at java.util.concurrent.ThreadPoolExecutor.runWorker(<unresolved string 0x0>)
> local variable: java.util.concurrent.ThreadPoolExecutor#7
> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#119
> local variable: java.lang.Thread#134
> local variable: org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker#79
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(<unresolved string 0x0>)
> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#119
> at java.lang.Thread.run(<unresolved string 0x0>)
> local variable: java.lang.Thread#134
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)