You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Vova Shelgunov <vv...@gmail.com> on 2016/10/19 12:00:22 UTC
OutOfMemoryError: Java heap space when sending large file to jetty endpoint
Hi,
I catched this OutOfMemoryError when I sent a 250 MB file to jetty endpoint:
java.lang.OutOfMemoryError: Java heap space
at java.lang.StringCoding.decode(StringCoding.java:215)
at java.lang.String.<init>(String.java:463)
at java.lang.String.<init>(String.java:515)
at
org.eclipse.jetty.servlets.MultiPartFilter$Wrapper.getParameterBytesAsString(MultiPartFilter.java:390)
at
org.eclipse.jetty.servlets.MultiPartFilter$Wrapper.getParameterValues(MultiPartFilter.java:337)
at
org.apache.camel.http.common.DefaultHttpBinding.populateRequestParameters(DefaultHttpBinding.java:219)
at
org.apache.camel.http.common.DefaultHttpBinding.readHeaders(DefaultHttpBinding.java:174)
at
org.apache.camel.http.common.DefaultHttpBinding.readRequest(DefaultHttpBinding.java:110)
at
org.apache.camel.http.common.HttpMessage.<init>(HttpMessage.java:52)
at
org.apache.camel.component.jetty.CamelContinuationServlet.doService(CamelContinuationServlet.java:161)
at
org.apache.camel.http.common.CamelServlet.service(CamelServlet.java:74)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
at
org.eclipse.jetty.servlets.MultiPartFilter.doFilter(MultiPartFilter.java:200)
at
org.apache.camel.component.jetty.CamelFilterWrapper.doFilter(CamelFilterWrapper.java:43)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
My configuration:
<route id="ds-rest-archive-upload-processor">
<from uri="jetty:
https://storage:8443/data/archive?sslContextParametersRef=sslContextParameters&requestBufferSize=32768&responseBufferSize=32768&httpMethodRestrict=POST
<https://storage:8443/data/archive?sslContextParametersRef=sslContextParameters&requestBufferSize=32768&responseBufferSize=32768&httpMethodRestrict=POST>"
/>
<to uri="bean:uploadFromArchiveFileProcessor?method=process"
pattern="InOut" />
</route>
<bean id="jetty"
class="org.apache.camel.component.jetty9.JettyHttpComponent9">
<property name="endpointClass"
value="org.apache.camel.component.jetty9.JettyHttpEndpoint9" />
<property name="sslContextParameters" ref="sslContextParameters" />
<property name="useContinuation" value="true" />
<property name="requestBufferSize" value="32768" />
<property name="responseBufferSize" value="32768" />
<property name="threadPool">
<bean class="org.eclipse.jetty.util.thread.QueuedThreadPool">
<constructor-arg index="0" value="1000" />
</bean>
</property>
<property name="jettyHttpBinding">
<bean class="org.apache.camel.component.jetty.DefaultJettyHttpBinding">
<property name="transferException" value="false" />
</bean>
</property>
</bean>
My camel application is running in docker container with limited amount of
memory, and I did not expect, that a file will be read a string instead of
reading from InputStream.
Is there any way to consume large files with small amount of memory?
Regards,
Uladzimir
Re: OutOfMemoryError: Java heap space when sending large file to
jetty endpoint
Posted by Tomohisa Igarashi <tm...@gmail.com>.
Yes if you can see file content is actually loaded in populateRequestParameters(), I think it's likely to be a bug. Please make sure it's not just a file name, and update the description with what you found. If you can attach a minimal reproducer project that would be better.
On 10/20/2016 07:35 PM, Vova Shelgunov wrote:
> I can add a description that parameter is read twice.
>
> 2016-10-20 13:34 GMT+03:00 Vova Shelgunov <vv...@gmail.com>:
>
>> *Tomohisa*, actually initially I posted a bug in Jira (
>> https://issues.apache.org/jira/browse/CAMEL-10399), but Claus closed it,
>> can I reopen it?
>>
>> 2016-10-20 10:49 GMT+03:00 Tomohisa Igarashi <tm...@gmail.com>:
>>
>>> Hmm that smells like a bug in DefaultHttpBinding.populateRequestParameters(),
>>> it shouldn't read a parameter value which is already handled in
>>> populateAttachments(). Would you file a JIRA for it?
>>>
>>> On 10/20/2016 03:18 PM, Vova Shelgunov wrote:
>>>
>>>> Hi,
>>>>
>>>> I tried to debug populateRequestParameters in DefaultHttpBinding and
>>>> noticed that request parameters contain file (with byte[] array value) that
>>>> I am sending to endpoint:
>>>>
>>>>
>>>> \u200b
>>>> \u200b
>>>> It is a reason why I have OOM error.
>>>>
>>>> Let me note that I am sending file using google http client:
>>>>
>>>>
>>>> I attached MultipartFormDataContent.
>>>>
>>>> As for allowJavaSerializedObject I did not change default value (it is
>>>> false).
>>>>
>>>> 2016-10-20 4:24 GMT+03:00 Tomohisa Igarashi <tm.igarashi@gmail.com
>>>> <ma...@gmail.com>>:
>>>>
>>>> Hi,
>>>>
>>>> It seems like the OOM occurs when it's mapping request parameters,
>>>> but not request body which is passed as an InputStream by default. Do you
>>>> have any large parameter other than body?
>>>> Note that if you enable allowJavaSerializedObject, then body is
>>>> deseriallized as a Java Object, may cause OOM.
>>>>
>>>> Thanks,
>>>> Tomo
>>>>
>>>>
>>>> On 10/19/2016 09:39 PM, Vova Shelgunov wrote:
>>>>
>>>> Hi Andrea,
>>>>
>>>> Yes, I run application using jar file. Java has 1G heap.
>>>>
>>>> Actually I assumed that camel will not read file in single
>>>> string, but will
>>>> use InputStream to read the data.
>>>>
>>>> 2016-10-19 15:28 GMT+03:00 Andrea Cosentino
>>>> <an...@yahoo.com.invalid>:
>>>>
>>>> Is the camel-context deployed as a simple JAR? What are you
>>>> using inside
>>>> the Debian container? How much memory has your container?
>>>> Did you set JVM_OPTS witch specific values for max Heap size
>>>> and min Heap
>>>> size?
>>>>
>>>> --
>>>> Andrea Cosentino
>>>> ----------------------------------
>>>> Apache Camel PMC Member
>>>> Apache Karaf Committer
>>>> Apache Servicemix Committer
>>>> Email: ancosen1985@yahoo.com <ma...@yahoo.com>
>>>> Twitter: @oscerd2
>>>> Github: oscerd
>>>>
>>>>
>>>>
>>>> On Wednesday, October 19, 2016 2:00 PM, Vova Shelgunov <
>>>> vvshvv@gmail.com <ma...@gmail.com>>
>>>>
>>>> wrote:
>>>> Hi,
>>>>
>>>> I catched this OutOfMemoryError when I sent a 250 MB file to
>>>> jetty
>>>> endpoint:
>>>>
>>>> java.lang.OutOfMemoryError: Java heap space
>>>> at java.lang.StringCoding.decode(
>>>> StringCoding.java:215)
>>>> at java.lang.String.<init>(String.java:463)
>>>> at java.lang.String.<init>(String.java:515)
>>>> at
>>>> org.eclipse.jetty.servlets.MultiPartFilter$Wrapper.
>>>> getParameterBytesAsString(MultiPartFilter.java:390)
>>>> at
>>>> org.eclipse.jetty.servlets.MultiPartFilter$Wrapper.getParame
>>>> terValues(
>>>> MultiPartFilter.java:337)
>>>> at
>>>> org.apache.camel.http.common.DefaultHttpBinding.populateRequ
>>>> estParameters(
>>>> DefaultHttpBinding.java:219)
>>>> at
>>>> org.apache.camel.http.common.DefaultHttpBinding.readHeaders(
>>>> DefaultHttpBinding.java:174)
>>>> at
>>>> org.apache.camel.http.common.DefaultHttpBinding.readRequest(
>>>> DefaultHttpBinding.java:110)
>>>> at
>>>> org.apache.camel.http.common.HttpMessage.<init>(HttpMessage.
>>>> java:52)
>>>> at
>>>> org.apache.camel.component.jetty.CamelContinuationServlet.do
>>>> <http://ty.CamelContinuationServlet.do>Service(
>>>>
>>>> CamelContinuationServlet.java:161)
>>>> at
>>>> org.apache.camel.http.common.CamelServlet.service(CamelServl
>>>> et.java:74)
>>>> at javax.servlet.http.HttpServlet
>>>> .service(HttpServlet.java:790)
>>>> at
>>>> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder
>>>> .java:812)
>>>> at
>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
>>>> doFilter(ServletHandler.java:1669)
>>>> at
>>>> org.eclipse.jetty.servlets.MultiPartFilter.doFilter(
>>>> MultiPartFilter.java:200)
>>>> at
>>>> org.apache.camel.component.jetty.CamelFilterWrapper.
>>>> doFilter(CamelFilterWrapper.java:43)
>>>> at
>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
>>>> doFilter(ServletHandler.java:1652)
>>>> at
>>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHan
>>>> dler.java:585)
>>>> at
>>>> org.eclipse.jetty.server.handler.ContextHandler.
>>>> doHandle(ContextHandler.java:1127)
>>>> at
>>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHand
>>>> ler.java:515)
>>>> at
>>>> org.eclipse.jetty.server.handler.ContextHandler.
>>>> doScope(ContextHandler.java:1061)
>>>> at
>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(
>>>> ScopedHandler.java:141)
>>>> at
>>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(
>>>> HandlerWrapper.java:97)
>>>> at org.eclipse.jetty.server.Serve
>>>> r.handle(Server.java:499)
>>>> at org.eclipse.jetty.server.HttpC
>>>> hannel.handle(HttpChannel.java:311)
>>>> at
>>>> org.eclipse.jetty.server.HttpConnection.onFillable(
>>>> HttpConnection.java:257)
>>>> at
>>>> org.eclipse.jetty.io <http://org.eclipse.jetty.io>.
>>>> AbstractConnection$2.run(AbstractConnection.java:544)
>>>> at
>>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
>>>> QueuedThreadPool.java:635)
>>>> at
>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
>>>> QueuedThreadPool.java:555)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>> My configuration:
>>>>
>>>> <route id="ds-rest-archive-upload-processor">
>>>> <from uri="jetty:
>>>> https://storage:8443/data/archive?sslContextParametersRef= <
>>>> https://storage:8443/data/archive?sslContextParametersRef=>
>>>> sslContextParameters&requestBufferSize=32768&responseBufferS
>>>> ize=32768&
>>>> httpMethodRestrict=POST
>>>> <https://storage:8443/data/archive?sslContextParametersRef=
>>>> <https://storage:8443/data/archive?sslContextParametersRef=>
>>>>
>>>> sslContextParameters&requestBufferSize=32768&responseBufferS
>>>> ize=32768&
>>>> httpMethodRestrict=POST>"
>>>> />
>>>>
>>>> <to uri="bean:uploadFromArchiveFileProcessor?method=process"
>>>> pattern="InOut" />
>>>>
>>>> </route>
>>>>
>>>> <bean id="jetty"
>>>> class="org.apache.camel.component.jetty9.JettyHttpComponent9
>>>> ">
>>>> <property name="endpointClass"
>>>> value="org.apache.camel.component.jetty9.JettyHttpEndpoint9"
>>>> />
>>>> <property name="sslContextParameters"
>>>> ref="sslContextParameters" />
>>>> <property name="useContinuation" value="true" />
>>>> <property name="requestBufferSize" value="32768" />
>>>> <property name="responseBufferSize" value="32768" />
>>>>
>>>> <property name="threadPool">
>>>> <bean class="org.eclipse.jetty.util.
>>>> thread.QueuedThreadPool">
>>>> <constructor-arg index="0" value="1000" />
>>>> </bean>
>>>> </property>
>>>>
>>>> <property name="jettyHttpBinding">
>>>> <bean class="org.apache.camel.compon
>>>> ent.jetty.DefaultJettyHttpBinding">
>>>> <property name="transferException" value="false" />
>>>> </bean>
>>>> </property>
>>>> </bean>
>>>>
>>>> My camel application is running in docker container with
>>>> limited amount of
>>>> memory, and I did not expect, that a file will be read a
>>>> string instead of
>>>> reading from InputStream.
>>>>
>>>> Is there any way to consume large files with small amount of
>>>> memory?
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Uladzimir
>>>>
>>>>
>>>>
>>>>
>>
>
Re: OutOfMemoryError: Java heap space when sending large file to
jetty endpoint
Posted by Vova Shelgunov <vv...@gmail.com>.
I can add a description that parameter is read twice.
2016-10-20 13:34 GMT+03:00 Vova Shelgunov <vv...@gmail.com>:
> *Tomohisa*, actually initially I posted a bug in Jira (
> https://issues.apache.org/jira/browse/CAMEL-10399), but Claus closed it,
> can I reopen it?
>
> 2016-10-20 10:49 GMT+03:00 Tomohisa Igarashi <tm...@gmail.com>:
>
>> Hmm that smells like a bug in DefaultHttpBinding.populateRequestParameters(),
>> it shouldn't read a parameter value which is already handled in
>> populateAttachments(). Would you file a JIRA for it?
>>
>> On 10/20/2016 03:18 PM, Vova Shelgunov wrote:
>>
>>> Hi,
>>>
>>> I tried to debug populateRequestParameters in DefaultHttpBinding and
>>> noticed that request parameters contain file (with byte[] array value) that
>>> I am sending to endpoint:
>>>
>>>
>>>
>>>
>>> It is a reason why I have OOM error.
>>>
>>> Let me note that I am sending file using google http client:
>>>
>>>
>>> I attached MultipartFormDataContent.
>>>
>>> As for allowJavaSerializedObject I did not change default value (it is
>>> false).
>>>
>>> 2016-10-20 4:24 GMT+03:00 Tomohisa Igarashi <tm.igarashi@gmail.com
>>> <ma...@gmail.com>>:
>>>
>>> Hi,
>>>
>>> It seems like the OOM occurs when it's mapping request parameters,
>>> but not request body which is passed as an InputStream by default. Do you
>>> have any large parameter other than body?
>>> Note that if you enable allowJavaSerializedObject, then body is
>>> deseriallized as a Java Object, may cause OOM.
>>>
>>> Thanks,
>>> Tomo
>>>
>>>
>>> On 10/19/2016 09:39 PM, Vova Shelgunov wrote:
>>>
>>> Hi Andrea,
>>>
>>> Yes, I run application using jar file. Java has 1G heap.
>>>
>>> Actually I assumed that camel will not read file in single
>>> string, but will
>>> use InputStream to read the data.
>>>
>>> 2016-10-19 15:28 GMT+03:00 Andrea Cosentino
>>> <an...@yahoo.com.invalid>:
>>>
>>> Is the camel-context deployed as a simple JAR? What are you
>>> using inside
>>> the Debian container? How much memory has your container?
>>> Did you set JVM_OPTS witch specific values for max Heap size
>>> and min Heap
>>> size?
>>>
>>> --
>>> Andrea Cosentino
>>> ----------------------------------
>>> Apache Camel PMC Member
>>> Apache Karaf Committer
>>> Apache Servicemix Committer
>>> Email: ancosen1985@yahoo.com <ma...@yahoo.com>
>>> Twitter: @oscerd2
>>> Github: oscerd
>>>
>>>
>>>
>>> On Wednesday, October 19, 2016 2:00 PM, Vova Shelgunov <
>>> vvshvv@gmail.com <ma...@gmail.com>>
>>>
>>> wrote:
>>> Hi,
>>>
>>> I catched this OutOfMemoryError when I sent a 250 MB file to
>>> jetty
>>> endpoint:
>>>
>>> java.lang.OutOfMemoryError: Java heap space
>>> at java.lang.StringCoding.decode(
>>> StringCoding.java:215)
>>> at java.lang.String.<init>(String.java:463)
>>> at java.lang.String.<init>(String.java:515)
>>> at
>>> org.eclipse.jetty.servlets.MultiPartFilter$Wrapper.
>>> getParameterBytesAsString(MultiPartFilter.java:390)
>>> at
>>> org.eclipse.jetty.servlets.MultiPartFilter$Wrapper.getParame
>>> terValues(
>>> MultiPartFilter.java:337)
>>> at
>>> org.apache.camel.http.common.DefaultHttpBinding.populateRequ
>>> estParameters(
>>> DefaultHttpBinding.java:219)
>>> at
>>> org.apache.camel.http.common.DefaultHttpBinding.readHeaders(
>>> DefaultHttpBinding.java:174)
>>> at
>>> org.apache.camel.http.common.DefaultHttpBinding.readRequest(
>>> DefaultHttpBinding.java:110)
>>> at
>>> org.apache.camel.http.common.HttpMessage.<init>(HttpMessage.
>>> java:52)
>>> at
>>> org.apache.camel.component.jetty.CamelContinuationServlet.do
>>> <http://ty.CamelContinuationServlet.do>Service(
>>>
>>> CamelContinuationServlet.java:161)
>>> at
>>> org.apache.camel.http.common.CamelServlet.service(CamelServl
>>> et.java:74)
>>> at javax.servlet.http.HttpServlet
>>> .service(HttpServlet.java:790)
>>> at
>>> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder
>>> .java:812)
>>> at
>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
>>> doFilter(ServletHandler.java:1669)
>>> at
>>> org.eclipse.jetty.servlets.MultiPartFilter.doFilter(
>>> MultiPartFilter.java:200)
>>> at
>>> org.apache.camel.component.jetty.CamelFilterWrapper.
>>> doFilter(CamelFilterWrapper.java:43)
>>> at
>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
>>> doFilter(ServletHandler.java:1652)
>>> at
>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHan
>>> dler.java:585)
>>> at
>>> org.eclipse.jetty.server.handler.ContextHandler.
>>> doHandle(ContextHandler.java:1127)
>>> at
>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHand
>>> ler.java:515)
>>> at
>>> org.eclipse.jetty.server.handler.ContextHandler.
>>> doScope(ContextHandler.java:1061)
>>> at
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(
>>> ScopedHandler.java:141)
>>> at
>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(
>>> HandlerWrapper.java:97)
>>> at org.eclipse.jetty.server.Serve
>>> r.handle(Server.java:499)
>>> at org.eclipse.jetty.server.HttpC
>>> hannel.handle(HttpChannel.java:311)
>>> at
>>> org.eclipse.jetty.server.HttpConnection.onFillable(
>>> HttpConnection.java:257)
>>> at
>>> org.eclipse.jetty.io <http://org.eclipse.jetty.io>.
>>> AbstractConnection$2.run(AbstractConnection.java:544)
>>> at
>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
>>> QueuedThreadPool.java:635)
>>> at
>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
>>> QueuedThreadPool.java:555)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> My configuration:
>>>
>>> <route id="ds-rest-archive-upload-processor">
>>> <from uri="jetty:
>>> https://storage:8443/data/archive?sslContextParametersRef= <
>>> https://storage:8443/data/archive?sslContextParametersRef=>
>>> sslContextParameters&requestBufferSize=32768&responseBufferS
>>> ize=32768&
>>> httpMethodRestrict=POST
>>> <https://storage:8443/data/archive?sslContextParametersRef=
>>> <https://storage:8443/data/archive?sslContextParametersRef=>
>>>
>>> sslContextParameters&requestBufferSize=32768&responseBufferS
>>> ize=32768&
>>> httpMethodRestrict=POST>"
>>> />
>>>
>>> <to uri="bean:uploadFromArchiveFileProcessor?method=process"
>>> pattern="InOut" />
>>>
>>> </route>
>>>
>>> <bean id="jetty"
>>> class="org.apache.camel.component.jetty9.JettyHttpComponent9
>>> ">
>>> <property name="endpointClass"
>>> value="org.apache.camel.component.jetty9.JettyHttpEndpoint9"
>>> />
>>> <property name="sslContextParameters"
>>> ref="sslContextParameters" />
>>> <property name="useContinuation" value="true" />
>>> <property name="requestBufferSize" value="32768" />
>>> <property name="responseBufferSize" value="32768" />
>>>
>>> <property name="threadPool">
>>> <bean class="org.eclipse.jetty.util.
>>> thread.QueuedThreadPool">
>>> <constructor-arg index="0" value="1000" />
>>> </bean>
>>> </property>
>>>
>>> <property name="jettyHttpBinding">
>>> <bean class="org.apache.camel.compon
>>> ent.jetty.DefaultJettyHttpBinding">
>>> <property name="transferException" value="false" />
>>> </bean>
>>> </property>
>>> </bean>
>>>
>>> My camel application is running in docker container with
>>> limited amount of
>>> memory, and I did not expect, that a file will be read a
>>> string instead of
>>> reading from InputStream.
>>>
>>> Is there any way to consume large files with small amount of
>>> memory?
>>>
>>>
>>> Regards,
>>>
>>> Uladzimir
>>>
>>>
>>>
>>>
>
Re: OutOfMemoryError: Java heap space when sending large file to
jetty endpoint
Posted by Vova Shelgunov <vv...@gmail.com>.
*Tomohisa*, actually initially I posted a bug in Jira (
https://issues.apache.org/jira/browse/CAMEL-10399), but Claus closed it,
can I reopen it?
2016-10-20 10:49 GMT+03:00 Tomohisa Igarashi <tm...@gmail.com>:
> Hmm that smells like a bug in DefaultHttpBinding.populateRequestParameters(),
> it shouldn't read a parameter value which is already handled in
> populateAttachments(). Would you file a JIRA for it?
>
> On 10/20/2016 03:18 PM, Vova Shelgunov wrote:
>
>> Hi,
>>
>> I tried to debug populateRequestParameters in DefaultHttpBinding and
>> noticed that request parameters contain file (with byte[] array value) that
>> I am sending to endpoint:
>>
>>
>>
>>
>> It is a reason why I have OOM error.
>>
>> Let me note that I am sending file using google http client:
>>
>>
>> I attached MultipartFormDataContent.
>>
>> As for allowJavaSerializedObject I did not change default value (it is
>> false).
>>
>> 2016-10-20 4:24 GMT+03:00 Tomohisa Igarashi <tm.igarashi@gmail.com
>> <ma...@gmail.com>>:
>>
>> Hi,
>>
>> It seems like the OOM occurs when it's mapping request parameters,
>> but not request body which is passed as an InputStream by default. Do you
>> have any large parameter other than body?
>> Note that if you enable allowJavaSerializedObject, then body is
>> deseriallized as a Java Object, may cause OOM.
>>
>> Thanks,
>> Tomo
>>
>>
>> On 10/19/2016 09:39 PM, Vova Shelgunov wrote:
>>
>> Hi Andrea,
>>
>> Yes, I run application using jar file. Java has 1G heap.
>>
>> Actually I assumed that camel will not read file in single
>> string, but will
>> use InputStream to read the data.
>>
>> 2016-10-19 15:28 GMT+03:00 Andrea Cosentino
>> <an...@yahoo.com.invalid>:
>>
>> Is the camel-context deployed as a simple JAR? What are you
>> using inside
>> the Debian container? How much memory has your container?
>> Did you set JVM_OPTS witch specific values for max Heap size
>> and min Heap
>> size?
>>
>> --
>> Andrea Cosentino
>> ----------------------------------
>> Apache Camel PMC Member
>> Apache Karaf Committer
>> Apache Servicemix Committer
>> Email: ancosen1985@yahoo.com <ma...@yahoo.com>
>> Twitter: @oscerd2
>> Github: oscerd
>>
>>
>>
>> On Wednesday, October 19, 2016 2:00 PM, Vova Shelgunov <
>> vvshvv@gmail.com <ma...@gmail.com>>
>>
>> wrote:
>> Hi,
>>
>> I catched this OutOfMemoryError when I sent a 250 MB file to
>> jetty
>> endpoint:
>>
>> java.lang.OutOfMemoryError: Java heap space
>> at java.lang.StringCoding.decode(StringCoding.java:215)
>> at java.lang.String.<init>(String.java:463)
>> at java.lang.String.<init>(String.java:515)
>> at
>> org.eclipse.jetty.servlets.MultiPartFilter$Wrapper.
>> getParameterBytesAsString(MultiPartFilter.java:390)
>> at
>> org.eclipse.jetty.servlets.MultiPartFilter$Wrapper.getParame
>> terValues(
>> MultiPartFilter.java:337)
>> at
>> org.apache.camel.http.common.DefaultHttpBinding.populateRequ
>> estParameters(
>> DefaultHttpBinding.java:219)
>> at
>> org.apache.camel.http.common.DefaultHttpBinding.readHeaders(
>> DefaultHttpBinding.java:174)
>> at
>> org.apache.camel.http.common.DefaultHttpBinding.readRequest(
>> DefaultHttpBinding.java:110)
>> at
>> org.apache.camel.http.common.HttpMessage.<init>(HttpMessage.
>> java:52)
>> at
>> org.apache.camel.component.jetty.CamelContinuationServlet.do
>> <http://ty.CamelContinuationServlet.do>Service(
>>
>> CamelContinuationServlet.java:161)
>> at
>> org.apache.camel.http.common.CamelServlet.service(CamelServl
>> et.java:74)
>> at javax.servlet.http.HttpServlet
>> .service(HttpServlet.java:790)
>> at
>> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder
>> .java:812)
>> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
>> doFilter(ServletHandler.java:1669)
>> at
>> org.eclipse.jetty.servlets.MultiPartFilter.doFilter(
>> MultiPartFilter.java:200)
>> at
>> org.apache.camel.component.jetty.CamelFilterWrapper.
>> doFilter(CamelFilterWrapper.java:43)
>> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
>> doFilter(ServletHandler.java:1652)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHan
>> dler.java:585)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.
>> doHandle(ContextHandler.java:1127)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHand
>> ler.java:515)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.
>> doScope(ContextHandler.java:1061)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(
>> ScopedHandler.java:141)
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(
>> HandlerWrapper.java:97)
>> at org.eclipse.jetty.server.Serve
>> r.handle(Server.java:499)
>> at org.eclipse.jetty.server.HttpC
>> hannel.handle(HttpChannel.java:311)
>> at
>> org.eclipse.jetty.server.HttpConnection.onFillable(
>> HttpConnection.java:257)
>> at
>> org.eclipse.jetty.io <http://org.eclipse.jetty.io>.
>> AbstractConnection$2.run(AbstractConnection.java:544)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
>> QueuedThreadPool.java:635)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
>> QueuedThreadPool.java:555)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> My configuration:
>>
>> <route id="ds-rest-archive-upload-processor">
>> <from uri="jetty:
>> https://storage:8443/data/archive?sslContextParametersRef= <
>> https://storage:8443/data/archive?sslContextParametersRef=>
>> sslContextParameters&requestBufferSize=32768&responseBufferS
>> ize=32768&
>> httpMethodRestrict=POST
>> <https://storage:8443/data/archive?sslContextParametersRef= <
>> https://storage:8443/data/archive?sslContextParametersRef=>
>>
>> sslContextParameters&requestBufferSize=32768&responseBufferS
>> ize=32768&
>> httpMethodRestrict=POST>"
>> />
>>
>> <to uri="bean:uploadFromArchiveFileProcessor?method=process"
>> pattern="InOut" />
>>
>> </route>
>>
>> <bean id="jetty"
>> class="org.apache.camel.component.jetty9.JettyHttpComponent9
>> ">
>> <property name="endpointClass"
>> value="org.apache.camel.component.jetty9.JettyHttpEndpoint9"
>> />
>> <property name="sslContextParameters"
>> ref="sslContextParameters" />
>> <property name="useContinuation" value="true" />
>> <property name="requestBufferSize" value="32768" />
>> <property name="responseBufferSize" value="32768" />
>>
>> <property name="threadPool">
>> <bean class="org.eclipse.jetty.util.thread.QueuedThreadPool">
>> <constructor-arg index="0" value="1000" />
>> </bean>
>> </property>
>>
>> <property name="jettyHttpBinding">
>> <bean class="org.apache.camel.compon
>> ent.jetty.DefaultJettyHttpBinding">
>> <property name="transferException" value="false" />
>> </bean>
>> </property>
>> </bean>
>>
>> My camel application is running in docker container with
>> limited amount of
>> memory, and I did not expect, that a file will be read a
>> string instead of
>> reading from InputStream.
>>
>> Is there any way to consume large files with small amount of
>> memory?
>>
>>
>> Regards,
>>
>> Uladzimir
>>
>>
>>
>>
Re: OutOfMemoryError: Java heap space when sending large file to
jetty endpoint
Posted by Tomohisa Igarashi <tm...@gmail.com>.
Hmm that smells like a bug in DefaultHttpBinding.populateRequestParameters(), it shouldn't read a parameter value which is already handled in populateAttachments(). Would you file a JIRA for it?
On 10/20/2016 03:18 PM, Vova Shelgunov wrote:
> Hi,
>
> I tried to debug populateRequestParameters in DefaultHttpBinding and noticed that request parameters contain file (with byte[] array value) that I am sending to endpoint:
>
>
> \u200b
> \u200b
> It is a reason why I have OOM error.
>
> Let me note that I am sending file using google http client:
>
>
> I attached MultipartFormDataContent.
>
> As for allowJavaSerializedObject I did not change default value (it is false).
>
> 2016-10-20 4:24 GMT+03:00 Tomohisa Igarashi <tm.igarashi@gmail.com <ma...@gmail.com>>:
>
> Hi,
>
> It seems like the OOM occurs when it's mapping request parameters, but not request body which is passed as an InputStream by default. Do you have any large parameter other than body?
> Note that if you enable allowJavaSerializedObject, then body is deseriallized as a Java Object, may cause OOM.
>
> Thanks,
> Tomo
>
>
> On 10/19/2016 09:39 PM, Vova Shelgunov wrote:
>
> Hi Andrea,
>
> Yes, I run application using jar file. Java has 1G heap.
>
> Actually I assumed that camel will not read file in single string, but will
> use InputStream to read the data.
>
> 2016-10-19 15:28 GMT+03:00 Andrea Cosentino <an...@yahoo.com.invalid>:
>
> Is the camel-context deployed as a simple JAR? What are you using inside
> the Debian container? How much memory has your container?
> Did you set JVM_OPTS witch specific values for max Heap size and min Heap
> size?
>
> --
> Andrea Cosentino
> ----------------------------------
> Apache Camel PMC Member
> Apache Karaf Committer
> Apache Servicemix Committer
> Email: ancosen1985@yahoo.com <ma...@yahoo.com>
> Twitter: @oscerd2
> Github: oscerd
>
>
>
> On Wednesday, October 19, 2016 2:00 PM, Vova Shelgunov <vvshvv@gmail.com <ma...@gmail.com>>
> wrote:
> Hi,
>
> I catched this OutOfMemoryError when I sent a 250 MB file to jetty
> endpoint:
>
> java.lang.OutOfMemoryError: Java heap space
> at java.lang.StringCoding.decode(StringCoding.java:215)
> at java.lang.String.<init>(String.java:463)
> at java.lang.String.<init>(String.java:515)
> at
> org.eclipse.jetty.servlets.MultiPartFilter$Wrapper.
> getParameterBytesAsString(MultiPartFilter.java:390)
> at
> org.eclipse.jetty.servlets.MultiPartFilter$Wrapper.getParameterValues(
> MultiPartFilter.java:337)
> at
> org.apache.camel.http.common.DefaultHttpBinding.populateRequestParameters(
> DefaultHttpBinding.java:219)
> at
> org.apache.camel.http.common.DefaultHttpBinding.readHeaders(
> DefaultHttpBinding.java:174)
> at
> org.apache.camel.http.common.DefaultHttpBinding.readRequest(
> DefaultHttpBinding.java:110)
> at
> org.apache.camel.http.common.HttpMessage.<init>(HttpMessage.java:52)
> at
> org.apache.camel.component.jetty.CamelContinuationServlet.do <http://ty.CamelContinuationServlet.do>Service(
> CamelContinuationServlet.java:161)
> at
> org.apache.camel.http.common.CamelServlet.service(CamelServlet.java:74)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1669)
> at
> org.eclipse.jetty.servlets.MultiPartFilter.doFilter(
> MultiPartFilter.java:200)
> at
> org.apache.camel.component.jetty.CamelFilterWrapper.
> doFilter(CamelFilterWrapper.java:43)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1652)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> at
> org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1127)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at
> org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1061)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:257)
> at
> org.eclipse.jetty.io <http://org.eclipse.jetty.io>.AbstractConnection$2.run(AbstractConnection.java:544)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
> QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
>
> My configuration:
>
> <route id="ds-rest-archive-upload-processor">
> <from uri="jetty:
> https://storage:8443/data/archive?sslContextParametersRef= <https://storage:8443/data/archive?sslContextParametersRef=>
> sslContextParameters&requestBufferSize=32768&responseBufferSize=32768&
> httpMethodRestrict=POST
> <https://storage:8443/data/archive?sslContextParametersRef= <https://storage:8443/data/archive?sslContextParametersRef=>
> sslContextParameters&requestBufferSize=32768&responseBufferSize=32768&
> httpMethodRestrict=POST>"
> />
>
> <to uri="bean:uploadFromArchiveFileProcessor?method=process"
> pattern="InOut" />
>
> </route>
>
> <bean id="jetty"
> class="org.apache.camel.component.jetty9.JettyHttpComponent9">
> <property name="endpointClass"
> value="org.apache.camel.component.jetty9.JettyHttpEndpoint9" />
> <property name="sslContextParameters" ref="sslContextParameters" />
> <property name="useContinuation" value="true" />
> <property name="requestBufferSize" value="32768" />
> <property name="responseBufferSize" value="32768" />
>
> <property name="threadPool">
> <bean class="org.eclipse.jetty.util.thread.QueuedThreadPool">
> <constructor-arg index="0" value="1000" />
> </bean>
> </property>
>
> <property name="jettyHttpBinding">
> <bean class="org.apache.camel.component.jetty.DefaultJettyHttpBinding">
> <property name="transferException" value="false" />
> </bean>
> </property>
> </bean>
>
> My camel application is running in docker container with limited amount of
> memory, and I did not expect, that a file will be read a string instead of
> reading from InputStream.
>
> Is there any way to consume large files with small amount of memory?
>
>
> Regards,
>
> Uladzimir
>
>
>
Re: OutOfMemoryError: Java heap space when sending large file to
jetty endpoint
Posted by Vova Shelgunov <vv...@gmail.com>.
Hi,
I tried to debug populateRequestParameters in DefaultHttpBinding and
noticed that request parameters contain file (with byte[] array value) that
I am sending to endpoint:
It is a reason why I have OOM error.
Let me note that I am sending file using google http client:
I attached MultipartFormDataContent.
As for allowJavaSerializedObject I did not change default value (it is
false).
2016-10-20 4:24 GMT+03:00 Tomohisa Igarashi <tm...@gmail.com>:
> Hi,
>
> It seems like the OOM occurs when it's mapping request parameters, but not
> request body which is passed as an InputStream by default. Do you have any
> large parameter other than body?
> Note that if you enable allowJavaSerializedObject, then body is
> deseriallized as a Java Object, may cause OOM.
>
> Thanks,
> Tomo
>
>
> On 10/19/2016 09:39 PM, Vova Shelgunov wrote:
>
>> Hi Andrea,
>>
>> Yes, I run application using jar file. Java has 1G heap.
>>
>> Actually I assumed that camel will not read file in single string, but
>> will
>> use InputStream to read the data.
>>
>> 2016-10-19 15:28 GMT+03:00 Andrea Cosentino <ancosen1985@yahoo.com.invalid
>> >:
>>
>> Is the camel-context deployed as a simple JAR? What are you using inside
>>> the Debian container? How much memory has your container?
>>> Did you set JVM_OPTS witch specific values for max Heap size and min Heap
>>> size?
>>>
>>> --
>>> Andrea Cosentino
>>> ----------------------------------
>>> Apache Camel PMC Member
>>> Apache Karaf Committer
>>> Apache Servicemix Committer
>>> Email: ancosen1985@yahoo.com
>>> Twitter: @oscerd2
>>> Github: oscerd
>>>
>>>
>>>
>>> On Wednesday, October 19, 2016 2:00 PM, Vova Shelgunov <vvshvv@gmail.com
>>> >
>>> wrote:
>>> Hi,
>>>
>>> I catched this OutOfMemoryError when I sent a 250 MB file to jetty
>>> endpoint:
>>>
>>> java.lang.OutOfMemoryError: Java heap space
>>> at java.lang.StringCoding.decode(StringCoding.java:215)
>>> at java.lang.String.<init>(String.java:463)
>>> at java.lang.String.<init>(String.java:515)
>>> at
>>> org.eclipse.jetty.servlets.MultiPartFilter$Wrapper.
>>> getParameterBytesAsString(MultiPartFilter.java:390)
>>> at
>>> org.eclipse.jetty.servlets.MultiPartFilter$Wrapper.getParameterValues(
>>> MultiPartFilter.java:337)
>>> at
>>> org.apache.camel.http.common.DefaultHttpBinding.populateRequ
>>> estParameters(
>>> DefaultHttpBinding.java:219)
>>> at
>>> org.apache.camel.http.common.DefaultHttpBinding.readHeaders(
>>> DefaultHttpBinding.java:174)
>>> at
>>> org.apache.camel.http.common.DefaultHttpBinding.readRequest(
>>> DefaultHttpBinding.java:110)
>>> at
>>> org.apache.camel.http.common.HttpMessage.<init>(HttpMessage.java:52)
>>> at
>>> org.apache.camel.component.jetty.CamelContinuationServlet.doService(
>>> CamelContinuationServlet.java:161)
>>> at
>>> org.apache.camel.http.common.CamelServlet.service(CamelServlet.java:74)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>>> at
>>> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
>>> at
>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
>>> doFilter(ServletHandler.java:1669)
>>> at
>>> org.eclipse.jetty.servlets.MultiPartFilter.doFilter(
>>> MultiPartFilter.java:200)
>>> at
>>> org.apache.camel.component.jetty.CamelFilterWrapper.
>>> doFilter(CamelFilterWrapper.java:43)
>>> at
>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
>>> doFilter(ServletHandler.java:1652)
>>> at
>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHan
>>> dler.java:585)
>>> at
>>> org.eclipse.jetty.server.handler.ContextHandler.
>>> doHandle(ContextHandler.java:1127)
>>> at
>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHand
>>> ler.java:515)
>>> at
>>> org.eclipse.jetty.server.handler.ContextHandler.
>>> doScope(ContextHandler.java:1061)
>>> at
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(
>>> ScopedHandler.java:141)
>>> at
>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(
>>> HandlerWrapper.java:97)
>>> at org.eclipse.jetty.server.Server.handle(Server.java:499)
>>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.
>>> java:311)
>>> at
>>> org.eclipse.jetty.server.HttpConnection.onFillable(
>>> HttpConnection.java:257)
>>> at
>>> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnec
>>> tion.java:544)
>>> at
>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
>>> QueuedThreadPool.java:635)
>>> at
>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
>>> QueuedThreadPool.java:555)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> My configuration:
>>>
>>> <route id="ds-rest-archive-upload-processor">
>>> <from uri="jetty:
>>> https://storage:8443/data/archive?sslContextParametersRef=
>>> sslContextParameters&requestBufferSize=32768&responseBufferSize=32768&
>>> httpMethodRestrict=POST
>>> <https://storage:8443/data/archive?sslContextParametersRef=
>>> sslContextParameters&requestBufferSize=32768&responseBufferSize=32768&
>>> httpMethodRestrict=POST>"
>>> />
>>>
>>> <to uri="bean:uploadFromArchiveFileProcessor?method=process"
>>> pattern="InOut" />
>>>
>>> </route>
>>>
>>> <bean id="jetty"
>>> class="org.apache.camel.component.jetty9.JettyHttpComponent9">
>>> <property name="endpointClass"
>>> value="org.apache.camel.component.jetty9.JettyHttpEndpoint9" />
>>> <property name="sslContextParameters" ref="sslContextParameters" />
>>> <property name="useContinuation" value="true" />
>>> <property name="requestBufferSize" value="32768" />
>>> <property name="responseBufferSize" value="32768" />
>>>
>>> <property name="threadPool">
>>> <bean class="org.eclipse.jetty.util.thread.QueuedThreadPool">
>>> <constructor-arg index="0" value="1000" />
>>> </bean>
>>> </property>
>>>
>>> <property name="jettyHttpBinding">
>>> <bean class="org.apache.camel.component.jetty.DefaultJettyHttpBinding">
>>> <property name="transferException" value="false" />
>>> </bean>
>>> </property>
>>> </bean>
>>>
>>> My camel application is running in docker container with limited amount
>>> of
>>> memory, and I did not expect, that a file will be read a string instead
>>> of
>>> reading from InputStream.
>>>
>>> Is there any way to consume large files with small amount of memory?
>>>
>>>
>>> Regards,
>>>
>>> Uladzimir
>>>
>>>
>>
Re: OutOfMemoryError: Java heap space when sending large file to
jetty endpoint
Posted by Tomohisa Igarashi <tm...@gmail.com>.
Hi,
It seems like the OOM occurs when it's mapping request parameters, but not request body which is passed as an InputStream by default. Do you have any large parameter other than body?
Note that if you enable allowJavaSerializedObject, then body is deseriallized as a Java Object, may cause OOM.
Thanks,
Tomo
On 10/19/2016 09:39 PM, Vova Shelgunov wrote:
> Hi Andrea,
>
> Yes, I run application using jar file. Java has 1G heap.
>
> Actually I assumed that camel will not read file in single string, but will
> use InputStream to read the data.
>
> 2016-10-19 15:28 GMT+03:00 Andrea Cosentino <an...@yahoo.com.invalid>:
>
>> Is the camel-context deployed as a simple JAR? What are you using inside
>> the Debian container? How much memory has your container?
>> Did you set JVM_OPTS witch specific values for max Heap size and min Heap
>> size?
>>
>> --
>> Andrea Cosentino
>> ----------------------------------
>> Apache Camel PMC Member
>> Apache Karaf Committer
>> Apache Servicemix Committer
>> Email: ancosen1985@yahoo.com
>> Twitter: @oscerd2
>> Github: oscerd
>>
>>
>>
>> On Wednesday, October 19, 2016 2:00 PM, Vova Shelgunov <vv...@gmail.com>
>> wrote:
>> Hi,
>>
>> I catched this OutOfMemoryError when I sent a 250 MB file to jetty
>> endpoint:
>>
>> java.lang.OutOfMemoryError: Java heap space
>> at java.lang.StringCoding.decode(StringCoding.java:215)
>> at java.lang.String.<init>(String.java:463)
>> at java.lang.String.<init>(String.java:515)
>> at
>> org.eclipse.jetty.servlets.MultiPartFilter$Wrapper.
>> getParameterBytesAsString(MultiPartFilter.java:390)
>> at
>> org.eclipse.jetty.servlets.MultiPartFilter$Wrapper.getParameterValues(
>> MultiPartFilter.java:337)
>> at
>> org.apache.camel.http.common.DefaultHttpBinding.populateRequestParameters(
>> DefaultHttpBinding.java:219)
>> at
>> org.apache.camel.http.common.DefaultHttpBinding.readHeaders(
>> DefaultHttpBinding.java:174)
>> at
>> org.apache.camel.http.common.DefaultHttpBinding.readRequest(
>> DefaultHttpBinding.java:110)
>> at
>> org.apache.camel.http.common.HttpMessage.<init>(HttpMessage.java:52)
>> at
>> org.apache.camel.component.jetty.CamelContinuationServlet.doService(
>> CamelContinuationServlet.java:161)
>> at
>> org.apache.camel.http.common.CamelServlet.service(CamelServlet.java:74)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>> at
>> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
>> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
>> doFilter(ServletHandler.java:1669)
>> at
>> org.eclipse.jetty.servlets.MultiPartFilter.doFilter(
>> MultiPartFilter.java:200)
>> at
>> org.apache.camel.component.jetty.CamelFilterWrapper.
>> doFilter(CamelFilterWrapper.java:43)
>> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
>> doFilter(ServletHandler.java:1652)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.
>> doHandle(ContextHandler.java:1127)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.
>> doScope(ContextHandler.java:1061)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(
>> ScopedHandler.java:141)
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(
>> HandlerWrapper.java:97)
>> at org.eclipse.jetty.server.Server.handle(Server.java:499)
>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
>> at
>> org.eclipse.jetty.server.HttpConnection.onFillable(
>> HttpConnection.java:257)
>> at
>> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
>> QueuedThreadPool.java:635)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
>> QueuedThreadPool.java:555)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> My configuration:
>>
>> <route id="ds-rest-archive-upload-processor">
>> <from uri="jetty:
>> https://storage:8443/data/archive?sslContextParametersRef=
>> sslContextParameters&requestBufferSize=32768&responseBufferSize=32768&
>> httpMethodRestrict=POST
>> <https://storage:8443/data/archive?sslContextParametersRef=
>> sslContextParameters&requestBufferSize=32768&responseBufferSize=32768&
>> httpMethodRestrict=POST>"
>> />
>>
>> <to uri="bean:uploadFromArchiveFileProcessor?method=process"
>> pattern="InOut" />
>>
>> </route>
>>
>> <bean id="jetty"
>> class="org.apache.camel.component.jetty9.JettyHttpComponent9">
>> <property name="endpointClass"
>> value="org.apache.camel.component.jetty9.JettyHttpEndpoint9" />
>> <property name="sslContextParameters" ref="sslContextParameters" />
>> <property name="useContinuation" value="true" />
>> <property name="requestBufferSize" value="32768" />
>> <property name="responseBufferSize" value="32768" />
>>
>> <property name="threadPool">
>> <bean class="org.eclipse.jetty.util.thread.QueuedThreadPool">
>> <constructor-arg index="0" value="1000" />
>> </bean>
>> </property>
>>
>> <property name="jettyHttpBinding">
>> <bean class="org.apache.camel.component.jetty.DefaultJettyHttpBinding">
>> <property name="transferException" value="false" />
>> </bean>
>> </property>
>> </bean>
>>
>> My camel application is running in docker container with limited amount of
>> memory, and I did not expect, that a file will be read a string instead of
>> reading from InputStream.
>>
>> Is there any way to consume large files with small amount of memory?
>>
>>
>> Regards,
>>
>> Uladzimir
>>
>
Re: OutOfMemoryError: Java heap space when sending large file to
jetty endpoint
Posted by Vova Shelgunov <vv...@gmail.com>.
Hi Andrea,
Yes, I run application using jar file. Java has 1G heap.
Actually I assumed that camel will not read file in single string, but will
use InputStream to read the data.
2016-10-19 15:28 GMT+03:00 Andrea Cosentino <an...@yahoo.com.invalid>:
> Is the camel-context deployed as a simple JAR? What are you using inside
> the Debian container? How much memory has your container?
> Did you set JVM_OPTS witch specific values for max Heap size and min Heap
> size?
>
> --
> Andrea Cosentino
> ----------------------------------
> Apache Camel PMC Member
> Apache Karaf Committer
> Apache Servicemix Committer
> Email: ancosen1985@yahoo.com
> Twitter: @oscerd2
> Github: oscerd
>
>
>
> On Wednesday, October 19, 2016 2:00 PM, Vova Shelgunov <vv...@gmail.com>
> wrote:
> Hi,
>
> I catched this OutOfMemoryError when I sent a 250 MB file to jetty
> endpoint:
>
> java.lang.OutOfMemoryError: Java heap space
> at java.lang.StringCoding.decode(StringCoding.java:215)
> at java.lang.String.<init>(String.java:463)
> at java.lang.String.<init>(String.java:515)
> at
> org.eclipse.jetty.servlets.MultiPartFilter$Wrapper.
> getParameterBytesAsString(MultiPartFilter.java:390)
> at
> org.eclipse.jetty.servlets.MultiPartFilter$Wrapper.getParameterValues(
> MultiPartFilter.java:337)
> at
> org.apache.camel.http.common.DefaultHttpBinding.populateRequestParameters(
> DefaultHttpBinding.java:219)
> at
> org.apache.camel.http.common.DefaultHttpBinding.readHeaders(
> DefaultHttpBinding.java:174)
> at
> org.apache.camel.http.common.DefaultHttpBinding.readRequest(
> DefaultHttpBinding.java:110)
> at
> org.apache.camel.http.common.HttpMessage.<init>(HttpMessage.java:52)
> at
> org.apache.camel.component.jetty.CamelContinuationServlet.doService(
> CamelContinuationServlet.java:161)
> at
> org.apache.camel.http.common.CamelServlet.service(CamelServlet.java:74)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1669)
> at
> org.eclipse.jetty.servlets.MultiPartFilter.doFilter(
> MultiPartFilter.java:200)
> at
> org.apache.camel.component.jetty.CamelFilterWrapper.
> doFilter(CamelFilterWrapper.java:43)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1652)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> at
> org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1127)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at
> org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1061)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:257)
> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
> QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
>
> My configuration:
>
> <route id="ds-rest-archive-upload-processor">
> <from uri="jetty:
> https://storage:8443/data/archive?sslContextParametersRef=
> sslContextParameters&requestBufferSize=32768&responseBufferSize=32768&
> httpMethodRestrict=POST
> <https://storage:8443/data/archive?sslContextParametersRef=
> sslContextParameters&requestBufferSize=32768&responseBufferSize=32768&
> httpMethodRestrict=POST>"
> />
>
> <to uri="bean:uploadFromArchiveFileProcessor?method=process"
> pattern="InOut" />
>
> </route>
>
> <bean id="jetty"
> class="org.apache.camel.component.jetty9.JettyHttpComponent9">
> <property name="endpointClass"
> value="org.apache.camel.component.jetty9.JettyHttpEndpoint9" />
> <property name="sslContextParameters" ref="sslContextParameters" />
> <property name="useContinuation" value="true" />
> <property name="requestBufferSize" value="32768" />
> <property name="responseBufferSize" value="32768" />
>
> <property name="threadPool">
> <bean class="org.eclipse.jetty.util.thread.QueuedThreadPool">
> <constructor-arg index="0" value="1000" />
> </bean>
> </property>
>
> <property name="jettyHttpBinding">
> <bean class="org.apache.camel.component.jetty.DefaultJettyHttpBinding">
> <property name="transferException" value="false" />
> </bean>
> </property>
> </bean>
>
> My camel application is running in docker container with limited amount of
> memory, and I did not expect, that a file will be read a string instead of
> reading from InputStream.
>
> Is there any way to consume large files with small amount of memory?
>
>
> Regards,
>
> Uladzimir
>
Re: OutOfMemoryError: Java heap space when sending large file to
jetty endpoint
Posted by Andrea Cosentino <an...@yahoo.com.INVALID>.
Is the camel-context deployed as a simple JAR? What are you using inside the Debian container? How much memory has your container?
Did you set JVM_OPTS witch specific values for max Heap size and min Heap size?
--
Andrea Cosentino
----------------------------------
Apache Camel PMC Member
Apache Karaf Committer
Apache Servicemix Committer
Email: ancosen1985@yahoo.com
Twitter: @oscerd2
Github: oscerd
On Wednesday, October 19, 2016 2:00 PM, Vova Shelgunov <vv...@gmail.com> wrote:
Hi,
I catched this OutOfMemoryError when I sent a 250 MB file to jetty endpoint:
java.lang.OutOfMemoryError: Java heap space
at java.lang.StringCoding.decode(StringCoding.java:215)
at java.lang.String.<init>(String.java:463)
at java.lang.String.<init>(String.java:515)
at
org.eclipse.jetty.servlets.MultiPartFilter$Wrapper.getParameterBytesAsString(MultiPartFilter.java:390)
at
org.eclipse.jetty.servlets.MultiPartFilter$Wrapper.getParameterValues(MultiPartFilter.java:337)
at
org.apache.camel.http.common.DefaultHttpBinding.populateRequestParameters(DefaultHttpBinding.java:219)
at
org.apache.camel.http.common.DefaultHttpBinding.readHeaders(DefaultHttpBinding.java:174)
at
org.apache.camel.http.common.DefaultHttpBinding.readRequest(DefaultHttpBinding.java:110)
at
org.apache.camel.http.common.HttpMessage.<init>(HttpMessage.java:52)
at
org.apache.camel.component.jetty.CamelContinuationServlet.doService(CamelContinuationServlet.java:161)
at
org.apache.camel.http.common.CamelServlet.service(CamelServlet.java:74)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
at
org.eclipse.jetty.servlets.MultiPartFilter.doFilter(MultiPartFilter.java:200)
at
org.apache.camel.component.jetty.CamelFilterWrapper.doFilter(CamelFilterWrapper.java:43)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
My configuration:
<route id="ds-rest-archive-upload-processor">
<from uri="jetty:
https://storage:8443/data/archive?sslContextParametersRef=sslContextParameters&requestBufferSize=32768&responseBufferSize=32768&httpMethodRestrict=POST
<https://storage:8443/data/archive?sslContextParametersRef=sslContextParameters&requestBufferSize=32768&responseBufferSize=32768&httpMethodRestrict=POST>"
/>
<to uri="bean:uploadFromArchiveFileProcessor?method=process"
pattern="InOut" />
</route>
<bean id="jetty"
class="org.apache.camel.component.jetty9.JettyHttpComponent9">
<property name="endpointClass"
value="org.apache.camel.component.jetty9.JettyHttpEndpoint9" />
<property name="sslContextParameters" ref="sslContextParameters" />
<property name="useContinuation" value="true" />
<property name="requestBufferSize" value="32768" />
<property name="responseBufferSize" value="32768" />
<property name="threadPool">
<bean class="org.eclipse.jetty.util.thread.QueuedThreadPool">
<constructor-arg index="0" value="1000" />
</bean>
</property>
<property name="jettyHttpBinding">
<bean class="org.apache.camel.component.jetty.DefaultJettyHttpBinding">
<property name="transferException" value="false" />
</bean>
</property>
</bean>
My camel application is running in docker container with limited amount of
memory, and I did not expect, that a file will be read a string instead of
reading from InputStream.
Is there any way to consume large files with small amount of memory?
Regards,
Uladzimir