You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@servicemix.apache.org by steff aka sid <sr...@brockhaus-gruppe.de> on 2008/09/03 10:53:30 UTC

Huge amount of data (about 100 MByte)

Hi,
its me again :) I'am trying to handle a huge amount of data with servicemix
but always get an error message (I'am not so familar with tuning smx etc.).
So what I want to do is the following:

1. Get an xml file with ftp (see my other posts x) ). This xml files are
about 100 to 200 of MByte.
2. This files are send as a message to a bean component and this bean
component has to do some transformations by using Smooks. This sending
fails.
3. After the transformation the transformed files which are shrinked at
about 50Mbyte has to be send to another ftp or folder.

My questions:
- What is the best flow to use in such a use case, where can I define the
flow to use?
- What parameters affect this issue?
- How can I receive the message as Stream?

Regards
/Steffen

P.s. Here is the Exception I got:

Exception in thread "pool-flow.seda.servicemix-bean-thread-1"
java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2882)
        at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
        at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572)
        at java.lang.StringBuffer.append(StringBuffer.java:320)
        at java.io.StringWriter.write(StringWriter.java:60)
        at
org.apache.xml.serializer.ToStream.writeAttrString(ToStream.java:1935)
        at
org.apache.xml.serializer.ToStream.processAttributes(ToStream.java:1902)
        at org.apache.xml.serializer.ToStream.endElement(ToStream.java:1989)
        at
org.apache.xalan.transformer.TransformerIdentityImpl.endElement(TransformerIdentityImpl.java:1101)
        at org.apache.xml.serializer.TreeWalker.endNode(TreeWalker.java:458)
        at
org.apache.xml.serializer.TreeWalker.traverse(TreeWalker.java:149)
        at
org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:389)
        at
org.apache.servicemix.jbi.jaxp.SourceTransformer.toResult(SourceTransformer.java:125)
        at
org.apache.servicemix.jbi.jaxp.SourceTransformer.toResult(SourceTransformer.java:103)
        at
org.apache.servicemix.jbi.jaxp.SourceTransformer.toString(SourceTransformer.java:140)
        at
org.apache.servicemix.jbi.jaxp.SourceTransformer.toString(SourceTransformer.java:149)
        at
org.apache.servicemix.jbi.messaging.MessageExchangeImpl.display(MessageExchangeImpl.java:769)
        at
org.apache.servicemix.jbi.messaging.MessageExchangeImpl.toString(MessageExchangeImpl.java:743)
        at java.lang.String.valueOf(String.java:2827)
        at java.lang.StringBuilder.append(StringBuilder.java:115)
        at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:136)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

I am starting my servicemix with an max Heap of 1024Mbyte.


-----
Brockhaus GmbH
COMPETITIVE THROUGH KNOWLEDGE

Web:  http://www.brockhaus-gruppe.de www.brockhaus-gruppe.de  / 
http://www.brockhaus-group.com www.brockhaus-group.com 
-- 
View this message in context: http://www.nabble.com/Huge-amount-of-data-%28about-100-MByte%29-tp19285331p19285331.html
Sent from the ServiceMix - User mailing list archive at Nabble.com.


RE: Unable to read files in my xbean

Posted by Richa Khurana <ri...@globallogic.com>.
Here are the configurations-

File poller-
<beans xmlns:file="http://servicemix.apache.org/file/1.0"
	
xmlns:tutorial="urn:org:apache:servicemix:tutorial:camel" 
                   xmlns:tut="urn:servicemix:tutorial">
              <!-- add the sender endpoint here -->
              <file:sender service="tut:file" 
                           endpoint="sender"
             directory="C:\Temp" >
              </file:sender>
              <!-- add the poller endpoint here -->    
                        <file:poller service="tut:file" 
                                         endpoint="poller"
                                         file="C:\Test" 
                                         targetService="tutorial:jms">
	<property name="filter">
	<bean class="org.apache.commons.io.filefilter.WildcardFilter">
		<constructor-arg value="*.pdf" />
	</bean>
	</property>
								  

                        </file:poller>

            </beans>


JMS:
<?xml version="1.0" encoding="UTF-8"?>

<beans xmlns:jms="http://servicemix.apache.org/jms/1.0"
       xmlns:tutorial="urn:org:apache:servicemix:tutorial:camel"
       xmlns:amq="http://activemq.org/config/1.0">

    <jms:provider service="tutorial:jms"
                  endpoint="provider" 
                  destinationName="tutorial.camel.queue"
                  connectionFactory="#connectionFactory" />

    <jms:consumer service="tutorial:jms"
	          endpoint="consumer"
	          destinationName="tutorial.camel.queue"
	          connectionFactory="#connectionFactory"
	          targetService="tutorial:jms"
	          targetEndpoint="consumer" />
	
    <amq:connectionFactory id="connectionFactory"
brokerURL="tcp://localhost:61616" />

</beans>

Xbean:
<beans xmlns:bean="http://servicemix.apache.org/bean/1.0"
       xmlns:uploadWS="urn:servicemix:tutorial">

  <bean:endpoint service="uploadWS:myBean" endpoint="myBean"
bean="#myBean"/>
  <bean id="myBean" class="com.thirdpillar.uploadWS.abc"/>
  
</beans>

-----Original Message-----
From: Guillaume Nodet [mailto:gnodet@gmail.com] 
Sent: Friday, September 05, 2008 5:22 PM
To: users@servicemix.apache.org
Subject: Re: Unable to read files in my xbean

Could you please provide the xbean configurations and the code of your
custom bean ?

On Fri, Sep 5, 2008 at 1:49 PM, Richa Khurana
<ri...@globallogic.com> wrote:
> Hi,
>
> I am a newbie to servicemix and camel.. am using a filepoller to load
> files from a source folder and send them to a JMS queue and then to my
> xbean. it reaches upto my xbean but i cant read the file, it throws
the
> below exception
>
>
>
> DEBUG - SedaQueue                      -
> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1@1661f7b dequeued
> exchange: InOnly[
> id: ID:172.16.80.75-11c324d629e-8:0
> status: Active
>   role: provider
>   endpoint: myBean
> in: Unable to display: java.io.IOException: Stream closed
>
> Thanks,
> Richa
>



-- 
Cheers,
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/

Re: Unable to read files in my xbean

Posted by Guillaume Nodet <gn...@gmail.com>.
Could you please provide the xbean configurations and the code of your
custom bean ?

On Fri, Sep 5, 2008 at 1:49 PM, Richa Khurana
<ri...@globallogic.com> wrote:
> Hi,
>
> I am a newbie to servicemix and camel.. am using a filepoller to load
> files from a source folder and send them to a JMS queue and then to my
> xbean. it reaches upto my xbean but i cant read the file, it throws the
> below exception
>
>
>
> DEBUG - SedaQueue                      -
> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1@1661f7b dequeued
> exchange: InOnly[
> id: ID:172.16.80.75-11c324d629e-8:0
> status: Active
>   role: provider
>   endpoint: myBean
> in: Unable to display: java.io.IOException: Stream closed
>
> Thanks,
> Richa
>



-- 
Cheers,
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/

Re: Unable to read files in my xbean

Posted by Guillaume Nodet <gn...@gmail.com>.
Yes, this is expected.  You can retrieve the binary stream in the
attachments on the input normalized message.

On Fri, Sep 5, 2008 at 2:15 PM, Richa Khurana
<ri...@globallogic.com> wrote:
> I tried that also. But, then the source object on xbean side was null.
>
> -----Original Message-----
> From: Guillaume Nodet [mailto:gnodet@gmail.com]
> Sent: Friday, September 05, 2008 5:43 PM
> To: users@servicemix.apache.org
> Subject: Re: Unable to read files in my xbean
>
> I think the reason is that servicemix expects an xml as the content.
> You need to set up the file endpoint to use the BinaryFileMarshaler.
>
> On Fri, Sep 5, 2008 at 1:49 PM, Richa Khurana
> <ri...@globallogic.com> wrote:
>> Hi,
>>
>> I am a newbie to servicemix and camel.. am using a filepoller to load
>> files from a source folder and send them to a JMS queue and then to my
>> xbean. it reaches upto my xbean but i cant read the file, it throws
> the
>> below exception
>>
>>
>>
>> DEBUG - SedaQueue                      -
>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1@1661f7b dequeued
>> exchange: InOnly[
>> id: ID:172.16.80.75-11c324d629e-8:0
>> status: Active
>>   role: provider
>>   endpoint: myBean
>> in: Unable to display: java.io.IOException: Stream closed
>>
>> Thanks,
>> Richa
>>
>
>
>
> --
> Cheers,
> Guillaume Nodet
> ------------------------
> Blog: http://gnodet.blogspot.com/
>



-- 
Cheers,
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/

RE: Unable to read files in my xbean

Posted by Richa Khurana <ri...@globallogic.com>.
I tried that also. But, then the source object on xbean side was null.

-----Original Message-----
From: Guillaume Nodet [mailto:gnodet@gmail.com] 
Sent: Friday, September 05, 2008 5:43 PM
To: users@servicemix.apache.org
Subject: Re: Unable to read files in my xbean

I think the reason is that servicemix expects an xml as the content.
You need to set up the file endpoint to use the BinaryFileMarshaler.

On Fri, Sep 5, 2008 at 1:49 PM, Richa Khurana
<ri...@globallogic.com> wrote:
> Hi,
>
> I am a newbie to servicemix and camel.. am using a filepoller to load
> files from a source folder and send them to a JMS queue and then to my
> xbean. it reaches upto my xbean but i cant read the file, it throws
the
> below exception
>
>
>
> DEBUG - SedaQueue                      -
> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1@1661f7b dequeued
> exchange: InOnly[
> id: ID:172.16.80.75-11c324d629e-8:0
> status: Active
>   role: provider
>   endpoint: myBean
> in: Unable to display: java.io.IOException: Stream closed
>
> Thanks,
> Richa
>



-- 
Cheers,
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/

Re: Unable to read files in my xbean

Posted by Guillaume Nodet <gn...@gmail.com>.
I think the reason is that servicemix expects an xml as the content.
You need to set up the file endpoint to use the BinaryFileMarshaler.

On Fri, Sep 5, 2008 at 1:49 PM, Richa Khurana
<ri...@globallogic.com> wrote:
> Hi,
>
> I am a newbie to servicemix and camel.. am using a filepoller to load
> files from a source folder and send them to a JMS queue and then to my
> xbean. it reaches upto my xbean but i cant read the file, it throws the
> below exception
>
>
>
> DEBUG - SedaQueue                      -
> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1@1661f7b dequeued
> exchange: InOnly[
> id: ID:172.16.80.75-11c324d629e-8:0
> status: Active
>   role: provider
>   endpoint: myBean
> in: Unable to display: java.io.IOException: Stream closed
>
> Thanks,
> Richa
>



-- 
Cheers,
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/

Re: Unable to read files in my xbean

Posted by Lars Heinemann <la...@compart.net>.
Richa,

which FileMarshaler do you use for polling?
That the stream is closed means that it was read before until end of stream.
It can also be caused by having debug log level activated.

Regards
Lars



Am Freitag 05 September 2008 13:49:11 schrieb Richa Khurana:
> Hi,
>
> I am a newbie to servicemix and camel.. am using a filepoller to load
> files from a source folder and send them to a JMS queue and then to my
> xbean. it reaches upto my xbean but i cant read the file, it throws the
> below exception
>
>
>
> DEBUG - SedaQueue                      -
> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1@1661f7b dequeued
> exchange: InOnly[
> id: ID:172.16.80.75-11c324d629e-8:0
> status: Active
>    role: provider
>    endpoint: myBean
> in: Unable to display: java.io.IOException: Stream closed
>
> Thanks,
> Richa


Unable to read files in my xbean

Posted by Richa Khurana <ri...@globallogic.com>.
Hi,

I am a newbie to servicemix and camel.. am using a filepoller to load
files from a source folder and send them to a JMS queue and then to my
xbean. it reaches upto my xbean but i cant read the file, it throws the
below exception



DEBUG - SedaQueue                      -
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1@1661f7b dequeued
exchange: InOnly[
id: ID:172.16.80.75-11c324d629e-8:0
status: Active
   role: provider
   endpoint: myBean
in: Unable to display: java.io.IOException: Stream closed

Thanks,
Richa

Re: Huge amount of data (about 100 MByte)

Posted by Guillaume Nodet <gn...@gmail.com>.
Yes, you should avoid JMS and JCA flows as they will serialize the JBI
exchanges, thus loading the whole streams in memory.

On Tue, Sep 23, 2008 at 12:38 PM, Jussi Nummelin
<ju...@gmail.com> wrote:
> Thanks.
>
> But how about configurations for different flows, is there something
> that we should configure to enable streaming for messages within the
> NMR?
>
> - Jussi -
>
> On Tue, Sep 23, 2008 at 12:59 PM, Guillaume Nodet <gn...@gmail.com> wrote:
>> I think the CXF binding component does support attachments caching to
>> disk to avoid loading them entirely in memory.
>>
>> On Tue, Sep 23, 2008 at 10:37 AM, Jussi Nummelin
>> <ju...@gmail.com> wrote:
>>> We're mainly using SOAP as our incoming protocol. In typical cases we
>>> get the actual payload data as attachments which we need to rip of
>>> from the message and send them to further processing in the bus. These
>>> attachments can be hundreds of megs in size.
>>>
>>> We're thinking of some kind of a streaming solution 'cause typically
>>> we do not need to transform the actual payload data. The
>>> transformation and "real" processing happens only for the incoming
>>> message and not for the attachments.
>>>
>>> BR,
>>> - Jussi -
>>>
>>> On Mon, Sep 22, 2008 at 5:24 PM, Guillaume Nodet <gn...@gmail.com> wrote:
>>>> It mostly depends on the flow and components used.
>>>> Could you give more informations ?
>>>>
>>>> On Mon, Sep 22, 2008 at 3:06 PM, Jussi Nummelin
>>>> <ju...@gmail.com> wrote:
>>>>> Hi,
>>>>>
>>>>> We're facing the exact same issue. And no, turning DEBUG logging off
>>>>> does not solve it. :-)
>>>>>
>>>>> How to configure the flow(s) to handle huge input messages as streams?
>>>>> Or is this something that should be configured in the binding
>>>>> components in use?
>>>>>
>>>>> BR,
>>>>> - Jussi -
>>>>>
>>>>> On Wed, Sep 3, 2008 at 1:31 PM, Guillaume Nodet <gn...@gmail.com> wrote:
>>>>>> You need to turn off DEBUG logging, else the exchanges are displayed,
>>>>>> thus the whole xml document is loaded in memory and parsed.
>>>>>> Alternatively, you can set the system property named
>>>>>> "org.apache.servicemix.preserveContent" to "true" if you want to keep
>>>>>> DEBUG logging.
>>>>>>
>>>>>> On Wed, Sep 3, 2008 at 10:53 AM, steff aka sid
>>>>>> <sr...@brockhaus-gruppe.de> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>> its me again :) I'am trying to handle a huge amount of data with servicemix
>>>>>>> but always get an error message (I'am not so familar with tuning smx etc.).
>>>>>>> So what I want to do is the following:
>>>>>>>
>>>>>>> 1. Get an xml file with ftp (see my other posts x) ). This xml files are
>>>>>>> about 100 to 200 of MByte.
>>>>>>> 2. This files are send as a message to a bean component and this bean
>>>>>>> component has to do some transformations by using Smooks. This sending
>>>>>>> fails.
>>>>>>> 3. After the transformation the transformed files which are shrinked at
>>>>>>> about 50Mbyte has to be send to another ftp or folder.
>>>>>>>
>>>>>>> My questions:
>>>>>>> - What is the best flow to use in such a use case, where can I define the
>>>>>>> flow to use?
>>>>>>> - What parameters affect this issue?
>>>>>>> - How can I receive the message as Stream?
>>>>>>>
>>>>>>> Regards
>>>>>>> /Steffen
>>>>>>>
>>>>>>> P.s. Here is the Exception I got:
>>>>>>>
>>>>>>> Exception in thread "pool-flow.seda.servicemix-bean-thread-1"
>>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>>        at java.util.Arrays.copyOf(Arrays.java:2882)
>>>>>>>        at
>>>>>>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
>>>>>>>        at
>>>>>>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572)
>>>>>>>        at java.lang.StringBuffer.append(StringBuffer.java:320)
>>>>>>>        at java.io.StringWriter.write(StringWriter.java:60)
>>>>>>>        at
>>>>>>> org.apache.xml.serializer.ToStream.writeAttrString(ToStream.java:1935)
>>>>>>>        at
>>>>>>> org.apache.xml.serializer.ToStream.processAttributes(ToStream.java:1902)
>>>>>>>        at org.apache.xml.serializer.ToStream.endElement(ToStream.java:1989)
>>>>>>>        at
>>>>>>> org.apache.xalan.transformer.TransformerIdentityImpl.endElement(TransformerIdentityImpl.java:1101)
>>>>>>>        at org.apache.xml.serializer.TreeWalker.endNode(TreeWalker.java:458)
>>>>>>>        at
>>>>>>> org.apache.xml.serializer.TreeWalker.traverse(TreeWalker.java:149)
>>>>>>>        at
>>>>>>> org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:389)
>>>>>>>        at
>>>>>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toResult(SourceTransformer.java:125)
>>>>>>>        at
>>>>>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toResult(SourceTransformer.java:103)
>>>>>>>        at
>>>>>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toString(SourceTransformer.java:140)
>>>>>>>        at
>>>>>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toString(SourceTransformer.java:149)
>>>>>>>        at
>>>>>>> org.apache.servicemix.jbi.messaging.MessageExchangeImpl.display(MessageExchangeImpl.java:769)
>>>>>>>        at
>>>>>>> org.apache.servicemix.jbi.messaging.MessageExchangeImpl.toString(MessageExchangeImpl.java:743)
>>>>>>>        at java.lang.String.valueOf(String.java:2827)
>>>>>>>        at java.lang.StringBuilder.append(StringBuilder.java:115)
>>>>>>>        at
>>>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:136)
>>>>>>>        at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>>>>>>>        at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>        at java.lang.Thread.run(Thread.java:619)
>>>>>>>
>>>>>>> I am starting my servicemix with an max Heap of 1024Mbyte.
>>>>>>>
>>>>>>>
>>>>>>> -----
>>>>>>> Brockhaus GmbH
>>>>>>> COMPETITIVE THROUGH KNOWLEDGE
>>>>>>>
>>>>>>> Web:  http://www.brockhaus-gruppe.de www.brockhaus-gruppe.de  /
>>>>>>> http://www.brockhaus-group.com www.brockhaus-group.com
>>>>>>> --
>>>>>>> View this message in context: http://www.nabble.com/Huge-amount-of-data-%28about-100-MByte%29-tp19285331p19285331.html
>>>>>>> Sent from the ServiceMix - User mailing list archive at Nabble.com.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Cheers,
>>>>>> Guillaume Nodet
>>>>>> ------------------------
>>>>>> Blog: http://gnodet.blogspot.com/
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Cheers,
>>>> Guillaume Nodet
>>>> ------------------------
>>>> Blog: http://gnodet.blogspot.com/
>>>> ------------------------
>>>> Open Source SOA
>>>> http://open.iona.com
>>>>
>>>
>>
>>
>>
>> --
>> Cheers,
>> Guillaume Nodet
>> ------------------------
>> Blog: http://gnodet.blogspot.com/
>> ------------------------
>> Open Source SOA
>> http://open.iona.com
>>
>



-- 
Cheers,
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
------------------------
Open Source SOA
http://open.iona.com

Re: Huge amount of data (about 100 MByte)

Posted by Jussi Nummelin <ju...@gmail.com>.
Thanks.

But how about configurations for different flows, is there something
that we should configure to enable streaming for messages within the
NMR?

- Jussi -

On Tue, Sep 23, 2008 at 12:59 PM, Guillaume Nodet <gn...@gmail.com> wrote:
> I think the CXF binding component does support attachments caching to
> disk to avoid loading them entirely in memory.
>
> On Tue, Sep 23, 2008 at 10:37 AM, Jussi Nummelin
> <ju...@gmail.com> wrote:
>> We're mainly using SOAP as our incoming protocol. In typical cases we
>> get the actual payload data as attachments which we need to rip of
>> from the message and send them to further processing in the bus. These
>> attachments can be hundreds of megs in size.
>>
>> We're thinking of some kind of a streaming solution 'cause typically
>> we do not need to transform the actual payload data. The
>> transformation and "real" processing happens only for the incoming
>> message and not for the attachments.
>>
>> BR,
>> - Jussi -
>>
>> On Mon, Sep 22, 2008 at 5:24 PM, Guillaume Nodet <gn...@gmail.com> wrote:
>>> It mostly depends on the flow and components used.
>>> Could you give more informations ?
>>>
>>> On Mon, Sep 22, 2008 at 3:06 PM, Jussi Nummelin
>>> <ju...@gmail.com> wrote:
>>>> Hi,
>>>>
>>>> We're facing the exact same issue. And no, turning DEBUG logging off
>>>> does not solve it. :-)
>>>>
>>>> How to configure the flow(s) to handle huge input messages as streams?
>>>> Or is this something that should be configured in the binding
>>>> components in use?
>>>>
>>>> BR,
>>>> - Jussi -
>>>>
>>>> On Wed, Sep 3, 2008 at 1:31 PM, Guillaume Nodet <gn...@gmail.com> wrote:
>>>>> You need to turn off DEBUG logging, else the exchanges are displayed,
>>>>> thus the whole xml document is loaded in memory and parsed.
>>>>> Alternatively, you can set the system property named
>>>>> "org.apache.servicemix.preserveContent" to "true" if you want to keep
>>>>> DEBUG logging.
>>>>>
>>>>> On Wed, Sep 3, 2008 at 10:53 AM, steff aka sid
>>>>> <sr...@brockhaus-gruppe.de> wrote:
>>>>>>
>>>>>> Hi,
>>>>>> its me again :) I'am trying to handle a huge amount of data with servicemix
>>>>>> but always get an error message (I'am not so familar with tuning smx etc.).
>>>>>> So what I want to do is the following:
>>>>>>
>>>>>> 1. Get an xml file with ftp (see my other posts x) ). This xml files are
>>>>>> about 100 to 200 of MByte.
>>>>>> 2. This files are send as a message to a bean component and this bean
>>>>>> component has to do some transformations by using Smooks. This sending
>>>>>> fails.
>>>>>> 3. After the transformation the transformed files which are shrinked at
>>>>>> about 50Mbyte has to be send to another ftp or folder.
>>>>>>
>>>>>> My questions:
>>>>>> - What is the best flow to use in such a use case, where can I define the
>>>>>> flow to use?
>>>>>> - What parameters affect this issue?
>>>>>> - How can I receive the message as Stream?
>>>>>>
>>>>>> Regards
>>>>>> /Steffen
>>>>>>
>>>>>> P.s. Here is the Exception I got:
>>>>>>
>>>>>> Exception in thread "pool-flow.seda.servicemix-bean-thread-1"
>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>        at java.util.Arrays.copyOf(Arrays.java:2882)
>>>>>>        at
>>>>>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
>>>>>>        at
>>>>>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572)
>>>>>>        at java.lang.StringBuffer.append(StringBuffer.java:320)
>>>>>>        at java.io.StringWriter.write(StringWriter.java:60)
>>>>>>        at
>>>>>> org.apache.xml.serializer.ToStream.writeAttrString(ToStream.java:1935)
>>>>>>        at
>>>>>> org.apache.xml.serializer.ToStream.processAttributes(ToStream.java:1902)
>>>>>>        at org.apache.xml.serializer.ToStream.endElement(ToStream.java:1989)
>>>>>>        at
>>>>>> org.apache.xalan.transformer.TransformerIdentityImpl.endElement(TransformerIdentityImpl.java:1101)
>>>>>>        at org.apache.xml.serializer.TreeWalker.endNode(TreeWalker.java:458)
>>>>>>        at
>>>>>> org.apache.xml.serializer.TreeWalker.traverse(TreeWalker.java:149)
>>>>>>        at
>>>>>> org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:389)
>>>>>>        at
>>>>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toResult(SourceTransformer.java:125)
>>>>>>        at
>>>>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toResult(SourceTransformer.java:103)
>>>>>>        at
>>>>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toString(SourceTransformer.java:140)
>>>>>>        at
>>>>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toString(SourceTransformer.java:149)
>>>>>>        at
>>>>>> org.apache.servicemix.jbi.messaging.MessageExchangeImpl.display(MessageExchangeImpl.java:769)
>>>>>>        at
>>>>>> org.apache.servicemix.jbi.messaging.MessageExchangeImpl.toString(MessageExchangeImpl.java:743)
>>>>>>        at java.lang.String.valueOf(String.java:2827)
>>>>>>        at java.lang.StringBuilder.append(StringBuilder.java:115)
>>>>>>        at
>>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:136)
>>>>>>        at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>>>>>>        at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>        at java.lang.Thread.run(Thread.java:619)
>>>>>>
>>>>>> I am starting my servicemix with an max Heap of 1024Mbyte.
>>>>>>
>>>>>>
>>>>>> -----
>>>>>> Brockhaus GmbH
>>>>>> COMPETITIVE THROUGH KNOWLEDGE
>>>>>>
>>>>>> Web:  http://www.brockhaus-gruppe.de www.brockhaus-gruppe.de  /
>>>>>> http://www.brockhaus-group.com www.brockhaus-group.com
>>>>>> --
>>>>>> View this message in context: http://www.nabble.com/Huge-amount-of-data-%28about-100-MByte%29-tp19285331p19285331.html
>>>>>> Sent from the ServiceMix - User mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Cheers,
>>>>> Guillaume Nodet
>>>>> ------------------------
>>>>> Blog: http://gnodet.blogspot.com/
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Cheers,
>>> Guillaume Nodet
>>> ------------------------
>>> Blog: http://gnodet.blogspot.com/
>>> ------------------------
>>> Open Source SOA
>>> http://open.iona.com
>>>
>>
>
>
>
> --
> Cheers,
> Guillaume Nodet
> ------------------------
> Blog: http://gnodet.blogspot.com/
> ------------------------
> Open Source SOA
> http://open.iona.com
>

Re: Huge amount of data (about 100 MByte)

Posted by Guillaume Nodet <gn...@gmail.com>.
I think the CXF binding component does support attachments caching to
disk to avoid loading them entirely in memory.

On Tue, Sep 23, 2008 at 10:37 AM, Jussi Nummelin
<ju...@gmail.com> wrote:
> We're mainly using SOAP as our incoming protocol. In typical cases we
> get the actual payload data as attachments which we need to rip of
> from the message and send them to further processing in the bus. These
> attachments can be hundreds of megs in size.
>
> We're thinking of some kind of a streaming solution 'cause typically
> we do not need to transform the actual payload data. The
> transformation and "real" processing happens only for the incoming
> message and not for the attachments.
>
> BR,
> - Jussi -
>
> On Mon, Sep 22, 2008 at 5:24 PM, Guillaume Nodet <gn...@gmail.com> wrote:
>> It mostly depends on the flow and components used.
>> Could you give more informations ?
>>
>> On Mon, Sep 22, 2008 at 3:06 PM, Jussi Nummelin
>> <ju...@gmail.com> wrote:
>>> Hi,
>>>
>>> We're facing the exact same issue. And no, turning DEBUG logging off
>>> does not solve it. :-)
>>>
>>> How to configure the flow(s) to handle huge input messages as streams?
>>> Or is this something that should be configured in the binding
>>> components in use?
>>>
>>> BR,
>>> - Jussi -
>>>
>>> On Wed, Sep 3, 2008 at 1:31 PM, Guillaume Nodet <gn...@gmail.com> wrote:
>>>> You need to turn off DEBUG logging, else the exchanges are displayed,
>>>> thus the whole xml document is loaded in memory and parsed.
>>>> Alternatively, you can set the system property named
>>>> "org.apache.servicemix.preserveContent" to "true" if you want to keep
>>>> DEBUG logging.
>>>>
>>>> On Wed, Sep 3, 2008 at 10:53 AM, steff aka sid
>>>> <sr...@brockhaus-gruppe.de> wrote:
>>>>>
>>>>> Hi,
>>>>> its me again :) I'am trying to handle a huge amount of data with servicemix
>>>>> but always get an error message (I'am not so familar with tuning smx etc.).
>>>>> So what I want to do is the following:
>>>>>
>>>>> 1. Get an xml file with ftp (see my other posts x) ). This xml files are
>>>>> about 100 to 200 of MByte.
>>>>> 2. This files are send as a message to a bean component and this bean
>>>>> component has to do some transformations by using Smooks. This sending
>>>>> fails.
>>>>> 3. After the transformation the transformed files which are shrinked at
>>>>> about 50Mbyte has to be send to another ftp or folder.
>>>>>
>>>>> My questions:
>>>>> - What is the best flow to use in such a use case, where can I define the
>>>>> flow to use?
>>>>> - What parameters affect this issue?
>>>>> - How can I receive the message as Stream?
>>>>>
>>>>> Regards
>>>>> /Steffen
>>>>>
>>>>> P.s. Here is the Exception I got:
>>>>>
>>>>> Exception in thread "pool-flow.seda.servicemix-bean-thread-1"
>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>        at java.util.Arrays.copyOf(Arrays.java:2882)
>>>>>        at
>>>>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
>>>>>        at
>>>>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572)
>>>>>        at java.lang.StringBuffer.append(StringBuffer.java:320)
>>>>>        at java.io.StringWriter.write(StringWriter.java:60)
>>>>>        at
>>>>> org.apache.xml.serializer.ToStream.writeAttrString(ToStream.java:1935)
>>>>>        at
>>>>> org.apache.xml.serializer.ToStream.processAttributes(ToStream.java:1902)
>>>>>        at org.apache.xml.serializer.ToStream.endElement(ToStream.java:1989)
>>>>>        at
>>>>> org.apache.xalan.transformer.TransformerIdentityImpl.endElement(TransformerIdentityImpl.java:1101)
>>>>>        at org.apache.xml.serializer.TreeWalker.endNode(TreeWalker.java:458)
>>>>>        at
>>>>> org.apache.xml.serializer.TreeWalker.traverse(TreeWalker.java:149)
>>>>>        at
>>>>> org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:389)
>>>>>        at
>>>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toResult(SourceTransformer.java:125)
>>>>>        at
>>>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toResult(SourceTransformer.java:103)
>>>>>        at
>>>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toString(SourceTransformer.java:140)
>>>>>        at
>>>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toString(SourceTransformer.java:149)
>>>>>        at
>>>>> org.apache.servicemix.jbi.messaging.MessageExchangeImpl.display(MessageExchangeImpl.java:769)
>>>>>        at
>>>>> org.apache.servicemix.jbi.messaging.MessageExchangeImpl.toString(MessageExchangeImpl.java:743)
>>>>>        at java.lang.String.valueOf(String.java:2827)
>>>>>        at java.lang.StringBuilder.append(StringBuilder.java:115)
>>>>>        at
>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:136)
>>>>>        at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>>>>>        at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>        at java.lang.Thread.run(Thread.java:619)
>>>>>
>>>>> I am starting my servicemix with an max Heap of 1024Mbyte.
>>>>>
>>>>>
>>>>> -----
>>>>> Brockhaus GmbH
>>>>> COMPETITIVE THROUGH KNOWLEDGE
>>>>>
>>>>> Web:  http://www.brockhaus-gruppe.de www.brockhaus-gruppe.de  /
>>>>> http://www.brockhaus-group.com www.brockhaus-group.com
>>>>> --
>>>>> View this message in context: http://www.nabble.com/Huge-amount-of-data-%28about-100-MByte%29-tp19285331p19285331.html
>>>>> Sent from the ServiceMix - User mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Cheers,
>>>> Guillaume Nodet
>>>> ------------------------
>>>> Blog: http://gnodet.blogspot.com/
>>>>
>>>
>>
>>
>>
>> --
>> Cheers,
>> Guillaume Nodet
>> ------------------------
>> Blog: http://gnodet.blogspot.com/
>> ------------------------
>> Open Source SOA
>> http://open.iona.com
>>
>



-- 
Cheers,
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
------------------------
Open Source SOA
http://open.iona.com

Re: Huge amount of data (about 100 MByte)

Posted by Jussi Nummelin <ju...@gmail.com>.
We're mainly using SOAP as our incoming protocol. In typical cases we
get the actual payload data as attachments which we need to rip of
from the message and send them to further processing in the bus. These
attachments can be hundreds of megs in size.

We're thinking of some kind of a streaming solution 'cause typically
we do not need to transform the actual payload data. The
transformation and "real" processing happens only for the incoming
message and not for the attachments.

BR,
- Jussi -

On Mon, Sep 22, 2008 at 5:24 PM, Guillaume Nodet <gn...@gmail.com> wrote:
> It mostly depends on the flow and components used.
> Could you give more informations ?
>
> On Mon, Sep 22, 2008 at 3:06 PM, Jussi Nummelin
> <ju...@gmail.com> wrote:
>> Hi,
>>
>> We're facing the exact same issue. And no, turning DEBUG logging off
>> does not solve it. :-)
>>
>> How to configure the flow(s) to handle huge input messages as streams?
>> Or is this something that should be configured in the binding
>> components in use?
>>
>> BR,
>> - Jussi -
>>
>> On Wed, Sep 3, 2008 at 1:31 PM, Guillaume Nodet <gn...@gmail.com> wrote:
>>> You need to turn off DEBUG logging, else the exchanges are displayed,
>>> thus the whole xml document is loaded in memory and parsed.
>>> Alternatively, you can set the system property named
>>> "org.apache.servicemix.preserveContent" to "true" if you want to keep
>>> DEBUG logging.
>>>
>>> On Wed, Sep 3, 2008 at 10:53 AM, steff aka sid
>>> <sr...@brockhaus-gruppe.de> wrote:
>>>>
>>>> Hi,
>>>> its me again :) I'am trying to handle a huge amount of data with servicemix
>>>> but always get an error message (I'am not so familar with tuning smx etc.).
>>>> So what I want to do is the following:
>>>>
>>>> 1. Get an xml file with ftp (see my other posts x) ). This xml files are
>>>> about 100 to 200 of MByte.
>>>> 2. This files are send as a message to a bean component and this bean
>>>> component has to do some transformations by using Smooks. This sending
>>>> fails.
>>>> 3. After the transformation the transformed files which are shrinked at
>>>> about 50Mbyte has to be send to another ftp or folder.
>>>>
>>>> My questions:
>>>> - What is the best flow to use in such a use case, where can I define the
>>>> flow to use?
>>>> - What parameters affect this issue?
>>>> - How can I receive the message as Stream?
>>>>
>>>> Regards
>>>> /Steffen
>>>>
>>>> P.s. Here is the Exception I got:
>>>>
>>>> Exception in thread "pool-flow.seda.servicemix-bean-thread-1"
>>>> java.lang.OutOfMemoryError: Java heap space
>>>>        at java.util.Arrays.copyOf(Arrays.java:2882)
>>>>        at
>>>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
>>>>        at
>>>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572)
>>>>        at java.lang.StringBuffer.append(StringBuffer.java:320)
>>>>        at java.io.StringWriter.write(StringWriter.java:60)
>>>>        at
>>>> org.apache.xml.serializer.ToStream.writeAttrString(ToStream.java:1935)
>>>>        at
>>>> org.apache.xml.serializer.ToStream.processAttributes(ToStream.java:1902)
>>>>        at org.apache.xml.serializer.ToStream.endElement(ToStream.java:1989)
>>>>        at
>>>> org.apache.xalan.transformer.TransformerIdentityImpl.endElement(TransformerIdentityImpl.java:1101)
>>>>        at org.apache.xml.serializer.TreeWalker.endNode(TreeWalker.java:458)
>>>>        at
>>>> org.apache.xml.serializer.TreeWalker.traverse(TreeWalker.java:149)
>>>>        at
>>>> org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:389)
>>>>        at
>>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toResult(SourceTransformer.java:125)
>>>>        at
>>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toResult(SourceTransformer.java:103)
>>>>        at
>>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toString(SourceTransformer.java:140)
>>>>        at
>>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toString(SourceTransformer.java:149)
>>>>        at
>>>> org.apache.servicemix.jbi.messaging.MessageExchangeImpl.display(MessageExchangeImpl.java:769)
>>>>        at
>>>> org.apache.servicemix.jbi.messaging.MessageExchangeImpl.toString(MessageExchangeImpl.java:743)
>>>>        at java.lang.String.valueOf(String.java:2827)
>>>>        at java.lang.StringBuilder.append(StringBuilder.java:115)
>>>>        at
>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:136)
>>>>        at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>>>>        at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>        at java.lang.Thread.run(Thread.java:619)
>>>>
>>>> I am starting my servicemix with an max Heap of 1024Mbyte.
>>>>
>>>>
>>>> -----
>>>> Brockhaus GmbH
>>>> COMPETITIVE THROUGH KNOWLEDGE
>>>>
>>>> Web:  http://www.brockhaus-gruppe.de www.brockhaus-gruppe.de  /
>>>> http://www.brockhaus-group.com www.brockhaus-group.com
>>>> --
>>>> View this message in context: http://www.nabble.com/Huge-amount-of-data-%28about-100-MByte%29-tp19285331p19285331.html
>>>> Sent from the ServiceMix - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Cheers,
>>> Guillaume Nodet
>>> ------------------------
>>> Blog: http://gnodet.blogspot.com/
>>>
>>
>
>
>
> --
> Cheers,
> Guillaume Nodet
> ------------------------
> Blog: http://gnodet.blogspot.com/
> ------------------------
> Open Source SOA
> http://open.iona.com
>

Re: Huge amount of data (about 100 MByte)

Posted by Guillaume Nodet <gn...@gmail.com>.
It mostly depends on the flow and components used.
Could you give more informations ?

On Mon, Sep 22, 2008 at 3:06 PM, Jussi Nummelin
<ju...@gmail.com> wrote:
> Hi,
>
> We're facing the exact same issue. And no, turning DEBUG logging off
> does not solve it. :-)
>
> How to configure the flow(s) to handle huge input messages as streams?
> Or is this something that should be configured in the binding
> components in use?
>
> BR,
> - Jussi -
>
> On Wed, Sep 3, 2008 at 1:31 PM, Guillaume Nodet <gn...@gmail.com> wrote:
>> You need to turn off DEBUG logging, else the exchanges are displayed,
>> thus the whole xml document is loaded in memory and parsed.
>> Alternatively, you can set the system property named
>> "org.apache.servicemix.preserveContent" to "true" if you want to keep
>> DEBUG logging.
>>
>> On Wed, Sep 3, 2008 at 10:53 AM, steff aka sid
>> <sr...@brockhaus-gruppe.de> wrote:
>>>
>>> Hi,
>>> its me again :) I'am trying to handle a huge amount of data with servicemix
>>> but always get an error message (I'am not so familar with tuning smx etc.).
>>> So what I want to do is the following:
>>>
>>> 1. Get an xml file with ftp (see my other posts x) ). This xml files are
>>> about 100 to 200 of MByte.
>>> 2. This files are send as a message to a bean component and this bean
>>> component has to do some transformations by using Smooks. This sending
>>> fails.
>>> 3. After the transformation the transformed files which are shrinked at
>>> about 50Mbyte has to be send to another ftp or folder.
>>>
>>> My questions:
>>> - What is the best flow to use in such a use case, where can I define the
>>> flow to use?
>>> - What parameters affect this issue?
>>> - How can I receive the message as Stream?
>>>
>>> Regards
>>> /Steffen
>>>
>>> P.s. Here is the Exception I got:
>>>
>>> Exception in thread "pool-flow.seda.servicemix-bean-thread-1"
>>> java.lang.OutOfMemoryError: Java heap space
>>>        at java.util.Arrays.copyOf(Arrays.java:2882)
>>>        at
>>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
>>>        at
>>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572)
>>>        at java.lang.StringBuffer.append(StringBuffer.java:320)
>>>        at java.io.StringWriter.write(StringWriter.java:60)
>>>        at
>>> org.apache.xml.serializer.ToStream.writeAttrString(ToStream.java:1935)
>>>        at
>>> org.apache.xml.serializer.ToStream.processAttributes(ToStream.java:1902)
>>>        at org.apache.xml.serializer.ToStream.endElement(ToStream.java:1989)
>>>        at
>>> org.apache.xalan.transformer.TransformerIdentityImpl.endElement(TransformerIdentityImpl.java:1101)
>>>        at org.apache.xml.serializer.TreeWalker.endNode(TreeWalker.java:458)
>>>        at
>>> org.apache.xml.serializer.TreeWalker.traverse(TreeWalker.java:149)
>>>        at
>>> org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:389)
>>>        at
>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toResult(SourceTransformer.java:125)
>>>        at
>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toResult(SourceTransformer.java:103)
>>>        at
>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toString(SourceTransformer.java:140)
>>>        at
>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toString(SourceTransformer.java:149)
>>>        at
>>> org.apache.servicemix.jbi.messaging.MessageExchangeImpl.display(MessageExchangeImpl.java:769)
>>>        at
>>> org.apache.servicemix.jbi.messaging.MessageExchangeImpl.toString(MessageExchangeImpl.java:743)
>>>        at java.lang.String.valueOf(String.java:2827)
>>>        at java.lang.StringBuilder.append(StringBuilder.java:115)
>>>        at
>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:136)
>>>        at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>>>        at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>        at java.lang.Thread.run(Thread.java:619)
>>>
>>> I am starting my servicemix with an max Heap of 1024Mbyte.
>>>
>>>
>>> -----
>>> Brockhaus GmbH
>>> COMPETITIVE THROUGH KNOWLEDGE
>>>
>>> Web:  http://www.brockhaus-gruppe.de www.brockhaus-gruppe.de  /
>>> http://www.brockhaus-group.com www.brockhaus-group.com
>>> --
>>> View this message in context: http://www.nabble.com/Huge-amount-of-data-%28about-100-MByte%29-tp19285331p19285331.html
>>> Sent from the ServiceMix - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Cheers,
>> Guillaume Nodet
>> ------------------------
>> Blog: http://gnodet.blogspot.com/
>>
>



-- 
Cheers,
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
------------------------
Open Source SOA
http://open.iona.com

Re: Huge amount of data (about 100 MByte)

Posted by Jussi Nummelin <ju...@gmail.com>.
Hi,

We're facing the exact same issue. And no, turning DEBUG logging off
does not solve it. :-)

How to configure the flow(s) to handle huge input messages as streams?
Or is this something that should be configured in the binding
components in use?

BR,
- Jussi -

On Wed, Sep 3, 2008 at 1:31 PM, Guillaume Nodet <gn...@gmail.com> wrote:
> You need to turn off DEBUG logging, else the exchanges are displayed,
> thus the whole xml document is loaded in memory and parsed.
> Alternatively, you can set the system property named
> "org.apache.servicemix.preserveContent" to "true" if you want to keep
> DEBUG logging.
>
> On Wed, Sep 3, 2008 at 10:53 AM, steff aka sid
> <sr...@brockhaus-gruppe.de> wrote:
>>
>> Hi,
>> its me again :) I'am trying to handle a huge amount of data with servicemix
>> but always get an error message (I'am not so familar with tuning smx etc.).
>> So what I want to do is the following:
>>
>> 1. Get an xml file with ftp (see my other posts x) ). This xml files are
>> about 100 to 200 of MByte.
>> 2. This files are send as a message to a bean component and this bean
>> component has to do some transformations by using Smooks. This sending
>> fails.
>> 3. After the transformation the transformed files which are shrinked at
>> about 50Mbyte has to be send to another ftp or folder.
>>
>> My questions:
>> - What is the best flow to use in such a use case, where can I define the
>> flow to use?
>> - What parameters affect this issue?
>> - How can I receive the message as Stream?
>>
>> Regards
>> /Steffen
>>
>> P.s. Here is the Exception I got:
>>
>> Exception in thread "pool-flow.seda.servicemix-bean-thread-1"
>> java.lang.OutOfMemoryError: Java heap space
>>        at java.util.Arrays.copyOf(Arrays.java:2882)
>>        at
>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
>>        at
>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572)
>>        at java.lang.StringBuffer.append(StringBuffer.java:320)
>>        at java.io.StringWriter.write(StringWriter.java:60)
>>        at
>> org.apache.xml.serializer.ToStream.writeAttrString(ToStream.java:1935)
>>        at
>> org.apache.xml.serializer.ToStream.processAttributes(ToStream.java:1902)
>>        at org.apache.xml.serializer.ToStream.endElement(ToStream.java:1989)
>>        at
>> org.apache.xalan.transformer.TransformerIdentityImpl.endElement(TransformerIdentityImpl.java:1101)
>>        at org.apache.xml.serializer.TreeWalker.endNode(TreeWalker.java:458)
>>        at
>> org.apache.xml.serializer.TreeWalker.traverse(TreeWalker.java:149)
>>        at
>> org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:389)
>>        at
>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toResult(SourceTransformer.java:125)
>>        at
>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toResult(SourceTransformer.java:103)
>>        at
>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toString(SourceTransformer.java:140)
>>        at
>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toString(SourceTransformer.java:149)
>>        at
>> org.apache.servicemix.jbi.messaging.MessageExchangeImpl.display(MessageExchangeImpl.java:769)
>>        at
>> org.apache.servicemix.jbi.messaging.MessageExchangeImpl.toString(MessageExchangeImpl.java:743)
>>        at java.lang.String.valueOf(String.java:2827)
>>        at java.lang.StringBuilder.append(StringBuilder.java:115)
>>        at
>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:136)
>>        at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>>        at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>        at java.lang.Thread.run(Thread.java:619)
>>
>> I am starting my servicemix with an max Heap of 1024Mbyte.
>>
>>
>> -----
>> Brockhaus GmbH
>> COMPETITIVE THROUGH KNOWLEDGE
>>
>> Web:  http://www.brockhaus-gruppe.de www.brockhaus-gruppe.de  /
>> http://www.brockhaus-group.com www.brockhaus-group.com
>> --
>> View this message in context: http://www.nabble.com/Huge-amount-of-data-%28about-100-MByte%29-tp19285331p19285331.html
>> Sent from the ServiceMix - User mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Cheers,
> Guillaume Nodet
> ------------------------
> Blog: http://gnodet.blogspot.com/
>

Re: Huge amount of data (about 100 MByte)

Posted by Guillaume Nodet <gn...@gmail.com>.
You need to turn off DEBUG logging, else the exchanges are displayed,
thus the whole xml document is loaded in memory and parsed.
Alternatively, you can set the system property named
"org.apache.servicemix.preserveContent" to "true" if you want to keep
DEBUG logging.

On Wed, Sep 3, 2008 at 10:53 AM, steff aka sid
<sr...@brockhaus-gruppe.de> wrote:
>
> Hi,
> its me again :) I'am trying to handle a huge amount of data with servicemix
> but always get an error message (I'am not so familar with tuning smx etc.).
> So what I want to do is the following:
>
> 1. Get an xml file with ftp (see my other posts x) ). This xml files are
> about 100 to 200 of MByte.
> 2. This files are send as a message to a bean component and this bean
> component has to do some transformations by using Smooks. This sending
> fails.
> 3. After the transformation the transformed files which are shrinked at
> about 50Mbyte has to be send to another ftp or folder.
>
> My questions:
> - What is the best flow to use in such a use case, where can I define the
> flow to use?
> - What parameters affect this issue?
> - How can I receive the message as Stream?
>
> Regards
> /Steffen
>
> P.s. Here is the Exception I got:
>
> Exception in thread "pool-flow.seda.servicemix-bean-thread-1"
> java.lang.OutOfMemoryError: Java heap space
>        at java.util.Arrays.copyOf(Arrays.java:2882)
>        at
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
>        at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572)
>        at java.lang.StringBuffer.append(StringBuffer.java:320)
>        at java.io.StringWriter.write(StringWriter.java:60)
>        at
> org.apache.xml.serializer.ToStream.writeAttrString(ToStream.java:1935)
>        at
> org.apache.xml.serializer.ToStream.processAttributes(ToStream.java:1902)
>        at org.apache.xml.serializer.ToStream.endElement(ToStream.java:1989)
>        at
> org.apache.xalan.transformer.TransformerIdentityImpl.endElement(TransformerIdentityImpl.java:1101)
>        at org.apache.xml.serializer.TreeWalker.endNode(TreeWalker.java:458)
>        at
> org.apache.xml.serializer.TreeWalker.traverse(TreeWalker.java:149)
>        at
> org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:389)
>        at
> org.apache.servicemix.jbi.jaxp.SourceTransformer.toResult(SourceTransformer.java:125)
>        at
> org.apache.servicemix.jbi.jaxp.SourceTransformer.toResult(SourceTransformer.java:103)
>        at
> org.apache.servicemix.jbi.jaxp.SourceTransformer.toString(SourceTransformer.java:140)
>        at
> org.apache.servicemix.jbi.jaxp.SourceTransformer.toString(SourceTransformer.java:149)
>        at
> org.apache.servicemix.jbi.messaging.MessageExchangeImpl.display(MessageExchangeImpl.java:769)
>        at
> org.apache.servicemix.jbi.messaging.MessageExchangeImpl.toString(MessageExchangeImpl.java:743)
>        at java.lang.String.valueOf(String.java:2827)
>        at java.lang.StringBuilder.append(StringBuilder.java:115)
>        at
> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:136)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>        at java.lang.Thread.run(Thread.java:619)
>
> I am starting my servicemix with an max Heap of 1024Mbyte.
>
>
> -----
> Brockhaus GmbH
> COMPETITIVE THROUGH KNOWLEDGE
>
> Web:  http://www.brockhaus-gruppe.de www.brockhaus-gruppe.de  /
> http://www.brockhaus-group.com www.brockhaus-group.com
> --
> View this message in context: http://www.nabble.com/Huge-amount-of-data-%28about-100-MByte%29-tp19285331p19285331.html
> Sent from the ServiceMix - User mailing list archive at Nabble.com.
>
>



-- 
Cheers,
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/