You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Claus Straube <cl...@catify.com> on 2011/03/25 18:19:56 UTC
Streaming huge EDI file with camel
Hi all.
I've posted this on the smooks list, but there was nobody who could give
me a hint...
I'm trying to transform a huge edi file to xml. For reading the file I'm
using camel. Here is my route:
<snip>
from("file://target/in?noop=true")
.to("smooks://src/main/resources/smooks-config.xml")
.to("file://target/processing");
<snip>
My smooks config looks like this:
<snip>
<smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd"
xmlns:core="http://www.milyn.org/xsd/smooks/smooks-core-1.4.xsd"
xmlns:edi="http://www.milyn.org/xsd/smooks/edi-1.4.xsd">
<!-- Configure the EDI Reader to process the message stream into a stream
of SAX events. -->
<edi:reader mappingModel="mappingmodels/foo.xml" ignoreNewLines="true" />
<core:exports>
<core:result type="org.milyn.payload.StringResult"/>
</core:exports>
</smooks-resource-list>
<snip>
I'm getting a out of memory with a 30 MB edi file, but have to transform
250 MB files. So I need a hint how to get a data stream into smooks and
out again. Is there a sample who to handle big EDI files? Is smooks the
right tool for this?
Thanks in advance - Claus
Re: Streaming huge EDI file with camel
Posted by Charles Moulliard <cm...@gmail.com>.
Hi Claus,
Have you tried to split (if possible) and uses streams option afte the
from(file) component
from("file:target/inventory")
.split(body().tokenize("\n")).streaming()
Regards,
Charles Moulliard
Sr. Principal Solution Architect - FuseSource
Apache Committer
Blog : http://cmoulliard.blogspot.com
Twitter : http://twitter.com/cmoulliard
Linkedin : http://www.linkedin.com/in/charlesmoulliard
Skype: cmoulliard
On Fri, Mar 25, 2011 at 6:19 PM, Claus Straube <cl...@catify.com> wrote:
> Hi all.
>
> I've posted this on the smooks list, but there was nobody who could give me
> a hint...
>
> I'm trying to transform a huge edi file to xml. For reading the file I'm
> using camel. Here is my route:
>
> <snip>
> from("file://target/in?noop=true")
> .to("smooks://src/main/resources/smooks-config.xml")
> .to("file://target/processing");
> <snip>
>
> My smooks config looks like this:
>
> <snip>
> <smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd"
> xmlns:core="http://www.milyn.org/xsd/smooks/smooks-core-1.4.xsd"
> xmlns:edi="http://www.milyn.org/xsd/smooks/edi-1.4.xsd">
>
> <!-- Configure the EDI Reader to process the message stream into a stream
> of SAX events. -->
> <edi:reader mappingModel="mappingmodels/foo.xml" ignoreNewLines="true" />
>
> <core:exports>
> <core:result type="org.milyn.payload.StringResult"/>
> </core:exports>
>
> </smooks-resource-list>
> <snip>
>
> I'm getting a out of memory with a 30 MB edi file, but have to transform 250
> MB files. So I need a hint how to get a data stream into smooks and out
> again. Is there a sample who to handle big EDI files? Is smooks the right
> tool for this?
>
> Thanks in advance - Claus
>
Re: Streaming huge EDI file with camel
Posted by Daniel Bevenius <da...@gmail.com>.
The thread on the Smooks list is being answered and is active...we did not
ignore the post.
Den 27 mar 2011 11.03, "Claus Ibsen" <cl...@gmail.com> skrev:
On Sat, Mar 26, 2011 at 1:04 PM, Claus Straube <cl...@catify.com>
wrote:
> Hi Claus,
>
> I k...
Yeah thats fine. I would assume Smooks would be able to deal with big
files in a streaming fashion.
So I wonder what's the issue. Maybe there is a switch/option you need to
enable?
Or maybe that payload result has a different type for streaming
instead of StringResult?
> On 26.03.2011 11:37, Claus Ibsen wrote:
>>
>> Hi
>>
>> If you refer to the old smooks component...
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com...
Re: Streaming huge EDI file with camel
Posted by Claus Ibsen <cl...@gmail.com>.
On Sat, Mar 26, 2011 at 1:04 PM, Claus Straube <cl...@catify.com> wrote:
> Hi Claus,
>
> I know thats deprecated and I'm using their stuff. The post to the camel
> list was because it's still 'camel' and there is a chance that anybody
> community has used it before ;)
>
Yeah thats fine. I would assume Smooks would be able to deal with big
files in a streaming fashion.
So I wonder what's the issue. Maybe there is a switch/option you need to enable?
Or maybe that payload result has a different type for streaming
instead of StringResult?
> On 26.03.2011 11:37, Claus Ibsen wrote:
>>
>> Hi
>>
>> If you refer to the old smooks component from camel-extra then that's
>> @deprecated.
>>
>> Smooks framework now provides Camel integration out of the box, so use
>> their stuff for that.
>>
>>
>> On Fri, Mar 25, 2011 at 6:19 PM, Claus Straube<cl...@catify.com>
>> wrote:
>>>
>>> Hi all.
>>>
>>> I've posted this on the smooks list, but there was nobody who could give
>>> me
>>> a hint...
>>>
>>> I'm trying to transform a huge edi file to xml. For reading the file I'm
>>> using camel. Here is my route:
>>>
>>> <snip>
>>> from("file://target/in?noop=true")
>>> .to("smooks://src/main/resources/smooks-config.xml")
>>> .to("file://target/processing");
>>> <snip>
>>>
>>> My smooks config looks like this:
>>>
>>> <snip>
>>> <smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd"
>>> xmlns:core="http://www.milyn.org/xsd/smooks/smooks-core-1.4.xsd"
>>> xmlns:edi="http://www.milyn.org/xsd/smooks/edi-1.4.xsd">
>>>
>>> <!-- Configure the EDI Reader to process the message stream into a stream
>>> of SAX events. -->
>>> <edi:reader mappingModel="mappingmodels/foo.xml" ignoreNewLines="true" />
>>>
>>> <core:exports>
>>> <core:result type="org.milyn.payload.StringResult"/>
>>> </core:exports>
>>>
>>> </smooks-resource-list>
>>> <snip>
>>>
>>> I'm getting a out of memory with a 30 MB edi file, but have to transform
>>> 250
>>> MB files. So I need a hint how to get a data stream into smooks and out
>>> again. Is there a sample who to handle big EDI files? Is smooks the right
>>> tool for this?
>>>
>>> Thanks in advance - Claus
>>>
>>
>>
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: Streaming huge EDI file with camel
Posted by Claus Straube <cl...@catify.com>.
Hi Claus,
I know thats deprecated and I'm using their stuff. The post to the camel
list was because it's still 'camel' and there is a chance that anybody
community has used it before ;)
On 26.03.2011 11:37, Claus Ibsen wrote:
> Hi
>
> If you refer to the old smooks component from camel-extra then that's
> @deprecated.
>
> Smooks framework now provides Camel integration out of the box, so use
> their stuff for that.
>
>
> On Fri, Mar 25, 2011 at 6:19 PM, Claus Straube<cl...@catify.com> wrote:
>> Hi all.
>>
>> I've posted this on the smooks list, but there was nobody who could give me
>> a hint...
>>
>> I'm trying to transform a huge edi file to xml. For reading the file I'm
>> using camel. Here is my route:
>>
>> <snip>
>> from("file://target/in?noop=true")
>> .to("smooks://src/main/resources/smooks-config.xml")
>> .to("file://target/processing");
>> <snip>
>>
>> My smooks config looks like this:
>>
>> <snip>
>> <smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd"
>> xmlns:core="http://www.milyn.org/xsd/smooks/smooks-core-1.4.xsd"
>> xmlns:edi="http://www.milyn.org/xsd/smooks/edi-1.4.xsd">
>>
>> <!-- Configure the EDI Reader to process the message stream into a stream
>> of SAX events. -->
>> <edi:reader mappingModel="mappingmodels/foo.xml" ignoreNewLines="true" />
>>
>> <core:exports>
>> <core:result type="org.milyn.payload.StringResult"/>
>> </core:exports>
>>
>> </smooks-resource-list>
>> <snip>
>>
>> I'm getting a out of memory with a 30 MB edi file, but have to transform 250
>> MB files. So I need a hint how to get a data stream into smooks and out
>> again. Is there a sample who to handle big EDI files? Is smooks the right
>> tool for this?
>>
>> Thanks in advance - Claus
>>
>
>
Re: Streaming huge EDI file with camel
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
If you refer to the old smooks component from camel-extra then that's
@deprecated.
Smooks framework now provides Camel integration out of the box, so use
their stuff for that.
On Fri, Mar 25, 2011 at 6:19 PM, Claus Straube <cl...@catify.com> wrote:
> Hi all.
>
> I've posted this on the smooks list, but there was nobody who could give me
> a hint...
>
> I'm trying to transform a huge edi file to xml. For reading the file I'm
> using camel. Here is my route:
>
> <snip>
> from("file://target/in?noop=true")
> .to("smooks://src/main/resources/smooks-config.xml")
> .to("file://target/processing");
> <snip>
>
> My smooks config looks like this:
>
> <snip>
> <smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd"
> xmlns:core="http://www.milyn.org/xsd/smooks/smooks-core-1.4.xsd"
> xmlns:edi="http://www.milyn.org/xsd/smooks/edi-1.4.xsd">
>
> <!-- Configure the EDI Reader to process the message stream into a stream
> of SAX events. -->
> <edi:reader mappingModel="mappingmodels/foo.xml" ignoreNewLines="true" />
>
> <core:exports>
> <core:result type="org.milyn.payload.StringResult"/>
> </core:exports>
>
> </smooks-resource-list>
> <snip>
>
> I'm getting a out of memory with a 30 MB edi file, but have to transform 250
> MB files. So I need a hint how to get a data stream into smooks and out
> again. Is there a sample who to handle big EDI files? Is smooks the right
> tool for this?
>
> Thanks in advance - Claus
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: Streaming huge EDI file with camel
Posted by Claus Straube <cl...@catify.com>.
Thank's Danial. Was a good idea to link the thread. To make it clear -
I not ignored at the mylin list. But my problem to get the sax parser
running with <edi:reader... is not solved yet.
Best regards - Claus
Am Sonntag, den 27.03.2011, 15:04 +0200 schrieb Daniel Bevenius
<da...@gmail.com>:
> Sorry about that, here is the Smooks thread:
>
> http://old.nabble.com/Streaming-huge-EDI-file-with-camel-td31238361.html
>
> Regards,
>
> /Daniel
>
> On 27 March 2011 11:27, Claus Ibsen <cl...@gmail.com> wrote:
>
>> On Fri, Mar 25, 2011 at 6:19 PM, Claus Straube
>> <cl...@catify.com>
>> wrote:
>> > Hi all.
>> >
>> > I've posted this on the smooks list, but there was nobody who
>> could give
>> me
>> > a hint...
>> >
>>
>> btw its a good idea to post a link to that original thread so people
>> can look as well.
>>
>>
>>
>> > I'm trying to transform a huge edi file to xml. For reading the
>> file I'm
>> > using camel. Here is my route:
>> >
>> > <snip>
>> > from("file://target/in?noop=true")
>> > .to("smooks://src/main/resources/smooks-config.xml")
>> > .to("file://target/processing");
>> > <snip>
>> >
>> > My smooks config looks like this:
>> >
>> > <snip>
>> > <smooks-resource-list
>> xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd"
>> >
>> xmlns:core="http://www.milyn.org/xsd/smooks/smooks-core-1.4.xsd"
>> > xmlns:edi="http://www.milyn.org/xsd/smooks/edi-1.4.xsd">
>> >
>> > <!-- Configure the EDI Reader to process the message stream into a
>> stream
>> > of SAX events. -->
>> > <edi:reader mappingModel="mappingmodels/foo.xml"
>> ignoreNewLines="true" />
>> >
>> > <core:exports>
>> > <core:result type="org.milyn.payload.StringResult"/>
>> > </core:exports>
>> >
>> > </smooks-resource-list>
>> > <snip>
>> >
>> > I'm getting a out of memory with a 30 MB edi file, but have to
>> transform
>> 250
>> > MB files. So I need a hint how to get a data stream into smooks
>> and out
>> > again. Is there a sample who to handle big EDI files? Is smooks
>> the right
>> > tool for this?
>> >
>> > Thanks in advance - Claus
>> >
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> FuseSource
>> Email: cibsen@fusesource.com
>> Web: http://fusesource.com
>> Twitter: davsclaus
>> Blog: http://davsclaus.blogspot.com/
>> Author of Camel in Action: http://www.manning.com/ibsen/
>>
Re: Streaming huge EDI file with camel
Posted by Claus Ibsen <cl...@gmail.com>.
On Sun, Mar 27, 2011 at 3:04 PM, Daniel Bevenius
<da...@gmail.com> wrote:
> Sorry about that, here is the Smooks thread:
> http://old.nabble.com/Streaming-huge-EDI-file-with-camel-td31238361.html
>
Thanks, however I address that to Claus who started this post here.
Its just a good idea to refer to X when you mention it, so ppl can check it out.
> Regards,
>
> /Daniel
>
> On 27 March 2011 11:27, Claus Ibsen <cl...@gmail.com> wrote:
>
>> On Fri, Mar 25, 2011 at 6:19 PM, Claus Straube <cl...@catify.com>
>> wrote:
>> > Hi all.
>> >
>> > I've posted this on the smooks list, but there was nobody who could give
>> me
>> > a hint...
>> >
>>
>> btw its a good idea to post a link to that original thread so people
>> can look as well.
>>
>>
>>
>> > I'm trying to transform a huge edi file to xml. For reading the file I'm
>> > using camel. Here is my route:
>> >
>> > <snip>
>> > from("file://target/in?noop=true")
>> > .to("smooks://src/main/resources/smooks-config.xml")
>> > .to("file://target/processing");
>> > <snip>
>> >
>> > My smooks config looks like this:
>> >
>> > <snip>
>> > <smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd"
>> > xmlns:core="http://www.milyn.org/xsd/smooks/smooks-core-1.4.xsd"
>> > xmlns:edi="http://www.milyn.org/xsd/smooks/edi-1.4.xsd">
>> >
>> > <!-- Configure the EDI Reader to process the message stream into a stream
>> > of SAX events. -->
>> > <edi:reader mappingModel="mappingmodels/foo.xml" ignoreNewLines="true" />
>> >
>> > <core:exports>
>> > <core:result type="org.milyn.payload.StringResult"/>
>> > </core:exports>
>> >
>> > </smooks-resource-list>
>> > <snip>
>> >
>> > I'm getting a out of memory with a 30 MB edi file, but have to transform
>> 250
>> > MB files. So I need a hint how to get a data stream into smooks and out
>> > again. Is there a sample who to handle big EDI files? Is smooks the right
>> > tool for this?
>> >
>> > Thanks in advance - Claus
>> >
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> FuseSource
>> Email: cibsen@fusesource.com
>> Web: http://fusesource.com
>> Twitter: davsclaus
>> Blog: http://davsclaus.blogspot.com/
>> Author of Camel in Action: http://www.manning.com/ibsen/
>>
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: Streaming huge EDI file with camel
Posted by Daniel Bevenius <da...@gmail.com>.
Sorry about that, here is the Smooks thread:
http://old.nabble.com/Streaming-huge-EDI-file-with-camel-td31238361.html
Regards,
/Daniel
On 27 March 2011 11:27, Claus Ibsen <cl...@gmail.com> wrote:
> On Fri, Mar 25, 2011 at 6:19 PM, Claus Straube <cl...@catify.com>
> wrote:
> > Hi all.
> >
> > I've posted this on the smooks list, but there was nobody who could give
> me
> > a hint...
> >
>
> btw its a good idea to post a link to that original thread so people
> can look as well.
>
>
>
> > I'm trying to transform a huge edi file to xml. For reading the file I'm
> > using camel. Here is my route:
> >
> > <snip>
> > from("file://target/in?noop=true")
> > .to("smooks://src/main/resources/smooks-config.xml")
> > .to("file://target/processing");
> > <snip>
> >
> > My smooks config looks like this:
> >
> > <snip>
> > <smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd"
> > xmlns:core="http://www.milyn.org/xsd/smooks/smooks-core-1.4.xsd"
> > xmlns:edi="http://www.milyn.org/xsd/smooks/edi-1.4.xsd">
> >
> > <!-- Configure the EDI Reader to process the message stream into a stream
> > of SAX events. -->
> > <edi:reader mappingModel="mappingmodels/foo.xml" ignoreNewLines="true" />
> >
> > <core:exports>
> > <core:result type="org.milyn.payload.StringResult"/>
> > </core:exports>
> >
> > </smooks-resource-list>
> > <snip>
> >
> > I'm getting a out of memory with a 30 MB edi file, but have to transform
> 250
> > MB files. So I need a hint how to get a data stream into smooks and out
> > again. Is there a sample who to handle big EDI files? Is smooks the right
> > tool for this?
> >
> > Thanks in advance - Claus
> >
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus
> Blog: http://davsclaus.blogspot.com/
> Author of Camel in Action: http://www.manning.com/ibsen/
>
Re: Streaming huge EDI file with camel
Posted by Claus Ibsen <cl...@gmail.com>.
On Fri, Mar 25, 2011 at 6:19 PM, Claus Straube <cl...@catify.com> wrote:
> Hi all.
>
> I've posted this on the smooks list, but there was nobody who could give me
> a hint...
>
btw its a good idea to post a link to that original thread so people
can look as well.
> I'm trying to transform a huge edi file to xml. For reading the file I'm
> using camel. Here is my route:
>
> <snip>
> from("file://target/in?noop=true")
> .to("smooks://src/main/resources/smooks-config.xml")
> .to("file://target/processing");
> <snip>
>
> My smooks config looks like this:
>
> <snip>
> <smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd"
> xmlns:core="http://www.milyn.org/xsd/smooks/smooks-core-1.4.xsd"
> xmlns:edi="http://www.milyn.org/xsd/smooks/edi-1.4.xsd">
>
> <!-- Configure the EDI Reader to process the message stream into a stream
> of SAX events. -->
> <edi:reader mappingModel="mappingmodels/foo.xml" ignoreNewLines="true" />
>
> <core:exports>
> <core:result type="org.milyn.payload.StringResult"/>
> </core:exports>
>
> </smooks-resource-list>
> <snip>
>
> I'm getting a out of memory with a 30 MB edi file, but have to transform 250
> MB files. So I need a hint how to get a data stream into smooks and out
> again. Is there a sample who to handle big EDI files? Is smooks the right
> tool for this?
>
> Thanks in advance - Claus
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/