You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Tarjei Huse <ta...@scanmine.com> on 2010/09/29 12:44:35 UTC
Streaming splitter with custom split method
Hi, I got a huge file I want to split into smaller chunks and send each
chunk as a new message using the splitter. The file contains Protobuf
objects so I cannot use any of the normal splitting methods, but I want
to use the streaming() method on the route. How do I create a splitter
that supports this?
I.e:
FileSplitter fs = new FileSplitter();
from("file://myfile")
.routeId(InjectFiles.class.getSimpleName())
.split().method(fs,
"splitMessage").streaming().stopOnException()
--
Regards / Med vennlig hilsen
Tarjei Huse
Mobil: 920 63 413
Re: Streaming splitter with custom split method
Posted by Claus Ibsen <cl...@gmail.com>.
Yeah put the expression as a parameter to split, eg put it inside
split( HERE ).streaming()
On Thu, Sep 30, 2010 at 2:34 PM, Tarjei Huse <ta...@scanmine.com> wrote:
>
> Den 30. sep. 2010 kl. 14:05 skrev Claus Ibsen <cl...@gmail.com>:
>
>> On Thu, Sep 30, 2010 at 12:22 PM, Tarjei Huse <ta...@scanmine.com> wrote:
>>> On 09/29/2010 01:31 PM, Claus Ibsen wrote:
>>>> On Wed, Sep 29, 2010 at 12:44 PM, Tarjei Huse <ta...@scanmine.com> wrote:
>>>>> Hi, I got a huge file I want to split into smaller chunks and send each
>>>>> chunk as a new message using the splitter. The file contains Protobuf
>>>>> objects so I cannot use any of the normal splitting methods, but I want
>>>>> to use the streaming() method on the route. How do I create a splitter
>>>>> that supports this?
>>>>>
>>>> Just let the method return an Iterator type and then you can return
>>>> the data in chunks as you like.
>>>> The streaming mode will then walk the iterator on demand until hasNext
>>>> returns false.
>>> Duh!
>>>
>>> I'm quite sure that you have answered this question to me before and I
>>> have forgotten. Sorry about that. I'll try to update the wiki so I do
>>> not repeat my questions.
>>>
>>> One thing though, do I have to set streaming mode explicitly when using
>>> an iterator or will that come implicitly then?
>>>
>>
>> No the streaming() mode actually only controls whether Camel will walk
>> the iterate one time, or on-the-fly.
>> One time means it walks it beforehand and stores the result in memory,
>> so it knows the complete size before splitting begins.
>> That information is then populated on the Exchange. So you know that
>> the total size is eg 8. For streaming() you will not know the total
>> size.
> Ok. I'm not by my computer now, but I think I tried .split().method(myobj,'method').streaming() and got a compilation error.
>
> Mvh
> Tarjei Huse
> Mob 930 63 413
>
>
>>
>>
>>
>>> Regards,
>>> Tarjei
>>>>
>>>>> I.e:
>>>>> FileSplitter fs = new FileSplitter();
>>>>> from("file://myfile")
>>>>> .routeId(InjectFiles.class.getSimpleName())
>>>>> .split().method(fs,
>>>>> "splitMessage").streaming().stopOnException()
>>>>>
>>>>>
>>>>> --
>>>>> Regards / Med vennlig hilsen
>>>>> Tarjei Huse
>>>>> Mobil: 920 63 413
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Regards / Med vennlig hilsen
>>> Tarjei Huse
>>> Mobil: 920 63 413
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: Streaming splitter with custom split method
Posted by Tarjei Huse <ta...@scanmine.com>.
Den 30. sep. 2010 kl. 14:05 skrev Claus Ibsen <cl...@gmail.com>:
> On Thu, Sep 30, 2010 at 12:22 PM, Tarjei Huse <ta...@scanmine.com> wrote:
>> On 09/29/2010 01:31 PM, Claus Ibsen wrote:
>>> On Wed, Sep 29, 2010 at 12:44 PM, Tarjei Huse <ta...@scanmine.com> wrote:
>>>> Hi, I got a huge file I want to split into smaller chunks and send each
>>>> chunk as a new message using the splitter. The file contains Protobuf
>>>> objects so I cannot use any of the normal splitting methods, but I want
>>>> to use the streaming() method on the route. How do I create a splitter
>>>> that supports this?
>>>>
>>> Just let the method return an Iterator type and then you can return
>>> the data in chunks as you like.
>>> The streaming mode will then walk the iterator on demand until hasNext
>>> returns false.
>> Duh!
>>
>> I'm quite sure that you have answered this question to me before and I
>> have forgotten. Sorry about that. I'll try to update the wiki so I do
>> not repeat my questions.
>>
>> One thing though, do I have to set streaming mode explicitly when using
>> an iterator or will that come implicitly then?
>>
>
> No the streaming() mode actually only controls whether Camel will walk
> the iterate one time, or on-the-fly.
> One time means it walks it beforehand and stores the result in memory,
> so it knows the complete size before splitting begins.
> That information is then populated on the Exchange. So you know that
> the total size is eg 8. For streaming() you will not know the total
> size.
Ok. I'm not by my computer now, but I think I tried .split().method(myobj,'method').streaming() and got a compilation error.
Mvh
Tarjei Huse
Mob 930 63 413
>
>
>
>> Regards,
>> Tarjei
>>>
>>>> I.e:
>>>> FileSplitter fs = new FileSplitter();
>>>> from("file://myfile")
>>>> .routeId(InjectFiles.class.getSimpleName())
>>>> .split().method(fs,
>>>> "splitMessage").streaming().stopOnException()
>>>>
>>>>
>>>> --
>>>> Regards / Med vennlig hilsen
>>>> Tarjei Huse
>>>> Mobil: 920 63 413
>>>>
>>>>
>>>
>>>
>>
>>
>> --
>> Regards / Med vennlig hilsen
>> Tarjei Huse
>> Mobil: 920 63 413
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
Re: Streaming splitter with custom split method
Posted by Claus Ibsen <cl...@gmail.com>.
On Thu, Sep 30, 2010 at 12:22 PM, Tarjei Huse <ta...@scanmine.com> wrote:
> On 09/29/2010 01:31 PM, Claus Ibsen wrote:
>> On Wed, Sep 29, 2010 at 12:44 PM, Tarjei Huse <ta...@scanmine.com> wrote:
>>> Hi, I got a huge file I want to split into smaller chunks and send each
>>> chunk as a new message using the splitter. The file contains Protobuf
>>> objects so I cannot use any of the normal splitting methods, but I want
>>> to use the streaming() method on the route. How do I create a splitter
>>> that supports this?
>>>
>> Just let the method return an Iterator type and then you can return
>> the data in chunks as you like.
>> The streaming mode will then walk the iterator on demand until hasNext
>> returns false.
> Duh!
>
> I'm quite sure that you have answered this question to me before and I
> have forgotten. Sorry about that. I'll try to update the wiki so I do
> not repeat my questions.
>
> One thing though, do I have to set streaming mode explicitly when using
> an iterator or will that come implicitly then?
>
No the streaming() mode actually only controls whether Camel will walk
the iterate one time, or on-the-fly.
One time means it walks it beforehand and stores the result in memory,
so it knows the complete size before splitting begins.
That information is then populated on the Exchange. So you know that
the total size is eg 8. For streaming() you will not know the total
size.
> Regards,
> Tarjei
>>
>>> I.e:
>>> FileSplitter fs = new FileSplitter();
>>> from("file://myfile")
>>> .routeId(InjectFiles.class.getSimpleName())
>>> .split().method(fs,
>>> "splitMessage").streaming().stopOnException()
>>>
>>>
>>> --
>>> Regards / Med vennlig hilsen
>>> Tarjei Huse
>>> Mobil: 920 63 413
>>>
>>>
>>
>>
>
>
> --
> Regards / Med vennlig hilsen
> Tarjei Huse
> Mobil: 920 63 413
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: Streaming splitter with custom split method
Posted by Tarjei Huse <ta...@scanmine.com>.
On 09/29/2010 01:31 PM, Claus Ibsen wrote:
> On Wed, Sep 29, 2010 at 12:44 PM, Tarjei Huse <ta...@scanmine.com> wrote:
>> Hi, I got a huge file I want to split into smaller chunks and send each
>> chunk as a new message using the splitter. The file contains Protobuf
>> objects so I cannot use any of the normal splitting methods, but I want
>> to use the streaming() method on the route. How do I create a splitter
>> that supports this?
>>
> Just let the method return an Iterator type and then you can return
> the data in chunks as you like.
> The streaming mode will then walk the iterator on demand until hasNext
> returns false.
Duh!
I'm quite sure that you have answered this question to me before and I
have forgotten. Sorry about that. I'll try to update the wiki so I do
not repeat my questions.
One thing though, do I have to set streaming mode explicitly when using
an iterator or will that come implicitly then?
Regards,
Tarjei
>
>> I.e:
>> FileSplitter fs = new FileSplitter();
>> from("file://myfile")
>> .routeId(InjectFiles.class.getSimpleName())
>> .split().method(fs,
>> "splitMessage").streaming().stopOnException()
>>
>>
>> --
>> Regards / Med vennlig hilsen
>> Tarjei Huse
>> Mobil: 920 63 413
>>
>>
>
>
--
Regards / Med vennlig hilsen
Tarjei Huse
Mobil: 920 63 413
Re: Streaming splitter with custom split method
Posted by Claus Ibsen <cl...@gmail.com>.
On Wed, Sep 29, 2010 at 12:44 PM, Tarjei Huse <ta...@scanmine.com> wrote:
> Hi, I got a huge file I want to split into smaller chunks and send each
> chunk as a new message using the splitter. The file contains Protobuf
> objects so I cannot use any of the normal splitting methods, but I want
> to use the streaming() method on the route. How do I create a splitter
> that supports this?
>
Just let the method return an Iterator type and then you can return
the data in chunks as you like.
The streaming mode will then walk the iterator on demand until hasNext
returns false.
> I.e:
> FileSplitter fs = new FileSplitter();
> from("file://myfile")
> .routeId(InjectFiles.class.getSimpleName())
> .split().method(fs,
> "splitMessage").streaming().stopOnException()
>
>
> --
> Regards / Med vennlig hilsen
> Tarjei Huse
> Mobil: 920 63 413
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus