You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Henryk Konsek <he...@gmail.com> on 2012/02/09 13:41:49 UTC
Passing input stream to the route
Hi guys
Recently I've been assigned to the task of implementing callback
interface that provides InputStream of CSV data. My task was to take
the stream I received from the callback and pass it to the Camel to
perform processing on each line read. Something like:
interface DataProviderCallback {
void dataAvailable(InputStream csvData);
}
class CamelDataProviderCallback implements DataProviderCallback {
@Autowire
CamelContext context;
void dataAvailable(InputStream csvDataStream) {
// pass csvDataStream to Camel and do not block
}
}
Of course the mentioned stream of data produces gigabytes of CSV
records so I cannot read it entirely into the memory. I need to read
it line by line (i.e. record by record) and pass each line to the
Camel. I got some good experience with Camel Stream component for such
use cases. Unfortunately I cannot pass existing InputStream instance
to it.
I solved this using background thread (my callback cannot block next
ones) in which I read lines from the stream and pass them to the Camel
one by one (using ProducerTemplate). This solution is fine for my
particular purpose but not elegant in general.
What would be your suggestion to solve this issue? How to you handle
existing InputStream instances?
--
Henryk Konsek
Re: Passing input stream to the route
Posted by Henryk Konsek <he...@gmail.com>.
> You may get some idea by checking out the code of StreamComponent.
> But you still need to find a way to tell the StreamConsumer to pick up the
> Stream when your callback API is called.
I will stick with my custom glue code as it is more readable and
maintainable for my client's team.
Thanks for suggestions Willem.
Best regards.
--
Henryk Konsek
Re: Passing input stream to the route
Posted by Willem Jiang <wi...@gmail.com>.
Hi,
You may get some idea by checking out the code of StreamComponent.
But you still need to find a way to tell the StreamConsumer to pick up
the Stream when your callback API is called.
Maybe you can pass let the callback all the StreamConsumer process
method directly when the stream is ready.
In this way you don't need to use the ProducerTemplate to send the
lines to the camel route.
On Mon Feb 13 18:23:49 2012, Henryk Konsek wrote:
>> In this case you may consider to extends the StreamComponent and pass the
>> csvDataStream into it.
>
> Hi Willem. Thank you for the reply.
>
> Unfortunately I don't get your idea :) Could you elaborate it a little
> bit more? Maybe you could also provide a route / code example.
>
> Kind regards.
>
--
Willem
----------------------------------
FuseSource
Web: http://www.fusesource.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.javaeye.com (Chinese)
Twitter: willemjiang
Weibo: willemjiang
Re: Passing input stream to the route
Posted by Henryk Konsek <he...@gmail.com>.
> In this case you may consider to extends the StreamComponent and pass the
> csvDataStream into it.
Hi Willem. Thank you for the reply.
Unfortunately I don't get your idea :) Could you elaborate it a little
bit more? Maybe you could also provide a route / code example.
Kind regards.
--
Henryk Konsek
Re: Passing input stream to the route
Posted by Willem Jiang <wi...@gmail.com>.
I guess you just need to pass the lines into camel, you don't need to
put the result into other stream.
In this case you may consider to extends the StreamComponent and pass
the csvDataStream into it.
Then you don't need to use the producer template any more.
On Thu Feb 9 20:41:49 2012, Henryk Konsek wrote:
> Hi guys
>
> Recently I've been assigned to the task of implementing callback
> interface that provides InputStream of CSV data. My task was to take
> the stream I received from the callback and pass it to the Camel to
> perform processing on each line read. Something like:
>
> interface DataProviderCallback {
>
> void dataAvailable(InputStream csvData);
>
> }
>
> class CamelDataProviderCallback implements DataProviderCallback {
>
> @Autowire
> CamelContext context;
>
> void dataAvailable(InputStream csvDataStream) {
> // pass csvDataStream to Camel and do not block
> }
>
> }
>
> Of course the mentioned stream of data produces gigabytes of CSV
> records so I cannot read it entirely into the memory. I need to read
> it line by line (i.e. record by record) and pass each line to the
> Camel. I got some good experience with Camel Stream component for such
> use cases. Unfortunately I cannot pass existing InputStream instance
> to it.
>
> I solved this using background thread (my callback cannot block next
> ones) in which I read lines from the stream and pass them to the Camel
> one by one (using ProducerTemplate). This solution is fine for my
> particular purpose but not elegant in general.
>
> What would be your suggestion to solve this issue? How to you handle
> existing InputStream instances?
>
--
Willem
----------------------------------
FuseSource
Web: http://www.fusesource.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.javaeye.com (Chinese)
Twitter: willemjiang
Weibo: willemjiang