You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by cristisor <cr...@yahoo.com> on 2013/03/06 15:02:07 UTC
Aggregate exchanges until before EoF
Hello,
Hi guys,
I'm processing a file line by line and I want to aggregate all the exchanges
(lines) until the one that has "Exchange.SPLIT_COMPLETE", basically I want
to send the aggregated exchanges just before aggregating the final one. When
the last exchange arrives it should be in the case where oldExchange is
null. Is this possible?
--
View this message in context: http://camel.465427.n5.nabble.com/Aggregate-exchanges-until-before-EoF-tp5728680.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Aggregate exchanges until before EoF
Posted by Claus Ibsen <cl...@gmail.com>.
On Wed, Mar 6, 2013 at 4:05 PM, cristisor <cr...@yahoo.com> wrote:
> Yes, I already used aggregators to build batches of data that I will later
> use for db operations and they work great. But I don't know how to implement
> a custom splitter that, for example, sets the file complet flag only on the
> last split line, is this possible?
>
Yeah its possible but you would need to write a custom code for that.
You can wrap the existing from Camel and then read the data one-ahead.
And then you would be able to know the the file is EOF when you are at
the 2nd last line.
I guess you want streaming mode, if you are not in streaming mode,
then the splitter has a total size as well.
Then you would know if you are at total - 1 currently.
But in streaming mode the splitter does not know the total size, and
has only current index and if its completed.
PS: There is also a group option on the tokenizer. You may be able to
enhance that existing code to make it discard X last lines.
Not sure if this is a common use-case for others, to be able to group
X lines and discard Y lines at the end.
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Aggregate-exchanges-until-before-EoF-tp5728680p5728689.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: Aggregate exchanges until before EoF
Posted by cristisor <cr...@yahoo.com>.
Yes, I already used aggregators to build batches of data that I will later
use for db operations and they work great. But I don't know how to implement
a custom splitter that, for example, sets the file complet flag only on the
last split line, is this possible?
--
View this message in context: http://camel.465427.n5.nabble.com/Aggregate-exchanges-until-before-EoF-tp5728680p5728689.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Aggregate exchanges until before EoF
Posted by Claus Ibsen <cl...@gmail.com>.
On Wed, Mar 6, 2013 at 3:51 PM, cristisor <cr...@yahoo.com> wrote:
> No, I want to send that last line without aggregating it, or aggregating it
> as the first and only exchange of that aggregation strategy. What I actually
> need is to have a final exchange, containing a line from the file and the
> FILE_COMPLETE header, that gets sent through a few service units, each
> processes its data, and the last service unit finds that it's the end of the
> file and resets a flag into the database so that the next file from the
> folder gets read. If I aggregate it with other exchanges, then split
> everything into the next service unit and so on I can't keep track of the
> end of the file, I risk having more than one exchange with this header.
>
> If you need a better explanation please let me know.
> Thanks.
>
In your aggregation strategy implementation you can "merge" the data
anyhow you like.
So you can check that its the last line, and put it on the front of
the data if that is what you wanted to do etc.
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Aggregate-exchanges-until-before-EoF-tp5728680p5728686.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: Aggregate exchanges until before EoF
Posted by cristisor <cr...@yahoo.com>.
No, I want to send that last line without aggregating it, or aggregating it
as the first and only exchange of that aggregation strategy. What I actually
need is to have a final exchange, containing a line from the file and the
FILE_COMPLETE header, that gets sent through a few service units, each
processes its data, and the last service unit finds that it's the end of the
file and resets a flag into the database so that the next file from the
folder gets read. If I aggregate it with other exchanges, then split
everything into the next service unit and so on I can't keep track of the
end of the file, I risk having more than one exchange with this header.
If you need a better explanation please let me know.
Thanks.
--
View this message in context: http://camel.465427.n5.nabble.com/Aggregate-exchanges-until-before-EoF-tp5728680p5728686.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Aggregate exchanges until before EoF
Posted by Claus Ibsen <cl...@gmail.com>.
On Wed, Mar 6, 2013 at 3:02 PM, cristisor <cr...@yahoo.com> wrote:
> Hello,
>
> Hi guys,
>
> I'm processing a file line by line and I want to aggregate all the exchanges
> (lines) until the one that has "Exchange.SPLIT_COMPLETE", basically I want
> to send the aggregated exchanges just before aggregating the final one. When
> the last exchange arrives it should be in the case where oldExchange is
> null. Is this possible?
>
Do you want to skip that last line?
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Aggregate-exchanges-until-before-EoF-tp5728680.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen