You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Baris Acar <ba...@acar.org.uk> on 2014/10/27 22:30:52 UTC
Aggregate up to N "ASAP"
Hi,
Is there a way to aggregate the first 100 messages from a seda if >=100 are
available, but just take however many are available if there are fewer?
I think I want aggregate(...).completionSize(100).completionTimeout(0), ie
complete on 100 items or 0, whichever condition is met "first", but I'm not
convinced that this will actually do the right thing.
Is there a known correct solution for this?
Thanks
Baris.
Re: Aggregate up to N "ASAP"
Posted by Baris Acar <ba...@acar.org.uk>.
Timeout=0 is the very essence of what I'm trying to express, I'm not sure how I can be both "correct" and "making no sense" :-)
I'd have guessed this was a common use case and would be a useful feature. Consider a processor posting to a seda which consumes messages and posts them in batch to a variable speed sink, say a network resource. It's reasonable for some applications to want to avoid any pointless idle time if there are 10 messages available immediately that could be dealt with. If the network link is being slower than expected then by the time it next polls the seda, there may be 100 messages to pick.
I think the problem is that completionSize specifies both a maximum and minimum size. Ideally I would specify a range (eg "wait for at least 5 messages; if more than 5 messages are already available, take them all, up to a limit of 100"). Is this something that would be difficult to plumb into the existing aggregator?
—
Sent from Mailbox
On Mon, Oct 27, 2014 at 9:55 PM, akoufoudakis <ak...@gmail.com>
wrote:
> And, of course, time out, set to 0, makes no sense...
> --
> View this message in context: http://camel.465427.n5.nabble.com/Aggregate-up-to-N-ASAP-tp5758134p5758140.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Aggregate up to N "ASAP"
Posted by akoufoudakis <ak...@gmail.com>.
And, of course, time out, set to 0, makes no sense...
--
View this message in context: http://camel.465427.n5.nabble.com/Aggregate-up-to-N-ASAP-tp5758134p5758140.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Aggregate up to N "ASAP"
Posted by Henryk Konsek <he...@gmail.com>.
Hi Baris,
> Is there a way to aggregate the first 100 messages from a seda if >=100 are
> available, but just take however many are available if there are fewer?
As aggregating messages is a process, not a moment in time, you need
to specify some completion timeout duration. Too short timeout doesn't
really make sense, as router won't be able to collect anything in such
sort time.
Cheers.
--
Henryk Konsek
http://henryk-konsek.blogspot.com
Re: Aggregate up to N "ASAP"
Posted by akoufoudakis <ak...@gmail.com>.
Hello, Baris!
I'm afraid, you are asking to find a better solution for 2 x 2 = 4 problem.
Your solution is right. I would do it the same way.
More over, here's an example from book "Camel in Action" by Claus Ibsen:
public void configure() throws Exception {
from("direct:start")
.log("Sending ${body}")
.aggregate(xpath("/order/@customer"), new MyAggregationStrategy())
.completionSize(2).completionTimeout(5000)
.log("Sending out ${body}")
.to("mock:result");
}
So, what you are trying to do is correct.
--
View this message in context: http://camel.465427.n5.nabble.com/Aggregate-up-to-N-ASAP-tp5758134p5758138.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Aggregate up to N "ASAP"
Posted by bacar <ba...@acar.org.uk>.
Thanks. I'm not sure on whether the current design of the aggregator actually
allows for this though? I think the current design is "push" based: an
upstream processor "pushes" exchanges into the aggregator; the aggregator's
completion mechanism then decides whether to submit or not, yet.
I think my end goal would require a "pull" mechanism (analagous to the
receiveNoWait() call from my code sample). Basically I'd need a way to say
"dearest upstream exchange deliverer, do you have anything available that I
could consume at this point in time? If not I'm going to press ahead, but if
you do, I'd like it please".
Perhaps something like this does exist, but I've missed it?
--
View this message in context: http://camel.465427.n5.nabble.com/Aggregate-up-to-N-ASAP-tp5758134p5758311.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Aggregate up to N "ASAP"
Posted by akoufoudakis <ak...@gmail.com>.
Then, you could try to write your own completionPredicate. Some additional
info can be found here: http://camel.apache.org/aggregator2.html.
--
View this message in context: http://camel.465427.n5.nabble.com/Aggregate-up-to-N-ASAP-tp5758134p5758290.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Aggregate up to N "ASAP"
Posted by Henryk Konsek <he...@gmail.com>.
> Henryk, that sounds like a limitation in the design of aggregators in Camel,
> rather than a necessarily desirable end goal.
Your aggregation predicate would make sense only for consumers which
can pull messages in the batch. If your consumer can do it, then you
should create custom aggregation completion predicate that is aware of
the fact that message received from consumer is in fact a batch of the
messages. Then your predicate could decide to complete aggregation if
received batch is big enough or continue to aggregate otherwise.
Cheers.
--
Henryk Konsek
http://henryk-konsek.blogspot.com
Re: Aggregate up to N "ASAP"
Posted by bacar <ba...@acar.org.uk>.
Henryk, that sounds like a limitation in the design of aggregators in Camel,
rather than a necessarily desirable end goal.
--
View this message in context: http://camel.465427.n5.nabble.com/Aggregate-up-to-N-ASAP-tp5758134p5758312.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Aggregate up to N "ASAP"
Posted by Henryk Konsek <he...@gmail.com>.
> Any waiting is redundant here, and only wastes capacity. I'd find
> it surprising if nobody else has a requirement to just process whatever has
> arrived and not wait for any more.
Time frames or message count (or the other predicate) are required to
define "whatever" in "whatever has arrived" :) .
Messages are collected one by one. It indicates that your completion
predicate would complete as soon as aggregator collected one message.
That's why it makes no sense.
Laters!
--
Henryk Konsek
http://henryk-konsek.blogspot.com
Re: Aggregate up to N "ASAP"
Posted by bacar <ba...@acar.org.uk>.
I understand how they work. These proposals don't resolve the problem. I only
want to take /whatever is already available on the SEDA/. I recognize that
this will take longer than 0ms, but the point is that I do not want to wait
for /any more/ messages to arrive once I've grabbed whatever is available on
the SEDA. Any waiting is redundant here, and only wastes capacity. I'd find
it surprising if nobody else has a requirement to just process whatever has
arrived and not wait for any more. Both completionTimeout and
completionInterval will force worst-case scenarios where I have to wait.
To elucidate further: if I were writing it myself against eg JMS (instead of
using camel against a SEDA), I would have something a little like:
--
View this message in context: http://camel.465427.n5.nabble.com/Aggregate-up-to-N-ASAP-tp5758134p5758287.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Aggregate up to N "ASAP"
Posted by boday <be...@initekconsulting.com>.
bacar, as suggested...use a completionTimeout value greater than 0...this
will complete when a max size of 100 is reached OR after no new messages are
received for that amount of time...whichever is FIRST...
also, if you have a max delay requirement, then use the completionInterval()
instead to guarantee completion periodically instead of the
completionTimeout which requires idle time in between messages to complete,
etc...
bacar wrote
> Hi,
>
> Is there a way to aggregate the first 100 messages from a seda if >=100
> are
> available, but just take however many are available if there are fewer?
>
> I think I want aggregate(...).completionSize(100).completionTimeout(0), ie
> complete on 100 items or 0, whichever condition is met "first", but I'm
> not
> convinced that this will actually do the right thing.
>
> Is there a known correct solution for this?
>
> Thanks
> Baris.
-----
Ben O'Day
IT Consultant -http://consulting-notes.com
--
View this message in context: http://camel.465427.n5.nabble.com/Aggregate-up-to-N-ASAP-tp5758134p5758147.html
Sent from the Camel - Users mailing list archive at Nabble.com.