Large message handling

[James Green <ja...@gmail.com>]

Hi,

We are preparing to start handling very large messages. Currently we have
small (<1k) messages arriving from message queues and being processed, but
these new messages could be several mb each.

Each message is basically a JSON formatted payload with metadata and a
content-body. It's this content-body that may end up being much larger in
some cases.

We have speculated that this may be best handled by recognising a reference
within the message meta-data to a URI that actually has the content-body.
This feels like a pattern that may have already been invented. Not unlike
SOAP where references to files sent separately can be achieved.

Anyone know of such a system or even terms to hit Google with? And might
there be something in the Camel-sphere that loads external references
already?

Thanks,

James

Re: Large message handling

[souciance <so...@gmail.com>]

I can't speak about the approach but I have used camel to parse and
generate thousands of mb size Json messages and performance was more than
satisfactory.

Den 4 nov. 2016 3:54 em skrev "James Green [via Camel]" <
ml-node+s465427n5789754h37@n5.nabble.com>:
>
> Hi,
>
> We are preparing to start handling very large messages. Currently we have
> small (<1k) messages arriving from message queues and being processed,
but
> these new messages could be several mb each.
>
> Each message is basically a JSON formatted payload with metadata and a
> content-body. It's this content-body that may end up being much larger in
> some cases.
>
> We have speculated that this may be best handled by recognising a
reference
> within the message meta-data to a URI that actually has the content-body.
> This feels like a pattern that may have already been invented. Not unlike
> SOAP where references to files sent separately can be achieved.
>
> Anyone know of such a system or even terms to hit Google with? And might
> there be something in the Camel-sphere that loads external references
> already?
>
> Thanks,
>
> James
>
>
> ________________________________
> If you reply to this email, your message will be added to the discussion
below:
> http://camel.465427.n5.nabble.com/Large-message-handling-tp5789754.html
> To start a new topic under Camel - Users, email
ml-node+s465427n465428h31@n5.nabble.com
> To unsubscribe from Camel - Users, click here.
> NAML




--
View this message in context: http://camel.465427.n5.nabble.com/Large-message-handling-tp5789754p5789765.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Large message handling

[James Green <ja...@gmail.com>]

That's pretty much what we were describing.

I see there is no DSL for this so beyond it being a known algorithm it
remains a "write-your-own-implementation". I guess therefore it's a
relatively narrow use-case...


On 4 November 2016 at 16:50, Quinn Stevenson <qu...@pronoia-solutions.com>
wrote:

> I think the EIP you’re referring to is the “Claim Check” pattern (
> http://www.enterpriseintegrationpatterns.com/patterns/messaging/
> StoreInLibrary.html <http://www.enterpriseintegrationpatterns.
> com/patterns/messaging/StoreInLibrary.html> ), and Camel does support
> that EIP ( http://camel.apache.org/claim-check.html <
> http://camel.apache.org/claim-check.html> )
>
> HTH
>
>
> > On Nov 4, 2016, at 8:53 AM, James Green <ja...@gmail.com>
> wrote:
> >
> > Hi,
> >
> > We are preparing to start handling very large messages. Currently we have
> > small (<1k) messages arriving from message queues and being processed,
> but
> > these new messages could be several mb each.
> >
> > Each message is basically a JSON formatted payload with metadata and a
> > content-body. It's this content-body that may end up being much larger in
> > some cases.
> >
> > We have speculated that this may be best handled by recognising a
> reference
> > within the message meta-data to a URI that actually has the content-body.
> > This feels like a pattern that may have already been invented. Not unlike
> > SOAP where references to files sent separately can be achieved.
> >
> > Anyone know of such a system or even terms to hit Google with? And might
> > there be something in the Camel-sphere that loads external references
> > already?
> >
> > Thanks,
> >
> > James
>
>

Re: Large message handling

[Quinn Stevenson <qu...@pronoia-solutions.com>]

I think the EIP you’re referring to is the “Claim Check” pattern ( http://www.enterpriseintegrationpatterns.com/patterns/messaging/StoreInLibrary.html <http://www.enterpriseintegrationpatterns.com/patterns/messaging/StoreInLibrary.html> ), and Camel does support that EIP ( http://camel.apache.org/claim-check.html <http://camel.apache.org/claim-check.html> )

HTH


> On Nov 4, 2016, at 8:53 AM, James Green <ja...@gmail.com> wrote:
> 
> Hi,
> 
> We are preparing to start handling very large messages. Currently we have
> small (<1k) messages arriving from message queues and being processed, but
> these new messages could be several mb each.
> 
> Each message is basically a JSON formatted payload with metadata and a
> content-body. It's this content-body that may end up being much larger in
> some cases.
> 
> We have speculated that this may be best handled by recognising a reference
> within the message meta-data to a URI that actually has the content-body.
> This feels like a pattern that may have already been invented. Not unlike
> SOAP where references to files sent separately can be achieved.
> 
> Anyone know of such a system or even terms to hit Google with? And might
> there be something in the Camel-sphere that loads external references
> already?
> 
> Thanks,
> 
> James