Processing large PDF files using apache camel 2.10

[ravi21588 <ra...@gmail.com>]

Hi all,
I have an integration scenario  to generate PDF from a document generation
system.
The document generation system generates byte [] data.
The data can be huge enough,I need to transport the Byte array to jms queue.
I have two components in camel for this integration which is connected using
vm component.
Iam using the attachment part of the exchange to put the byte array content.
I hope the attachment will be stored in memory during transfer between these
components.
The load is huge there a possibility of memory issue.
Can you please let me know if there is better approach using camel.



--
View this message in context: http://camel.465427.n5.nabble.com/Processing-large-PDF-files-using-apache-camel-2-10-tp5791476.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Processing large PDF files using apache camel 2.10

[ravi21588 <ra...@gmail.com>]

Hi Claus,
Thanks for the reply.
Do you mean to place the pdf in a file location and use the stream component
http://camel.apache.org/stream.html to get data as stream?

Also can you please tell me how the data transfer happens using stream
component?

Using claim check EIP also at some point stores the message in to file store
or database  from exchange body as below example.

public static final class CheckLuggageBean {        
    public void checkLuggage(Exchange exchange, @Body String body,
@XPath("/order/@custId") String custId) {   
        // store the message body into the data store, using the custId as
the claim check
        *dataStore.put(custId, body);*
        // add the claim check as a header
        exchange.getIn().setHeader("claimCheck", custId);
        // remove the body from the message
        exchange.getIn().setBody(null);
    }
} 


and retrieve it at later point of time from the store and put it back in
exchange body as below.

public static final class DataEnricherBean {
    public void addDataBackIn(Exchange exchange, @Header("claimCheck")
String claimCheck) { 
        // query the data store using the claim check as the key and add the
data
        // back into the message body
        *exchange.getIn().setBody(dataStore.get(claimCheck));*
        // remove the message data from the data store
        dataStore.remove(claimCheck);
        // remove the claim check header
        exchange.getIn().removeHeader("claimCheck");
    }
}   

I agree that instead of passing whole message in body in all the components
it is an better approach ,but even wont there be an memory issue?





--
View this message in context: http://camel.465427.n5.nabble.com/Processing-large-PDF-files-using-apache-camel-2-10-tp5791476p5791561.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Processing large PDF files using apache camel 2.10

[Claus Ibsen <cl...@gmail.com>]

Its likely better to store the PDF files on the file system, and read
them in streaming if you need to process them.

See this EIP
http://camel.apache.org/claim-check.html

If you store the pdf files as byte[] then that takes up direct memory.

It all depends on the memory available to you, the sizes of the PDFs,
and how many you process concurrently. There is no single solution for
every variation.

But storing the PDF on the file system allows you to have them
persisted as well, and not lose them if the JVM crashes.


On Wed, Dec 14, 2016 at 10:10 PM, ravi21588 <ra...@gmail.com> wrote:
> Hi all,
> I have an integration scenario  to generate PDF from a document generation
> system.
> The document generation system generates byte [] data.
> The data can be huge enough,I need to transport the Byte array to jms queue.
> I have two components in camel for this integration which is connected using
> vm component.
> Iam using the attachment part of the exchange to put the byte array content.
> I hope the attachment will be stored in memory during transfer between these
> components.
> The load is huge there a possibility of memory issue.
> Can you please let me know if there is better approach using camel.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Processing-large-PDF-files-using-apache-camel-2-10-tp5791476.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



-- 
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2