You are viewing a plain text version of this content. The canonical link for it is here.
Posted to fop-users@xmlgraphics.apache.org by Allen Wang <12...@qq.com.INVALID> on 2024/07/18 14:07:50 UTC
回复： Fop multiple thread generate performance
Dear Dr. David and Dr. Simon,
Thanks for your help and good idea.
As our service use the fop as framework. Yes, we did this cache for all the templates. it can save more time for us.
As our service need to generate more than one million reports. some of the report more than 100 pages.
More than 500 templates in our service, and we add all the templates in cache about 1GB xsl file. Use cache can save more time for our online
request.
All of us believe that Fop is an excellent framework. As some times our batch request one time need to generate 100k reports. So, May we ask more good ideas for save our time? the loop pseudo-code as blow. Sure used some thread pool. but each thread still need to generate more reports.
// one batch loop more than 100K records.
requestData.getData().forEach(singleData ->{
String xmlData = convertJson2Xml(singleData);
generateFopFactory(xmlData,templates);
});Above are our business background.
We font when we genereate the files. fop will load all the font files which config the path in xconf each time from desk.if we generate 100k reports. it need to load all the fonts more than 100k times. and all the fonts more about 200MB. it will spend more time. so we change the code. load the fonts files in the Cache. for eaxample we change below as below. we fond the generate the TIFF file can help us save 50% time. but when we generate PDF and PS file. The time saved is not obvious. May I ask if we can do as this? If we do likes this will be happen any bug? Or Could you give me some idea for save PDF and PS time? if has others location load font code need to our save to cache? Or Could you give use more others improve performance methonds. Fop is very fast. but our genrate reports numbers if more big. So, Our boss want to our find more ideas from Dr. David ,Dr. Simon and others fop doctors.
Million thanks for all doctors help.
------------------ 原始邮件 ------------------
发件人: "fop-users" <David.Law@apconsult.de>;
发送时间: 2024年7月17日(星期三) 晚上6:22
收件人: "fop-users"<fop-users@xmlgraphics.apache.org>;
主题: Re: Fop multiple thread generate performance
apart from the FopFactory which Simon mentioned,
I believe you can also do the same for TransformerFactory.
Also, you could consider caching the Stylesheet Templates.
They are Threadsafe & can be used by multiple Threads concurrently.
I'm a bit rusty, but below is some pseudo-code to illustrate some of that.
All the best,
Dave
package fop.example;
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.net.URI;
import java.util.HashMap;
import javax.xml.transform.Templates;
import javax.xml.transform.TransformerConfigurationException;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.stream.StreamSource;
import org.apache.fop.apps.FopFactory;
import org.apache.fop.apps.FopFactoryBuilder;
public class FopPseudoCode {
/*
* TODO a very simple Templates Cache. You might want to add more features like Timestamp checking
*/
private static final HashMap <String, Templates> TEMPLATES_CACHE = new HashMap<>();
private static final ThreadLocal<TransformerFactory> LOCAL_TX_FACTORY = new ThreadLocal<>() {
@Override
protected TransformerFactory initialValue() {
final TransformerFactory factory = TransformerFactory.newInstance();
/*
* TODO Customize your factory here
*/
return factory;
}
};
private static final ThreadLocal<FopFactory> LOCAL_FOP_FACTORY = new ThreadLocal<>() {
@Override
protected FopFactory initialValue() {
final URI defaultBaseURI = new File(".").toURI();
return new FopFactoryBuilder(defaultBaseURI)
/*
* TODO Customize your FopFactoryBuilder here
*/
.build();
}
};
public static void main(final String[] args) throws Exception {
final TransformerFactory txFactory = LOCAL_TX_FACTORY .get();
final FopFactory fopFactory = LOCAL_FOP_FACTORY.get();
final Templates docTemplates = readStylesheetTemplates(txFactory, "Invoice.xslt");
final byte[] docBytes = readDocBytes ( "Invoice.xml");
final byte[] foBytes = generateFO (docBytes, docTemplates);
final byte[] pdfBytes = generatePDFfromFO (txFactory, foBytes, fopFactory);
}
private static final Templates readStylesheetTemplates(final TransformerFactory factory, final String fileName) throws TransformerConfigurationException, IOException {
final Templates templatesCached = TEMPLATES_CACHE.get(fileName);
if (null != templatesCached) {
return templatesCached;
}
try (final InputStream ist = new ByteArrayInputStream(readDocStylesheet(fileName)))
{
final Templates templatesNew = factory.newTemplates(new StreamSource(ist));
TEMPLATES_CACHE.put(fileName, templatesNew);
return templatesNew;
}
}
private static final byte[] readDocStylesheet(final String fileName) {
return "<xsl:stylesheet>...".getBytes();
}
private static final byte[] readDocBytes(final String fileName) {
return "<Invoice>...".getBytes();
}
private static byte[] generateFO(final byte[] xmlDataBytes, final Templates templates) {
return "<fo:root xmlns:fo=...".getBytes();
}
private static byte[] generatePDFfromFO(final TransformerFactory factory, final byte[] bytesFO, final FopFactory fopFactory) {
return "%PDF-1.4...".getBytes();
}
}
On 17/07/2024 08:16, Simon Steiner wrote:
Hi,
You can use multiple threads, each thread with their own fopfactory.
Thanks
From: Allen Wang <1289495104@qq.com.INVALID>
Sent: 14 July 2024 08:39
To: fop-users <fop-users@xmlgraphics.apache.org>
Cc: 1289495104 <1289495104@qq.com>
Subject: Fop multiple thread generate performance
Dear Fop Doctor.
May I ask if we use mutiple thread to generate more output files. if we can have any method to improve the performance.
Kindly, Could you give us some advise?
Thanks a lot.
RE: 回复： Fop multiple thread generate performance
Posted by Mark Gibson <ma...@staff.bluematrix.com>.
Could you architect your system to be multi-service, instead of multi-threaded?
We found that some documents our clients want to render can be very big, and depending on content, could take a lot of memory and resources. We decided not to go with the multi-thread route, as there would be times a single document would take up all the server resources.
We opted for a distributed set of render instances, each one would do only one at a time, and take requests off a work queue. If we need more capacity, we add more render service instances.
Of course, you could architect a multi-service approach where each service was also multi-threaded.
Just an idea.
From: Allen Wang <12...@qq.com.INVALID>
Sent: Thursday, July 18, 2024 3:08 PM
To: fop-users <fo...@xmlgraphics.apache.org>; David.Law <Da...@apconsult.de>
Cc: fop-users-sc.1669167783.coibeaaimjmplbfojhfk-1289495104=qq.com <fo...@xmlgraphics.apache.org>
Subject: 回复： Fop multiple thread generate performance
[EXTERNAL]
Dear Dr. David and Dr. Simon,
Thanks for your help and good idea.
As our service use the fop as framework. Yes, we did this cache for all the templates. it can save more time for us.
As our service need to generate more than one million reports. some of the report more than 100 pages.
More than 500 templates in our service, and we add all the templates in cache about 1GB xsl file. Use cache can save more time for our online
request.
All of us believe that Fop is an excellent framework. As some times our batch request one time need to generate 100k reports. So, May we ask more good ideas for save our time? the loop pseudo-code as blow. Sure used some thread pool. but each thread still need to generate more reports.
// one batch loop more than 100K records.
requestData.getData().forEach(singleData ->{
String xmlData = convertJson2Xml(singleData);
generateFopFactory(xmlData,templates);
});
Above are our business background.
We font when we genereate the files. fop will load all the font files which config the path in xconf each time from desk.if we generate 100k reports. it need to load all the fonts more than 100k times. and all the fonts more about 200MB. it will spend more time. so we change the code. load the fonts files in the Cache.
for eaxample we change below as below. we fond the generate the TIFF file can help us save 50% time. but when we generate PDF and PS file. The time saved is not obvious.
May I ask if we can do as this? If we do likes this will be happen any bug?
Or Could you give me some idea for save PDF and PS time? if has others location load font code need to our save to cache? Or Could you give use more others improve performance methonds. Fop is very fast. but our genrate reports numbers if more big. So, Our boss want to our find more ideas from Dr. David ,Dr. Simon and others fop doctors.
[cid:image001.png@01DADC34.2299B600]
Million thanks for all doctors help.
------------------ 原始邮件 ------------------
发件人: "fop-users" <Da...@apconsult.de>>;
发送时间: 2024年7月17日(星期三) 晚上6:22
收件人: "fop-users"<fo...@xmlgraphics.apache.org>>;
主题: Re: Fop multiple thread generate performance
Hi,
apart from the FopFactory which Simon mentioned,
I believe you can also do the same for TransformerFactory.
Also, you could consider caching the Stylesheet Templates.
They are Threadsafe & can be used by multiple Threads concurrently.
I'm a bit rusty, but below is some pseudo-code to illustrate some of that.
All the best,
Dave
package fop.example;
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.net.URI;
import java.util.HashMap;
import javax.xml.transform.Templates;
import javax.xml.transform.TransformerConfigurationException;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.stream.StreamSource;
import org.apache.fop.apps.FopFactory;
import org.apache.fop.apps.FopFactoryBuilder;
public
class FopPseudoCode {
/*
* TODO a very simple Templates Cache. You might want to add more features like Timestamp checking
*/
private
static
final HashMap <String, Templates>
TEMPLATES_CACHE =
new HashMap<>();
private
static
final ThreadLocal<TransformerFactory>
LOCAL_TX_FACTORY =
new ThreadLocal<>() {
@Override
protected TransformerFactory initialValue() {
final TransformerFactory
factory = TransformerFactory.newInstance();
/*
* TODO Customize your factory here
*/
return
factory;
}
};
private
static
final ThreadLocal<FopFactory>
LOCAL_FOP_FACTORY =
new ThreadLocal<>() {
@Override
protected FopFactory initialValue() {
final URI
defaultBaseURI =
new File(".").toURI();
return
new FopFactoryBuilder(defaultBaseURI)
/*
* TODO Customize your FopFactoryBuilder here
*/
.build();
}
};
public
static
void main(final String[]
args)
throws Exception {
final TransformerFactory
txFactory =
LOCAL_TX_FACTORY .get();
final FopFactory
fopFactory =
LOCAL_FOP_FACTORY.get();
final Templates
docTemplates =
readStylesheetTemplates(txFactory,
"Invoice.xslt");
final
byte[]
docBytes =
readDocBytes (
"Invoice.xml");
final
byte[]
foBytes =
generateFO (docBytes,
docTemplates);
final
byte[]
pdfBytes =
generatePDFfromFO (txFactory,
foBytes,
fopFactory);
}
private
static
final Templates readStylesheetTemplates(final TransformerFactory
factory,
final String
fileName)
throws TransformerConfigurationException, IOException {
final Templates
templatesCached =
TEMPLATES_CACHE.get(fileName);
if (null !=
templatesCached) {
return
templatesCached;
}
try (final InputStream
ist =
new ByteArrayInputStream(readDocStylesheet(fileName)))
{
final Templates
templatesNew =
factory.newTemplates(new StreamSource(ist));
TEMPLATES_CACHE.put(fileName,
templatesNew);
return
templatesNew;
}
}
private
static
final
byte[] readDocStylesheet(final String
fileName) {
return
"<xsl:stylesheet>...".getBytes();
}
private
static
final
byte[] readDocBytes(final String
fileName) {
return
"<Invoice>...".getBytes();
}
private
static
byte[] generateFO(final
byte[]
xmlDataBytes,
final Templates
templates) {
return
"<fo:root xmlns:fo=...".getBytes();
}
private
static
byte[] generatePDFfromFO(final TransformerFactory
factory,
final
byte[]
bytesFO,
final FopFactory
fopFactory) {
return
"%PDF-1.4...".getBytes();
}
}
On 17/07/2024 08:16, Simon Steiner wrote:
Hi,
You can use multiple threads, each thread with their own fopfactory.
Thanks
From: Allen Wang <12...@qq.com.INVALID>
Sent: 14 July 2024 08:39
To: fop-users <fo...@xmlgraphics.apache.org>
Cc: 1289495104 <12...@qq.com>
Subject: Fop multiple thread generate performance
Dear Fop Doctor.
May I ask if we use mutiple thread to generate more output files. if we can have any method to improve the performance.
Kindly, Could you give us some advise?
Thanks a lot.