You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pdfbox.apache.org by "Luis Filipe Nassif (JIRA)" <ji...@apache.org> on 2016/07/21 21:25:20 UTC
[jira] [Comment Edited] (PDFBOX-3429) Improve ExtractText
Concurrency
[ https://issues.apache.org/jira/browse/PDFBOX-3429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388439#comment-15388439 ]
Luis Filipe Nassif edited comment on PDFBOX-3429 at 7/21/16 9:24 PM:
---------------------------------------------------------------------
Hi Tilman,
I was going to write a test code with only PDFBox, without Tika, thank you very much. I've got same results with your code: ~100% cpu usage with v1.8.10 and ~80% with 2.0.2, with 12 threads (actually my system has 6 physical cores x 2 hiperthreading). I will attach the graphs. I modified a little bit your code to use only one specific file (will attach it too), parsed 10000 times, but it was chosen randomly, so I think there is nothing special with the file. Modified code below:
{code}
public class BenchmarkPDFBox3429
{
public static final int JOB_COUNT = 10000;
public static final int THREAD_COUNT = 12;
public static void main(String[] args) throws IOException, InterruptedException
{
java.util.logging.Logger.getLogger("org.apache").setLevel(java.util.logging.Level.OFF);
System.out.println("Benchmark is running");
System.out.printf("%s pdf strips with %s thread(s).", JOB_COUNT, THREAD_COUNT);
long startTime = System.currentTimeMillis();
ExecutorService executor = Executors.newFixedThreadPool(THREAD_COUNT);
File file = new File("f:/teste/pdf2/000000000000B265.pdf");
for (int i = 0; i < JOB_COUNT; i++)
{
executor.execute(new Job(file));
}
executor.shutdown();
while (!executor.isTerminated())
{
Thread.sleep(1000);
}
long duration = System.currentTimeMillis() - startTime;
System.out.println(String.format("\nFinished all jobs. %s ms", duration));
}
static class Job implements Runnable
{
private final File file;
Job(File file)
{
this.file = file;
}
@Override
public void run()
{
try
{
extractFromPdf(file);
}
catch (IOException e)
{
e.printStackTrace();
}
finally
{
}
}
public static String extractFromPdf(File file) throws IOException
{
PDDocument doc = null;
try
{
doc = PDDocument.load(file);
PDFTextStripper stripper = new PDFTextStripper();
return stripper.getText(doc);
}finally{
if(doc != null)
doc.close();
}
}
}
}
{code}
Also, I repeated the test with only 4 threads (1/3 of 12) and cpu usage was around 33% (1/3 of 100%) as expected, so maybe the problem arises only in high concurrent scenarios.
Finally, I repeated the test with 48 threads and got same results: cpu usage only around 80%.
was (Author: lfcnassif):
Hi Tilman,
I was going to write a test code with only PDFBox, without Tika, thank you very much. I've got same results with your code: ~100% cpu usage with v1.8.10 and ~80% with 2.0.2, with 12 threads (actually my system has 6 physical cores x 2 hiperthreading). I will attach the graphs. I modified a little bit your code to used only one specific file (will attach it too), parsed 10000, but it was chosen ramdonly, so I think there is nothing special with the file. Modified code below:
{code}
public class BenchmarkPDFBox3429
{
public static final int JOB_COUNT = 10000;
public static final int THREAD_COUNT = 12;
public static void main(String[] args) throws IOException, InterruptedException
{
java.util.logging.Logger.getLogger("org.apache").setLevel(java.util.logging.Level.OFF);
System.out.println("Benchmark is running");
System.out.printf("%s pdf strips with %s thread(s).", JOB_COUNT, THREAD_COUNT);
long startTime = System.currentTimeMillis();
ExecutorService executor = Executors.newFixedThreadPool(THREAD_COUNT);
File file = new File("f:/teste/pdf2/000000000000B265.pdf");
for (int i = 0; i < JOB_COUNT; i++)
{
executor.execute(new Job(file));
}
executor.shutdown();
while (!executor.isTerminated())
{
Thread.sleep(1000);
}
long duration = System.currentTimeMillis() - startTime;
System.out.println(String.format("\nFinished all jobs. %s ms", duration));
}
static class Job implements Runnable
{
private final File file;
Job(File file)
{
this.file = file;
}
@Override
public void run()
{
try
{
extractFromPdf(file);
}
catch (IOException e)
{
e.printStackTrace();
}
finally
{
}
}
public static String extractFromPdf(File file) throws IOException
{
PDDocument doc = null;
try
{
doc = PDDocument.load(file);
PDFTextStripper stripper = new PDFTextStripper();
return stripper.getText(doc);
}finally{
if(doc != null)
doc.close();
}
}
}
}
{code}
Also, I repeated the test with only 4 threads (1/3 of 12) and cpu usage was around 33% (1/3 of 100%) as expected, so maybe the problem arises only in high concurrent scenarios.
Finally, I repeated the test with 48 threads and got same results: cpu usage only around 80%.
> Improve ExtractText Concurrency
> -------------------------------
>
> Key: PDFBOX-3429
> URL: https://issues.apache.org/jira/browse/PDFBOX-3429
> Project: PDFBox
> Issue Type: Improvement
> Components: Text extraction
> Affects Versions: 2.0.1
> Environment: Win7, jdk1.8.0_60 x64
> Reporter: Luis Filipe Nassif
> Priority: Minor
> Labels: optimization
> Attachments: 000000000000B265.pdf, cpu-pdfbox-2.0.2.png, cpu-pdfbox1.8.10.png
>
>
> While testing Tika 1.13, which uses PDFBox 2.0.1, from a multithreaded text extraction application, I noted cpu usage aroung 80% in my 6 core computer when processing a dataset of ~75 thousands of pdfs (18GB). It took 5min25sec to complete the text extraction. With Tika 1.10, which uses PDFBox 1.8.10, cpu usage stays aroung 100%. It took 4min37sec to complete. The dataset is read from a ramdrive, so there is no i/o bottleneck. I suspect there is some new synchronization code that blocks the threads for a non trivial amount of time, resulting in less cpu usage than before.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pdfbox.apache.org
For additional commands, e-mail: dev-help@pdfbox.apache.org