You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pdfbox.apache.org by Kevin Day <ke...@trumpetinc.com> on 2024/12/13 02:29:26 UTC
Text extraction adding lots of strange spaces
Hello-
We are using PDFTextStripper, and have found some cases where there are a
*lot* of extraneous spaces being added to the output. It almost acts like
the stripper is thinking that the space width of the font is super tiny.
I managed to get a document that exhibits the behavior:
https://drive.google.com/file/d/1B2Mc4mMdsYfk9jKVqQ9OxEhKLRAxprrU/view?usp=sharing
The easiest way to see the behavior is in PDFDebugger, View->Show Stripper
Text Positions.
Note in the lower left corner of the document, there is text "999". The
text above and below that is fine, but the line with 999 has a *ton* of
extra space rectangles displated.
The extract text function in PDFDebugger doesn't sort, so that one comes
out fine, but if you use PDFTextStripper with sorting enabled (), the line
renders like this:
Withdrawals and distributions . . . $ ( 9 9 9 )
Note the many space characters, and that there are even spaces between each
9.
I also observe that the PDF has warning messages about fonts (not sure if
this might be involved):
[main] WARN org.apache.pdfbox.pdmodel.font.PDType1Font - Using fallback
font ArialMT for HelveticaLTStd-Roman
[main] WARN org.apache.fontbox.ttf.CmapSubtable - Format 14 cmap table is
not supported and will be ignored
It almost acts like the parenthesis on the line are triggering some
different detection mode where the font's space width is computing to be
much smaller than it should be.
Any ideas on what is going on or if it is fixable?
Thanks!
- K
Re: Text extraction adding lots of strange spaces
Posted by Kevin Day <ke...@trumpetinc.com>.
FYI - what I'm proposing is a slightly more refined approach than what is
suggested in the linked issue:
https://issues.apache.org/jira/browse/PDFBOX-3774
Here's how I would do this: In writePage(), keep track of the previous
character added to the line (around line 696). Then at the top of the loop
(around line 551) check if the current character is a space. If it is a
space, check to see if it geometrically completely overlaps the previous
character (probably checking only in X direction - and it will need to
account for directional adjustments ). If it does, then continue with the
next TextPosition.
Please let me know your thoughts.
- K
Kevin Day
*trumpet**p| *480.961.6003 x1002
*e| *kevin@trumpetinc.com
*www.trumpetinc.com <http://trumpetinc.com/> | *LinkedIn
<https://www.linkedin.com/company/trumpet-inc.>
On Fri, Dec 13, 2024 at 7:41 AM Kevin Day <ke...@trumpetinc.com> wrote:
> Ok, this is very interesting.
>
> So we have one text render that adds a bunch of spaces.
>
> Then we have another text render that puts visible characters on top of
> those spaces.
>
> Then we positional-sort the text positions, and they wind up interleaving,
> which kills the text extraction accuracy.
>
> Would a reasonable refinement to the algorithm be to look for overlapping
> glyphs? And if there is a space that overlaps a non-space, then ignore the
> space?
>
> I know that text extraction is a black art (disclosure: I wrote the
> original text extraction modules for iText, so lots of hard-earned
> experience here), but I think the above would be appropriate for all
> situations...
>
> Would you be up for reviewing a patch if I implement this?
>
> PS I also think that having an option to ignore spaces in rendering (the
> issue you linked to) would be a good idea. But that should be optional. I'm
> happy to include that in my patch if you would like.
>
> K
>
> On Thu, Dec 12, 2024, 9:40 PM Tilman Hausherr <TH...@t-online.de>
> wrote:
>
>> Hi,
>>
>> These spaces are really in the PDF:
>>
>> BT
>> /Content <</MCID 224 >>BDC
>> 1 i
>> /T1_4 1 Tf
>> 7 0 0 7 *195.4 110.502* Tm
>> *(\( \))Tj*
>> EMC
>> /Content <</MCID 225 >>BDC
>> /T1_0 1 Tf
>> -19.686 6.786 Td
>> (Beginning capital account)Tj
>> /T1_4 1 Tf
>> ( )Tj
>> 14.057 0 Td
>> (.)Tj
>> 1.714 0 Td
>> (.)Tj
>> 1.714 0 Td
>> (.)Tj
>> EMC
>> ET
>>
>>
>> And *195.4 110.502* is really the position, you can move your mouse
>> there in PDFDebugger.
>>
>> The font messages are not important here.
>> There is a way to get rid of such spaces, but it requires a source code
>> change, it is described here:
>> https://issues.apache.org/jira/browse/PDFBOX-3774
>>
>> However it's possible that other files would have a bad text extraction.
>>
>> Tilman
>>
>> On 13.12.2024 03:29, Kevin Day wrote:
>> > Hello-
>> >
>> > We are using PDFTextStripper, and have found some cases where there are
>> a
>> > *lot* of extraneous spaces being added to the output. It almost acts
>> like
>> > the stripper is thinking that the space width of the font is super tiny.
>> >
>> > I managed to get a document that exhibits the behavior:
>> >
>> >
>> https://drive.google.com/file/d/1B2Mc4mMdsYfk9jKVqQ9OxEhKLRAxprrU/view?usp=sharing
>> >
>> > The easiest way to see the behavior is in PDFDebugger, View->Show
>> Stripper
>> > Text Positions.
>> >
>> > Note in the lower left corner of the document, there is text "999". The
>> > text above and below that is fine, but the line with 999 has a *ton* of
>> > extra space rectangles displated.
>> >
>> > The extract text function in PDFDebugger doesn't sort, so that one comes
>> > out fine, but if you use PDFTextStripper with sorting enabled (), the
>> line
>> > renders like this:
>> >
>> > Withdrawals and distributions . . . $ ( 9 9 9 )
>> >
>> > Note the many space characters, and that there are even spaces between
>> each
>> > 9.
>> >
>> > I also observe that the PDF has warning messages about fonts (not sure
>> if
>> > this might be involved):
>> >
>> > [main] WARN org.apache.pdfbox.pdmodel.font.PDType1Font - Using fallback
>> > font ArialMT for HelveticaLTStd-Roman
>> >
>> > [main] WARN org.apache.fontbox.ttf.CmapSubtable - Format 14 cmap table
>> is
>> > not supported and will be ignored
>> >
>> >
>> >
>> > It almost acts like the parenthesis on the line are triggering some
>> > different detection mode where the font's space width is computing to be
>> > much smaller than it should be.
>> >
>> > Any ideas on what is going on or if it is fixable?
>> >
>> > Thanks!
>> >
>> > - K
>> >
>>
>
Re: Text extraction adding lots of strange spaces
Posted by Kevin Day <ke...@trumpetinc.com>.
Thanks, Tilman.
"This happens twice during build tests, of over 100 text extractions."
Thanks for explaining... From the code comment, I thought this was a
general behavior of JDK7+ sort, but it sounds like it is only a problem in
a rare edge case of specific compares. How exciting.
Cheers,
K
On Tue, Dec 17, 2024, 2:22 AM Tilman Hausherr <TH...@t-online.de> wrote:
> Hi,
>
> Thank you, the change has been committed.
> re 1: we'll see what happens... re "but it is code that needs to be
> maintained" - that is a general problem. Sometimes it's even difficult
> to maintain ones own code.
> re 2: No because most of the time, the faster built-in sort works fine.
> The slower mergesort is only used when the exception is thrown. This
> happens twice during build tests, of over 100 text extractions.
>
> Tilman
>
> On 16.12.2024 15:55, Kevin Day wrote:
> > I am attaching the patch file.
> >
> > And yes, this patch is simply PDFBOX-3774 as an option, a small
> > cosmetic change to use idiomatic Java for PDFBOX-5487, and a unit test
> > that demonstrates the overlapping.
> >
> >
> > A couple of additional thoughts:
> >
> > 1. I feel that PDFBOX-5487 isn't doing very much. The PDFBOX-3774
> > feature will address the problem fixed by PDFBOX-5487, and the
> > "problem" of having a space glyph entirely within the previous
> > character is a very restricted edge-case. In the end, the performance
> > hit is not a big deal, but it is code that needs to be maintained. I
> > thought I'd mention it in case the PDFBOX-5487 requester would be
> > happy with PDFBOX-3774 as a solution.
> >
> > 2. I noticed that there is a note about JDK7+ sorting
> > requiring transitive comparators. Given that the build requires
> > JDK8+, I wonder if it is time to remove the Collections.sort path (and
> > get rid of an exception throw, etc...)?
> >
> > - K
> >
> >
> >
> > On Mon, Dec 16, 2024 at 6:21 AM Tilman Hausherr
> > <TH...@t-online.de> wrote:
> >
> > On 16.12.2024 14:02, Kevin Day wrote:
> > > I just realized that there is an incorrect note in the
> getter/setter
> > > Javadocs about the setting only taking effect if sorting is
> enabled.
> > >
> > > That note can be removed. The new setting is valid regardless of
> > whether
> > > sorting is enabled.
> >
> > Hi,
> >
> > Could you please resend the patch as text attachment? Somehow the
> > mail
> > program messed this up.
> >
> > From what I understand, the patch is the suggestion from
> > PDFBOX-3774but
> > as an option, plus a test. The other change (re PDFBOX-5487) is a
> > (useful) cosmetic change. I wonder why I missed that when I
> > committed it.
> >
> > Tilman
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail:users-unsubscribe@pdfbox.apache.org
> > For additional commands, e-mail:users-help@pdfbox.apache.org
>
>
Re: Text extraction adding lots of strange spaces
Posted by Tilman Hausherr <TH...@t-online.de>.
Hi,
Thank you, the change has been committed.
re 1: we'll see what happens... re "but it is code that needs to be
maintained" - that is a general problem. Sometimes it's even difficult
to maintain ones own code.
re 2: No because most of the time, the faster built-in sort works fine.
The slower mergesort is only used when the exception is thrown. This
happens twice during build tests, of over 100 text extractions.
Tilman
On 16.12.2024 15:55, Kevin Day wrote:
> I am attaching the patch file.
>
> And yes, this patch is simply PDFBOX-3774 as an option, a small
> cosmetic change to use idiomatic Java for PDFBOX-5487, and a unit test
> that demonstrates the overlapping.
>
>
> A couple of additional thoughts:
>
> 1. I feel that PDFBOX-5487 isn't doing very much. The PDFBOX-3774
> feature will address the problem fixed by PDFBOX-5487, and the
> "problem" of having a space glyph entirely within the previous
> character is a very restricted edge-case. In the end, the performance
> hit is not a big deal, but it is code that needs to be maintained. I
> thought I'd mention it in case the PDFBOX-5487 requester would be
> happy with PDFBOX-3774 as a solution.
>
> 2. I noticed that there is a note about JDK7+ sorting
> requiring transitive comparators. Given that the build requires
> JDK8+, I wonder if it is time to remove the Collections.sort path (and
> get rid of an exception throw, etc...)?
>
> - K
>
>
>
> On Mon, Dec 16, 2024 at 6:21 AM Tilman Hausherr
> <TH...@t-online.de> wrote:
>
> On 16.12.2024 14:02, Kevin Day wrote:
> > I just realized that there is an incorrect note in the getter/setter
> > Javadocs about the setting only taking effect if sorting is enabled.
> >
> > That note can be removed. The new setting is valid regardless of
> whether
> > sorting is enabled.
>
> Hi,
>
> Could you please resend the patch as text attachment? Somehow the
> mail
> program messed this up.
>
> From what I understand, the patch is the suggestion from
> PDFBOX-3774but
> as an option, plus a test. The other change (re PDFBOX-5487) is a
> (useful) cosmetic change. I wonder why I missed that when I
> committed it.
>
> Tilman
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:users-unsubscribe@pdfbox.apache.org
> For additional commands, e-mail:users-help@pdfbox.apache.org
Re: Text extraction adding lots of strange spaces
Posted by Kevin Day <ke...@trumpetinc.com>.
I am attaching the patch file.
And yes, this patch is simply PDFBOX-3774 as an option, a small cosmetic
change to use idiomatic Java for PDFBOX-5487, and a unit test that
demonstrates the overlapping.
A couple of additional thoughts:
1. I feel that PDFBOX-5487 isn't doing very much. The PDFBOX-3774 feature
will address the problem fixed by PDFBOX-5487, and the "problem" of having
a space glyph entirely within the previous character is a very restricted
edge-case. In the end, the performance hit is not a big deal, but it is
code that needs to be maintained. I thought I'd mention it in case the
PDFBOX-5487 requester would be happy with PDFBOX-3774 as a solution.
2. I noticed that there is a note about JDK7+ sorting requiring transitive
comparators. Given that the build requires JDK8+, I wonder if it is time
to remove the Collections.sort path (and get rid of an exception throw,
etc...)?
- K
On Mon, Dec 16, 2024 at 6:21 AM Tilman Hausherr <TH...@t-online.de>
wrote:
> On 16.12.2024 14:02, Kevin Day wrote:
> > I just realized that there is an incorrect note in the getter/setter
> > Javadocs about the setting only taking effect if sorting is enabled.
> >
> > That note can be removed. The new setting is valid regardless of whether
> > sorting is enabled.
>
> Hi,
>
> Could you please resend the patch as text attachment? Somehow the mail
> program messed this up.
>
> From what I understand, the patch is the suggestion from PDFBOX-3774but
> as an option, plus a test. The other change (re PDFBOX-5487) is a
> (useful) cosmetic change. I wonder why I missed that when I committed it.
>
> Tilman
>
Re: Text extraction adding lots of strange spaces
Posted by Tilman Hausherr <TH...@t-online.de>.
On 16.12.2024 14:02, Kevin Day wrote:
> I just realized that there is an incorrect note in the getter/setter
> Javadocs about the setting only taking effect if sorting is enabled.
>
> That note can be removed. The new setting is valid regardless of whether
> sorting is enabled.
Hi,
Could you please resend the patch as text attachment? Somehow the mail
program messed this up.
From what I understand, the patch is the suggestion from PDFBOX-3774but
as an option, plus a test. The other change (re PDFBOX-5487) is a
(useful) cosmetic change. I wonder why I missed that when I committed it.
Tilman
Re: Text extraction adding lots of strange spaces
Posted by Kevin Day <ke...@trumpetinc.com>.
I just realized that there is an incorrect note in the getter/setter
Javadocs about the setting only taking effect if sorting is enabled.
That note can be removed. The new setting is valid regardless of whether
sorting is enabled.
K
On Sun, Dec 15, 2024, 10:14 PM Kevin Day <ke...@trumpetinc.com> wrote:
> ok - I've done a thorough evaluation of what would be required to properly
> suppress overlapping spaces (but leave in the non-overlapping spaces), and
> a lot of pathfinding.
>
> What is really needed is to compare two non-space positions and see if
> there *should* be a space between them, and if not, then suppress any
> spaces that appear between them. I know how to do this, but it would
> require a good bit of surgery. Basically, we would need to change to
> accumulating text positions by line (without processing them). Then once a
> line is accumulated, run through the line and insert word separators (and
> remove any overlapping spaces). I would probably do this by moving the
> word separation logic into the normalize(line) method. But that is a lot
> of surgery.
>
> My concern is that determining if two non-space glyphs are actually
> adjacent requires making a determination based on the width of a space
> glyph - and that introduces the risk of suppressing an overlapping space
> when it shouldn't. I'm not comfortable with that sort of regression -
> especially for something that is not adding a ton of value beyond what we
> can achieve by just removing Tj spaces entirely.
>
> If someone really cares about preserving the physical layout of the page,
> I believe it would be better to put time into making normalize() add space
> characters (or tabs maybe) based on the physical layout of the non-space
> characters, instead of attempting to preserve any spaces that are part of
> the content stream operations.
>
> So for now, I am going to suggest that we implement the setting to ignore
> space glyphs in the content stream (as suggested in
> https://issues.apache.org/jira/browse/PDFBOX-3774 ). I'm including a
> patch that does this below (one source change, plus a self-contained unit
> test). I wish this patch was more valuable! But if it helps anything,
> this change has a dramatic effect on the robustness of our application, so
> there is a ton of value to us to having PDFBox add this setting in 3.0.4.
>
> I'm happy to discuss the above further - I'm just not seeing a huge amount
> of value compared to the blanket removal of spaces...
>
>
> Index: pdfbox/src/main/java/org/apache/pdfbox/text/PDFTextStripper.java
> ===================================================================
> --- pdfbox/src/main/java/org/apache/pdfbox/text/PDFTextStripper.java
> (revision 1922522)
> +++ pdfbox/src/main/java/org/apache/pdfbox/text/PDFTextStripper.java
> (working copy)
> @@ -40,6 +40,7 @@
> import java.util.TreeMap;
> import java.util.TreeSet;
> import java.util.regex.Pattern;
> +import java.util.stream.Collectors;
>
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;
> @@ -147,6 +148,7 @@
> private boolean shouldSeparateByBeads = true;
> private boolean sortByPosition = false;
> private boolean addMoreFormatting = false;
> + private boolean ignoreContentStreamSpaceGlyphs = false;
>
> private float indentThreshold = defaultIndentThreshold;
> private float dropThreshold = defaultDropThreshold;
> @@ -524,11 +526,10 @@
> {
> IterativeMergeSort.sort(textList, comparator);
> }
> - finally
> - {
> - // PDFBOX-5487: Remove all space characters if
> contained within the adjacent letters
> - removeContainedSpaces(textList);
> - }
> +
> + // PDFBOX-5487: Remove all space characters if contained
> within the adjacent letters
> + removeContainedSpaces(textList);
> +
> }
>
> startArticle();
> @@ -556,6 +557,10 @@
> PositionWrapper current = new PositionWrapper(position);
> String characterValue = position.getUnicode();
>
> + // PDFBOX-3774 - conditionally ignore spaces from the
> content stream
> + if (" ".equals(characterValue) &&
> getIgnoreContentStreamSpaceGlyphs())
> + continue;
> +
> // Resets the average character width when we see a
> change in font
> // or a change in the font size
> if (lastPosition != null &&
> @@ -1273,6 +1278,30 @@
> sortByPosition = newSortByPosition;
> }
>
> +
> + /**
> + * This setting only applies if sortByPosition has been set to true.
> + *
> + * Determines whether spaces in the content stream text rendering
> instructions will be ignored during text extraction.
> + *
> + * @return whether space glyphs in the content stream text rendering
> instructions will be ignored during text extraction - default is false
> + */
> + public boolean getIgnoreContentStreamSpaceGlyphs() {
> + return ignoreContentStreamSpaceGlyphs;
> + }
> +
> + /**
> + * This setting only applies if sortByPosition has been set to true.
> + *
> + * Instruct the algorithm to ignore any spaces in the text rendering
> instructions in the content stream, and instead rely purely on the
> algorithm to determine where word breaks are.
> + * This can improve text extraction results where the content stream
> has text overlapping spaces, but could cause some word breaks to not be
> added to the output
> + *
> + * @param newIgnoreRenderedSpaces whether PDF Box should ignore
> context stream spaces
> + */
> + public void setIgnoreContentStreamSpaceGlyphs(boolean
> newIgnoreContentStreamSpaceGlyphs) {
> + ignoreContentStreamSpaceGlyphs = newIgnoreContentStreamSpaceGlyphs;
> + }
> +
> /**
> * Get the current space width-based tolerance value that is being
> used to estimate where spaces in text should be
> * added. Note that the default value for this has been determined
> from trial and error.
> Index:
> pdfbox/src/test/java/org/apache/pdfbox/text/PDFTextStripperOverlapTest.java
> ===================================================================
> ---
> pdfbox/src/test/java/org/apache/pdfbox/text/PDFTextStripperOverlapTest.java
> (revision 0)
> +++
> pdfbox/src/test/java/org/apache/pdfbox/text/PDFTextStripperOverlapTest.java
> (revision 0)
> @@ -0,0 +1,58 @@
> +package org.apache.pdfbox.text;
> +
> +import static org.junit.jupiter.api.Assertions.assertEquals;
> +
> +import org.apache.pdfbox.pdmodel.PDDocument;
> +import org.apache.pdfbox.pdmodel.PDPage;
> +import org.apache.pdfbox.pdmodel.PDPageContentStream;
> +import org.apache.pdfbox.pdmodel.font.PDFont;
> +import org.apache.pdfbox.pdmodel.font.PDType1Font;
> +import org.apache.pdfbox.pdmodel.font.Standard14Fonts.FontName;
> +import org.junit.jupiter.api.Test;
> +
> +public class PDFTextStripperOverlapTest {
> +
> + @Test
> + void testIgnoreContentStreamSpaceGlyphs() throws Exception
> + {
> + try (PDDocument doc = new PDDocument())
> + {
> + PDPage page = new PDPage();
> + try (PDPageContentStream cs = new PDPageContentStream(doc,
> page))
> + {
> + float fontHeight = 8;
> + float x = 50;
> + float y = page.getMediaBox().getHeight() - 50;
> + PDFont font = new PDType1Font(FontName.HELVETICA);
> + cs.beginText();
> + cs.setFont(font, fontHeight);
> + cs.newLineAtOffset(x, y);
> + cs.showText("( )");
> + cs.endText();
> +
> + int indent = 6;
> + float overlapX = x + indent *
> font.getAverageFontWidth()/1000f*fontHeight;
> + PDFont overlapFont = new
> PDType1Font(FontName.TIMES_ROMAN);
> + cs.beginText();
> + cs.setFont(overlapFont, fontHeight*2f);
> + cs.newLineAtOffset(overlapX, y);
> + cs.showText("overlap");
> + cs.endText();
> + }
> + doc.addPage(page);
> +
> + PDFTextStripper stripper = new PDFTextStripper();
> + stripper.setLineSeparator("\n");
> + stripper.setPageEnd("\n");
> + stripper.setStartPage(1);
> + stripper.setEndPage(1);
> + stripper.setSortByPosition(true);
> +
> + stripper.setIgnoreContentStreamSpaceGlyphs(true);
> + String text = stripper.getText(doc);
> + assertEquals("( overlap )\n", text);
> +
> + }
> + }
> +
> +}
>
>
Re: Text extraction adding lots of strange spaces
Posted by Kevin Day <ke...@trumpetinc.com>.
ok - I've done a thorough evaluation of what would be required to properly
suppress overlapping spaces (but leave in the non-overlapping spaces), and
a lot of pathfinding.
What is really needed is to compare two non-space positions and see if
there *should* be a space between them, and if not, then suppress any
spaces that appear between them. I know how to do this, but it would
require a good bit of surgery. Basically, we would need to change to
accumulating text positions by line (without processing them). Then once a
line is accumulated, run through the line and insert word separators (and
remove any overlapping spaces). I would probably do this by moving the
word separation logic into the normalize(line) method. But that is a lot
of surgery.
My concern is that determining if two non-space glyphs are actually
adjacent requires making a determination based on the width of a space
glyph - and that introduces the risk of suppressing an overlapping space
when it shouldn't. I'm not comfortable with that sort of regression -
especially for something that is not adding a ton of value beyond what we
can achieve by just removing Tj spaces entirely.
If someone really cares about preserving the physical layout of the page, I
believe it would be better to put time into making normalize() add space
characters (or tabs maybe) based on the physical layout of the non-space
characters, instead of attempting to preserve any spaces that are part of
the content stream operations.
So for now, I am going to suggest that we implement the setting to ignore
space glyphs in the content stream (as suggested in
https://issues.apache.org/jira/browse/PDFBOX-3774 ). I'm including a patch
that does this below (one source change, plus a self-contained unit test).
I wish this patch was more valuable! But if it helps anything, this change
has a dramatic effect on the robustness of our application, so there is a
ton of value to us to having PDFBox add this setting in 3.0.4.
I'm happy to discuss the above further - I'm just not seeing a huge amount
of value compared to the blanket removal of spaces...
Index: pdfbox/src/main/java/org/apache/pdfbox/text/PDFTextStripper.java
===================================================================
--- pdfbox/src/main/java/org/apache/pdfbox/text/PDFTextStripper.java
(revision 1922522)
+++ pdfbox/src/main/java/org/apache/pdfbox/text/PDFTextStripper.java
(working copy)
@@ -40,6 +40,7 @@
import java.util.TreeMap;
import java.util.TreeSet;
import java.util.regex.Pattern;
+import java.util.stream.Collectors;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
@@ -147,6 +148,7 @@
private boolean shouldSeparateByBeads = true;
private boolean sortByPosition = false;
private boolean addMoreFormatting = false;
+ private boolean ignoreContentStreamSpaceGlyphs = false;
private float indentThreshold = defaultIndentThreshold;
private float dropThreshold = defaultDropThreshold;
@@ -524,11 +526,10 @@
{
IterativeMergeSort.sort(textList, comparator);
}
- finally
- {
- // PDFBOX-5487: Remove all space characters if
contained within the adjacent letters
- removeContainedSpaces(textList);
- }
+
+ // PDFBOX-5487: Remove all space characters if contained
within the adjacent letters
+ removeContainedSpaces(textList);
+
}
startArticle();
@@ -556,6 +557,10 @@
PositionWrapper current = new PositionWrapper(position);
String characterValue = position.getUnicode();
+ // PDFBOX-3774 - conditionally ignore spaces from the
content stream
+ if (" ".equals(characterValue) &&
getIgnoreContentStreamSpaceGlyphs())
+ continue;
+
// Resets the average character width when we see a change
in font
// or a change in the font size
if (lastPosition != null &&
@@ -1273,6 +1278,30 @@
sortByPosition = newSortByPosition;
}
+
+ /**
+ * This setting only applies if sortByPosition has been set to true.
+ *
+ * Determines whether spaces in the content stream text rendering
instructions will be ignored during text extraction.
+ *
+ * @return whether space glyphs in the content stream text rendering
instructions will be ignored during text extraction - default is false
+ */
+ public boolean getIgnoreContentStreamSpaceGlyphs() {
+ return ignoreContentStreamSpaceGlyphs;
+ }
+
+ /**
+ * This setting only applies if sortByPosition has been set to true.
+ *
+ * Instruct the algorithm to ignore any spaces in the text rendering
instructions in the content stream, and instead rely purely on the
algorithm to determine where word breaks are.
+ * This can improve text extraction results where the content stream
has text overlapping spaces, but could cause some word breaks to not be
added to the output
+ *
+ * @param newIgnoreRenderedSpaces whether PDF Box should ignore
context stream spaces
+ */
+ public void setIgnoreContentStreamSpaceGlyphs(boolean
newIgnoreContentStreamSpaceGlyphs) {
+ ignoreContentStreamSpaceGlyphs = newIgnoreContentStreamSpaceGlyphs;
+ }
+
/**
* Get the current space width-based tolerance value that is being
used to estimate where spaces in text should be
* added. Note that the default value for this has been determined
from trial and error.
Index:
pdfbox/src/test/java/org/apache/pdfbox/text/PDFTextStripperOverlapTest.java
===================================================================
---
pdfbox/src/test/java/org/apache/pdfbox/text/PDFTextStripperOverlapTest.java
(revision 0)
+++
pdfbox/src/test/java/org/apache/pdfbox/text/PDFTextStripperOverlapTest.java
(revision 0)
@@ -0,0 +1,58 @@
+package org.apache.pdfbox.text;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+import org.apache.pdfbox.pdmodel.PDDocument;
+import org.apache.pdfbox.pdmodel.PDPage;
+import org.apache.pdfbox.pdmodel.PDPageContentStream;
+import org.apache.pdfbox.pdmodel.font.PDFont;
+import org.apache.pdfbox.pdmodel.font.PDType1Font;
+import org.apache.pdfbox.pdmodel.font.Standard14Fonts.FontName;
+import org.junit.jupiter.api.Test;
+
+public class PDFTextStripperOverlapTest {
+
+ @Test
+ void testIgnoreContentStreamSpaceGlyphs() throws Exception
+ {
+ try (PDDocument doc = new PDDocument())
+ {
+ PDPage page = new PDPage();
+ try (PDPageContentStream cs = new PDPageContentStream(doc,
page))
+ {
+ float fontHeight = 8;
+ float x = 50;
+ float y = page.getMediaBox().getHeight() - 50;
+ PDFont font = new PDType1Font(FontName.HELVETICA);
+ cs.beginText();
+ cs.setFont(font, fontHeight);
+ cs.newLineAtOffset(x, y);
+ cs.showText("( )");
+ cs.endText();
+
+ int indent = 6;
+ float overlapX = x + indent *
font.getAverageFontWidth()/1000f*fontHeight;
+ PDFont overlapFont = new PDType1Font(FontName.TIMES_ROMAN);
+ cs.beginText();
+ cs.setFont(overlapFont, fontHeight*2f);
+ cs.newLineAtOffset(overlapX, y);
+ cs.showText("overlap");
+ cs.endText();
+ }
+ doc.addPage(page);
+
+ PDFTextStripper stripper = new PDFTextStripper();
+ stripper.setLineSeparator("\n");
+ stripper.setPageEnd("\n");
+ stripper.setStartPage(1);
+ stripper.setEndPage(1);
+ stripper.setSortByPosition(true);
+
+ stripper.setIgnoreContentStreamSpaceGlyphs(true);
+ String text = stripper.getText(doc);
+ assertEquals("( overlap )\n", text);
+
+ }
+ }
+
+}
Re: Text extraction adding lots of strange spaces
Posted by Tilman Hausherr <TH...@t-online.de>.
On 14.12.2024 18:02, Kevin Day wrote:
> I will also review that jira commit you referenced. It seems like it might
> be related.
That one has now been committed because the writer confirmed that it
works. You can find it in the stripper by searching for PDFBOX-5487.
If you want I can send you the ~100 files I use for local build
regression tests. It's more than the ones in the repository (in
pdfbox\src\test\resources\input ) because of copyrights. (Beyond that
there's the 250000 files regression test but that one we run only once
before release)
Tilman
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
For additional commands, e-mail: users-help@pdfbox.apache.org
Re: Text extraction adding lots of strange spaces
Posted by Kevin Day <ke...@trumpetinc.com>.
I will also review that jira commit you referenced. It seems like it might
be related.
On Sat, Dec 14, 2024, 9:59 AM Kevin Day <ke...@trumpetinc.com> wrote:
> Great, I will put that together in the next couple of days.
>
> I haven't looked at the iText code in several years - this will be
> greenfield development work.
>
> Take care,
>
> K
>
> On Fri, Dec 13, 2024, 9:21 PM Tilman Hausherr <TH...@t-online.de>
> wrote:
>
>> On 13.12.2024 15:41, Kevin Day wrote:
>> > Ok, this is very interesting.
>> >
>> > So we have one text render that adds a bunch of spaces.
>> >
>> > Then we have another text render that puts visible characters on top of
>> > those spaces.
>> >
>> > Then we positional-sort the text positions, and they wind up
>> interleaving,
>> > which kills the text extraction accuracy.
>> >
>> > Would a reasonable refinement to the algorithm be to look for
>> overlapping
>> > glyphs? And if there is a space that overlaps a non-space, then ignore
>> the
>> > space?
>> >
>> > I know that text extraction is a black art (disclosure: I wrote the
>> > original text extraction modules for iText, so lots of hard-earned
>> > experience here), but I think the above would be appropriate for all
>> > situations...
>> >
>> > Would you be up for reviewing a patch if I implement this?
>> >
>> > PS I also think that having an option to ignore spaces in rendering (the
>> > issue you linked to) would be a good idea. But that should be optional.
>> I'm
>> > happy to include that in my patch if you would like.
>>
>> Yes to both!
>>
>> Note that there is a pending patch about a corner case:
>>
>> https://github.com/apache/pdfbox/pull/155
>>
>> But don't use code from itext unless you are allowed to, i.e. check the
>> papers that you signed. If you patch is more than just a small bugfix
>> you may have to sign something with us as well.
>>
>> https://www.apache.org/licenses/icla.pdf
>>
>> Tilman
>>
>>
>>
>> >
>> > K
>> >
>> > On Thu, Dec 12, 2024, 9:40 PM Tilman Hausherr <TH...@t-online.de>
>> wrote:
>> >
>> >> Hi,
>> >>
>> >> These spaces are really in the PDF:
>> >>
>> >> BT
>> >> /Content <</MCID 224 >>BDC
>> >> 1 i
>> >> /T1_4 1 Tf
>> >> 7 0 0 7 *195.4 110.502* Tm
>> >> *(\( \))Tj*
>> >> EMC
>> >> /Content <</MCID 225 >>BDC
>> >> /T1_0 1 Tf
>> >> -19.686 6.786 Td
>> >> (Beginning capital account)Tj
>> >> /T1_4 1 Tf
>> >> ( )Tj
>> >> 14.057 0 Td
>> >> (.)Tj
>> >> 1.714 0 Td
>> >> (.)Tj
>> >> 1.714 0 Td
>> >> (.)Tj
>> >> EMC
>> >> ET
>> >>
>> >>
>> >> And *195.4 110.502* is really the position, you can move your mouse
>> >> there in PDFDebugger.
>> >>
>> >> The font messages are not important here.
>> >> There is a way to get rid of such spaces, but it requires a source code
>> >> change, it is described here:
>> >> https://issues.apache.org/jira/browse/PDFBOX-3774
>> >>
>> >> However it's possible that other files would have a bad text
>> extraction.
>> >>
>> >> Tilman
>> >>
>> >> On 13.12.2024 03:29, Kevin Day wrote:
>> >>> Hello-
>> >>>
>> >>> We are using PDFTextStripper, and have found some cases where there
>> are a
>> >>> *lot* of extraneous spaces being added to the output. It almost acts
>> >> like
>> >>> the stripper is thinking that the space width of the font is super
>> tiny.
>> >>>
>> >>> I managed to get a document that exhibits the behavior:
>> >>>
>> >>>
>> >>
>> https://drive.google.com/file/d/1B2Mc4mMdsYfk9jKVqQ9OxEhKLRAxprrU/view?usp=sharing
>> >>> The easiest way to see the behavior is in PDFDebugger, View->Show
>> >> Stripper
>> >>> Text Positions.
>> >>>
>> >>> Note in the lower left corner of the document, there is text "999".
>> The
>> >>> text above and below that is fine, but the line with 999 has a *ton*
>> of
>> >>> extra space rectangles displated.
>> >>>
>> >>> The extract text function in PDFDebugger doesn't sort, so that one
>> comes
>> >>> out fine, but if you use PDFTextStripper with sorting enabled (), the
>> >> line
>> >>> renders like this:
>> >>>
>> >>> Withdrawals and distributions . . . $ ( 9 9 9 )
>> >>>
>> >>> Note the many space characters, and that there are even spaces between
>> >> each
>> >>> 9.
>> >>>
>> >>> I also observe that the PDF has warning messages about fonts (not
>> sure if
>> >>> this might be involved):
>> >>>
>> >>> [main] WARN org.apache.pdfbox.pdmodel.font.PDType1Font - Using
>> fallback
>> >>> font ArialMT for HelveticaLTStd-Roman
>> >>>
>> >>> [main] WARN org.apache.fontbox.ttf.CmapSubtable - Format 14 cmap
>> table is
>> >>> not supported and will be ignored
>> >>>
>> >>>
>> >>>
>> >>> It almost acts like the parenthesis on the line are triggering some
>> >>> different detection mode where the font's space width is computing to
>> be
>> >>> much smaller than it should be.
>> >>>
>> >>> Any ideas on what is going on or if it is fixable?
>> >>>
>> >>> Thanks!
>> >>>
>> >>> - K
>> >>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
>> For additional commands, e-mail: users-help@pdfbox.apache.org
>>
>>
Re: Text extraction adding lots of strange spaces
Posted by Kevin Day <ke...@trumpetinc.com>.
Great, I will put that together in the next couple of days.
I haven't looked at the iText code in several years - this will be
greenfield development work.
Take care,
K
On Fri, Dec 13, 2024, 9:21 PM Tilman Hausherr <TH...@t-online.de> wrote:
> On 13.12.2024 15:41, Kevin Day wrote:
> > Ok, this is very interesting.
> >
> > So we have one text render that adds a bunch of spaces.
> >
> > Then we have another text render that puts visible characters on top of
> > those spaces.
> >
> > Then we positional-sort the text positions, and they wind up
> interleaving,
> > which kills the text extraction accuracy.
> >
> > Would a reasonable refinement to the algorithm be to look for overlapping
> > glyphs? And if there is a space that overlaps a non-space, then ignore
> the
> > space?
> >
> > I know that text extraction is a black art (disclosure: I wrote the
> > original text extraction modules for iText, so lots of hard-earned
> > experience here), but I think the above would be appropriate for all
> > situations...
> >
> > Would you be up for reviewing a patch if I implement this?
> >
> > PS I also think that having an option to ignore spaces in rendering (the
> > issue you linked to) would be a good idea. But that should be optional.
> I'm
> > happy to include that in my patch if you would like.
>
> Yes to both!
>
> Note that there is a pending patch about a corner case:
>
> https://github.com/apache/pdfbox/pull/155
>
> But don't use code from itext unless you are allowed to, i.e. check the
> papers that you signed. If you patch is more than just a small bugfix
> you may have to sign something with us as well.
>
> https://www.apache.org/licenses/icla.pdf
>
> Tilman
>
>
>
> >
> > K
> >
> > On Thu, Dec 12, 2024, 9:40 PM Tilman Hausherr <TH...@t-online.de>
> wrote:
> >
> >> Hi,
> >>
> >> These spaces are really in the PDF:
> >>
> >> BT
> >> /Content <</MCID 224 >>BDC
> >> 1 i
> >> /T1_4 1 Tf
> >> 7 0 0 7 *195.4 110.502* Tm
> >> *(\( \))Tj*
> >> EMC
> >> /Content <</MCID 225 >>BDC
> >> /T1_0 1 Tf
> >> -19.686 6.786 Td
> >> (Beginning capital account)Tj
> >> /T1_4 1 Tf
> >> ( )Tj
> >> 14.057 0 Td
> >> (.)Tj
> >> 1.714 0 Td
> >> (.)Tj
> >> 1.714 0 Td
> >> (.)Tj
> >> EMC
> >> ET
> >>
> >>
> >> And *195.4 110.502* is really the position, you can move your mouse
> >> there in PDFDebugger.
> >>
> >> The font messages are not important here.
> >> There is a way to get rid of such spaces, but it requires a source code
> >> change, it is described here:
> >> https://issues.apache.org/jira/browse/PDFBOX-3774
> >>
> >> However it's possible that other files would have a bad text extraction.
> >>
> >> Tilman
> >>
> >> On 13.12.2024 03:29, Kevin Day wrote:
> >>> Hello-
> >>>
> >>> We are using PDFTextStripper, and have found some cases where there
> are a
> >>> *lot* of extraneous spaces being added to the output. It almost acts
> >> like
> >>> the stripper is thinking that the space width of the font is super
> tiny.
> >>>
> >>> I managed to get a document that exhibits the behavior:
> >>>
> >>>
> >>
> https://drive.google.com/file/d/1B2Mc4mMdsYfk9jKVqQ9OxEhKLRAxprrU/view?usp=sharing
> >>> The easiest way to see the behavior is in PDFDebugger, View->Show
> >> Stripper
> >>> Text Positions.
> >>>
> >>> Note in the lower left corner of the document, there is text "999".
> The
> >>> text above and below that is fine, but the line with 999 has a *ton* of
> >>> extra space rectangles displated.
> >>>
> >>> The extract text function in PDFDebugger doesn't sort, so that one
> comes
> >>> out fine, but if you use PDFTextStripper with sorting enabled (), the
> >> line
> >>> renders like this:
> >>>
> >>> Withdrawals and distributions . . . $ ( 9 9 9 )
> >>>
> >>> Note the many space characters, and that there are even spaces between
> >> each
> >>> 9.
> >>>
> >>> I also observe that the PDF has warning messages about fonts (not sure
> if
> >>> this might be involved):
> >>>
> >>> [main] WARN org.apache.pdfbox.pdmodel.font.PDType1Font - Using fallback
> >>> font ArialMT for HelveticaLTStd-Roman
> >>>
> >>> [main] WARN org.apache.fontbox.ttf.CmapSubtable - Format 14 cmap table
> is
> >>> not supported and will be ignored
> >>>
> >>>
> >>>
> >>> It almost acts like the parenthesis on the line are triggering some
> >>> different detection mode where the font's space width is computing to
> be
> >>> much smaller than it should be.
> >>>
> >>> Any ideas on what is going on or if it is fixable?
> >>>
> >>> Thanks!
> >>>
> >>> - K
> >>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> For additional commands, e-mail: users-help@pdfbox.apache.org
>
>
Re: Text extraction adding lots of strange spaces
Posted by Tilman Hausherr <TH...@t-online.de>.
On 13.12.2024 15:41, Kevin Day wrote:
> Ok, this is very interesting.
>
> So we have one text render that adds a bunch of spaces.
>
> Then we have another text render that puts visible characters on top of
> those spaces.
>
> Then we positional-sort the text positions, and they wind up interleaving,
> which kills the text extraction accuracy.
>
> Would a reasonable refinement to the algorithm be to look for overlapping
> glyphs? And if there is a space that overlaps a non-space, then ignore the
> space?
>
> I know that text extraction is a black art (disclosure: I wrote the
> original text extraction modules for iText, so lots of hard-earned
> experience here), but I think the above would be appropriate for all
> situations...
>
> Would you be up for reviewing a patch if I implement this?
>
> PS I also think that having an option to ignore spaces in rendering (the
> issue you linked to) would be a good idea. But that should be optional. I'm
> happy to include that in my patch if you would like.
Yes to both!
Note that there is a pending patch about a corner case:
https://github.com/apache/pdfbox/pull/155
But don't use code from itext unless you are allowed to, i.e. check the
papers that you signed. If you patch is more than just a small bugfix
you may have to sign something with us as well.
https://www.apache.org/licenses/icla.pdf
Tilman
>
> K
>
> On Thu, Dec 12, 2024, 9:40 PM Tilman Hausherr <TH...@t-online.de> wrote:
>
>> Hi,
>>
>> These spaces are really in the PDF:
>>
>> BT
>> /Content <</MCID 224 >>BDC
>> 1 i
>> /T1_4 1 Tf
>> 7 0 0 7 *195.4 110.502* Tm
>> *(\( \))Tj*
>> EMC
>> /Content <</MCID 225 >>BDC
>> /T1_0 1 Tf
>> -19.686 6.786 Td
>> (Beginning capital account)Tj
>> /T1_4 1 Tf
>> ( )Tj
>> 14.057 0 Td
>> (.)Tj
>> 1.714 0 Td
>> (.)Tj
>> 1.714 0 Td
>> (.)Tj
>> EMC
>> ET
>>
>>
>> And *195.4 110.502* is really the position, you can move your mouse
>> there in PDFDebugger.
>>
>> The font messages are not important here.
>> There is a way to get rid of such spaces, but it requires a source code
>> change, it is described here:
>> https://issues.apache.org/jira/browse/PDFBOX-3774
>>
>> However it's possible that other files would have a bad text extraction.
>>
>> Tilman
>>
>> On 13.12.2024 03:29, Kevin Day wrote:
>>> Hello-
>>>
>>> We are using PDFTextStripper, and have found some cases where there are a
>>> *lot* of extraneous spaces being added to the output. It almost acts
>> like
>>> the stripper is thinking that the space width of the font is super tiny.
>>>
>>> I managed to get a document that exhibits the behavior:
>>>
>>>
>> https://drive.google.com/file/d/1B2Mc4mMdsYfk9jKVqQ9OxEhKLRAxprrU/view?usp=sharing
>>> The easiest way to see the behavior is in PDFDebugger, View->Show
>> Stripper
>>> Text Positions.
>>>
>>> Note in the lower left corner of the document, there is text "999". The
>>> text above and below that is fine, but the line with 999 has a *ton* of
>>> extra space rectangles displated.
>>>
>>> The extract text function in PDFDebugger doesn't sort, so that one comes
>>> out fine, but if you use PDFTextStripper with sorting enabled (), the
>> line
>>> renders like this:
>>>
>>> Withdrawals and distributions . . . $ ( 9 9 9 )
>>>
>>> Note the many space characters, and that there are even spaces between
>> each
>>> 9.
>>>
>>> I also observe that the PDF has warning messages about fonts (not sure if
>>> this might be involved):
>>>
>>> [main] WARN org.apache.pdfbox.pdmodel.font.PDType1Font - Using fallback
>>> font ArialMT for HelveticaLTStd-Roman
>>>
>>> [main] WARN org.apache.fontbox.ttf.CmapSubtable - Format 14 cmap table is
>>> not supported and will be ignored
>>>
>>>
>>>
>>> It almost acts like the parenthesis on the line are triggering some
>>> different detection mode where the font's space width is computing to be
>>> much smaller than it should be.
>>>
>>> Any ideas on what is going on or if it is fixable?
>>>
>>> Thanks!
>>>
>>> - K
>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
For additional commands, e-mail: users-help@pdfbox.apache.org
Re: Text extraction adding lots of strange spaces
Posted by Kevin Day <ke...@trumpetinc.com>.
Ok, this is very interesting.
So we have one text render that adds a bunch of spaces.
Then we have another text render that puts visible characters on top of
those spaces.
Then we positional-sort the text positions, and they wind up interleaving,
which kills the text extraction accuracy.
Would a reasonable refinement to the algorithm be to look for overlapping
glyphs? And if there is a space that overlaps a non-space, then ignore the
space?
I know that text extraction is a black art (disclosure: I wrote the
original text extraction modules for iText, so lots of hard-earned
experience here), but I think the above would be appropriate for all
situations...
Would you be up for reviewing a patch if I implement this?
PS I also think that having an option to ignore spaces in rendering (the
issue you linked to) would be a good idea. But that should be optional. I'm
happy to include that in my patch if you would like.
K
On Thu, Dec 12, 2024, 9:40 PM Tilman Hausherr <TH...@t-online.de> wrote:
> Hi,
>
> These spaces are really in the PDF:
>
> BT
> /Content <</MCID 224 >>BDC
> 1 i
> /T1_4 1 Tf
> 7 0 0 7 *195.4 110.502* Tm
> *(\( \))Tj*
> EMC
> /Content <</MCID 225 >>BDC
> /T1_0 1 Tf
> -19.686 6.786 Td
> (Beginning capital account)Tj
> /T1_4 1 Tf
> ( )Tj
> 14.057 0 Td
> (.)Tj
> 1.714 0 Td
> (.)Tj
> 1.714 0 Td
> (.)Tj
> EMC
> ET
>
>
> And *195.4 110.502* is really the position, you can move your mouse
> there in PDFDebugger.
>
> The font messages are not important here.
> There is a way to get rid of such spaces, but it requires a source code
> change, it is described here:
> https://issues.apache.org/jira/browse/PDFBOX-3774
>
> However it's possible that other files would have a bad text extraction.
>
> Tilman
>
> On 13.12.2024 03:29, Kevin Day wrote:
> > Hello-
> >
> > We are using PDFTextStripper, and have found some cases where there are a
> > *lot* of extraneous spaces being added to the output. It almost acts
> like
> > the stripper is thinking that the space width of the font is super tiny.
> >
> > I managed to get a document that exhibits the behavior:
> >
> >
> https://drive.google.com/file/d/1B2Mc4mMdsYfk9jKVqQ9OxEhKLRAxprrU/view?usp=sharing
> >
> > The easiest way to see the behavior is in PDFDebugger, View->Show
> Stripper
> > Text Positions.
> >
> > Note in the lower left corner of the document, there is text "999". The
> > text above and below that is fine, but the line with 999 has a *ton* of
> > extra space rectangles displated.
> >
> > The extract text function in PDFDebugger doesn't sort, so that one comes
> > out fine, but if you use PDFTextStripper with sorting enabled (), the
> line
> > renders like this:
> >
> > Withdrawals and distributions . . . $ ( 9 9 9 )
> >
> > Note the many space characters, and that there are even spaces between
> each
> > 9.
> >
> > I also observe that the PDF has warning messages about fonts (not sure if
> > this might be involved):
> >
> > [main] WARN org.apache.pdfbox.pdmodel.font.PDType1Font - Using fallback
> > font ArialMT for HelveticaLTStd-Roman
> >
> > [main] WARN org.apache.fontbox.ttf.CmapSubtable - Format 14 cmap table is
> > not supported and will be ignored
> >
> >
> >
> > It almost acts like the parenthesis on the line are triggering some
> > different detection mode where the font's space width is computing to be
> > much smaller than it should be.
> >
> > Any ideas on what is going on or if it is fixable?
> >
> > Thanks!
> >
> > - K
> >
>
Re: Text extraction adding lots of strange spaces
Posted by Tilman Hausherr <TH...@t-online.de>.
Hi,
These spaces are really in the PDF:
BT
/Content <</MCID 224 >>BDC
1 i
/T1_4 1 Tf
7 0 0 7 *195.4 110.502* Tm
*(\( \))Tj*
EMC
/Content <</MCID 225 >>BDC
/T1_0 1 Tf
-19.686 6.786 Td
(Beginning capital account)Tj
/T1_4 1 Tf
( )Tj
14.057 0 Td
(.)Tj
1.714 0 Td
(.)Tj
1.714 0 Td
(.)Tj
EMC
ET
And *195.4 110.502* is really the position, you can move your mouse
there in PDFDebugger.
The font messages are not important here.
There is a way to get rid of such spaces, but it requires a source code
change, it is described here:
https://issues.apache.org/jira/browse/PDFBOX-3774
However it's possible that other files would have a bad text extraction.
Tilman
On 13.12.2024 03:29, Kevin Day wrote:
> Hello-
>
> We are using PDFTextStripper, and have found some cases where there are a
> *lot* of extraneous spaces being added to the output. It almost acts like
> the stripper is thinking that the space width of the font is super tiny.
>
> I managed to get a document that exhibits the behavior:
>
> https://drive.google.com/file/d/1B2Mc4mMdsYfk9jKVqQ9OxEhKLRAxprrU/view?usp=sharing
>
> The easiest way to see the behavior is in PDFDebugger, View->Show Stripper
> Text Positions.
>
> Note in the lower left corner of the document, there is text "999". The
> text above and below that is fine, but the line with 999 has a *ton* of
> extra space rectangles displated.
>
> The extract text function in PDFDebugger doesn't sort, so that one comes
> out fine, but if you use PDFTextStripper with sorting enabled (), the line
> renders like this:
>
> Withdrawals and distributions . . . $ ( 9 9 9 )
>
> Note the many space characters, and that there are even spaces between each
> 9.
>
> I also observe that the PDF has warning messages about fonts (not sure if
> this might be involved):
>
> [main] WARN org.apache.pdfbox.pdmodel.font.PDType1Font - Using fallback
> font ArialMT for HelveticaLTStd-Roman
>
> [main] WARN org.apache.fontbox.ttf.CmapSubtable - Format 14 cmap table is
> not supported and will be ignored
>
>
>
> It almost acts like the parenthesis on the line are triggering some
> different detection mode where the font's space width is computing to be
> much smaller than it should be.
>
> Any ideas on what is going on or if it is fixable?
>
> Thanks!
>
> - K
>