You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pdfbox.apache.org by Gunnar Brand <Gu...@interface-projects.de> on 2021/03/24 13:40:52 UTC
Empty cmap in TTF Files.
Hi.
I am working on merging original PDFs and the PDF/HOCR output of Tesseract, as to create a searchable PDF. Transplanting the glyphless font used by tesseract was no problem, it doesn’t matter if I simply use the font in the original PDF or use cloneutil, when saving the file the font is embedded properly.
The problem is when I show text using a content stream, I get a “No Glyph for …” exception. I traced this down to the glyphless font containing empty cmap tables. There is a CIDToGIDMap. Coincidentally PDFBOX-5103 just addressed this issue with a reverse mapping if the cmap is null. But the cmap is just empty and will return 0 for any character code, so this new feature will never work in this case.
For testing I modified TrueTypeFont.getUnicodeCmapImpl(isStrict) so that it ignores empty cmap subtables (even the fallback at the end of the method now being a loop). With this PDFBox will happily use the tesseract glyphless font. Now I lack the knowledge if empty cmaps make any sense at all and if they do I will simply write raw show text commands, but maybe it is something to consider?
Gunnar
Re: Empty cmap in TTF Files.
Posted by Tilman Hausherr <TH...@t-online.de>.
Attachments are swallowed. In the meantime I looked at a PDF file
created with tesseract. Yes, PDFBox claims that the ttf file has a
cmapTable with two empty subtables. I tried around a bit and went
nowhere. I think the solution would be to create an improved font by
misusing PDCIDFontType2Embedder, and feeding it with correct cmap data
and then saving this font. Or use a real font, but make it invisible
(text rendering mode 3).
The behavior of PDFBox makes sense, this prevents the user from using
glyphs that don't exist.
One weird thing is that DTL OTMaster claims the two subtables aren't
empty. (but maybe this is some sort of default) At the same time, they
claim that the length is 10.
Tilman
Am 25.03.2021 um 13:30 schrieb Gunnar Brand:
> Hi.
>
> The process is as follows:
> 1) For images: use the image
> For PDFs: render each page to 300 dpi (since optimized PDFs don't necessarily have a single big image), maybe even with text if text extraction returned gibberish (missing unicode mapping).
> 2) Use tesseract to OCR image/page with PDF and HOCR output. (for pages: create an imageless PDF). The HOCR is used for additional page layout information and word confidence values.
> 3) For images, use the HOCR to filter the PDF text stream and add layout information
> For PDFs, insert the tesseract PDF text stream into the orignal PDF's page (+add that glyphless font), use the HOCR to filter and add layout information.
>
> For step 3, I would like to use a normal PDPageContentStream to add the content instead of working with a raw stream. But that step fails since I cannot use the showText() method with a Font that has an empty cmap.
>
> I attached an empty tesseract PDF with the glyphless font. Appending text using the font to the single page in there will fail immediately with the exception due to the empty cmap. Adding the font to any other PDF and trying to show text using it will fail as well.
>
> I can probably get away with just creating/transfering the Tj commands raw, but I was wondering if the empty cmap behaviour is ok or would it be better to ignore empty cmaps (i.e. look for a non empty one first and return null if none can be found in TrueTypeFont.getUnicodeCmapImpl).
>
> Gunnar
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: Tilman Hausherr <TH...@t-online.de>
> Gesendet: Donnerstag, 25. März 2021 04:37
> An: users@pdfbox.apache.org
> Betreff: Re: Empty cmap in TTF Files.
>
> Am 24.03.2021 um 14:40 schrieb Gunnar Brand:
>> Hi.
>>
>> I am working on merging original PDFs and the PDF/HOCR output of Tesseract, as to create a searchable PDF. Transplanting the glyphless font used by tesseract was no problem, it doesn’t matter if I simply use the font in the original PDF or use cloneutil, when saving the file the font is embedded properly.
>>
>> The problem is when I show text using a content stream, I get a “No Glyph for …” exception. I traced this down to the glyphless font containing empty cmap tables. There is a CIDToGIDMap. Coincidentally PDFBOX-5103 just addressed this issue with a reverse mapping if the cmap is null. But the cmap is just empty and will return 0 for any character code, so this new feature will never work in this case.
>>
>> For testing I modified TrueTypeFont.getUnicodeCmapImpl(isStrict) so that it ignores empty cmap subtables (even the fallback at the end of the method now being a loop). With this PDFBox will happily use the tesseract glyphless font. Now I lack the knowledge if empty cmaps make any sense at all and if they do I will simply write raw show text commands, but maybe it is something to consider?
>>
>> Gunnar
> I tried tesseract some time ago and it generates searchable PDFs out of the box, why not use that?
>
> Can you upload one of your files to a sharehoster so that I understand what this is about?
>
> Tilman
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> For additional commands, e-mail: users-help@pdfbox.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> For additional commands, e-mail: users-help@pdfbox.apache.org
AW: Empty cmap in TTF Files.
Posted by Gunnar Brand <Gu...@interface-projects.de>.
Hi Constantine
I worked on it a bit.
In the end I don't render the HOCR directly but simply transfer the Tesseract PDF data over to the PDF.
For this I clone the glyphless font (which is embedded as resource in the jar) and add it to the target page and then while transfering the PDF data change the font commands to the new font name on the fly. I also add a lot of marked content since I have the info from the HOCR file (something Tesseract should have done).
I still tried to get the HOCR rendering working, which is a big PITA once you enter the realm of sloped or rotated text. Basically all HOCR does is draw a rectangle around words and lines and giving you a single pointer to the baseline of the first word. So you later need to calculate intersections of the baseline with word rectangles etc etc. I probably spent more time on it than I should but I wanted to get it working.
When doing the PDF transfer I found a way to mix PDContentStream with raw commands in a nicer way, and this works fine with the "broken" glyphless font, too.
All you do is set font size, set the text matrix or use newLineAtOFfset , set the text zoom and then write a raw COSString and a raw Tj operator.
There are issues with RTL text I haven't looked at yet (you probably need to reverse the order as tesseract does) and vertical text is another thing probably not working.
I will dump the code right here. Please note that you don't really need to create a XFormObject, you can use the page directly and apppend the text data.
The HOCR classes I am using is a hastily written HOCR parser for the tesseract hocr files. All it does is parse blocks, paragraphs, line and word XML into java classes and transfer coordinates into the PDF coordinate system (bottom to top). It does some advanced math for the baseline and rotation handling, especially to get the start coordinates for lines and words . (slope only makes sense for unrotated text, though). If all you have is 90 degree angled stuff it is kinda easy to calculate these.
I added some marked content code (excesivly with confidence properties on word level) that might make text extractors happy later on. It also requires the raw method since the other stream does not do inline dictionaries at all. (There is a inline dicitionary property "ActualText" that can overwrite the whole content of the glyphs inside, maybe worth trying out, too.)
http://kba.cloud/hocr-spec/1.2/
Gunnar
class PageMerger
{
private final static Charset UTF16BE = Charset.forName("UTF-16BE");
private final static Operator BDC = Operator.getOperator(OperatorName.BEGIN_MARKED_CONTENT_SEQ);
private final static COSName DIV = COSName.getPDFName("Div");
private final static COSName P = COSName.getPDFName("P");
private final static COSName SPAN = COSName.getPDFName("Span");
// might be a good idea to thing about this, see PDF spec
private final static COSName REVERSED_CHARS = COSName.getPDFName("ReversedChars");
private final static COSName LANG = COSName.getPDFName("Lang");
private final static COSName WRITING_MODE = COSName.getPDFName("WritingMode");
private final static COSString LTR = new COSString("LrTb");
private final static COSString RTL = new COSString("RlTb");
private final static COSName CONFIDENCE = COSName.getPDFName("X_Confidence");
private final PDDocument _doc;
private PDFont _glyphless;
public PageMerger(PDDocument doc) throws IOException
{
_doc = doc;
}
public void addHocrText(PDPage page, HocrPage hocr) throws IOException
{
final PDRectangle pageBox = page.getCropBox(), formBox = rectangle(hocr.bbox);
PDFormXObject form = new PDFormXObject(_doc);
form.setBBox(formBox);
form.setResources(new PDResources());
final COSName fname = addGlyphless(form.getResources());
final float glyphScale = 100f * _glyphless.getBoundingBox().getHeight() / _glyphless.getBoundingBox().getWidth();
PDPageContentStream cs = new PDPageContentStream(_doc, form, form.getContentStream().createOutputStream(COSName.FLATE_DECODE));
ContentStreamWriter csw = new ContentStreamWriter(new PDPageContentStreamAdapter(cs));
for ( HocrBlock block : hocr.blocks ) {
cs.beginMarkedContent(DIV);
cs.beginText();
cs.setRenderingMode(RenderingMode.NEITHER);
float size = -1f;
for ( HocrParagraph para : block.paragraphs ) {
if ( para.lang==null && para.dir==null ) {
cs.beginMarkedContent(P);
} else {
COSDictionary dict = new COSDictionary();
dict.setDirect(true);
if ( para.lang!=null ) dict.setString(LANG, para.lang);
if ( para.dir!=null ) dict.setItem(WRITING_MODE, para.dir=="ltr" ? LTR : RTL);
csw.writeTokens(P, dict, BDC);
}
for ( HocrLine line : para.lines ) {
cs.beginMarkedContent(SPAN);
Point2D p = line.getStart();
cs.setTextMatrix(Matrix.getRotateInstance(Math.toRadians(line.getRotation()), (float)p.getX(), (float)p.getY()));
if ( line.size!=size ) cs.setFont(_glyphless, size = line.size);
HocrWord last = null;
for ( Iterator<HocrWord> lit = line.words.iterator(); lit.hasNext(); ) {
HocrWord word = lit.next();
COSDictionary dict = new COSDictionary();
dict.setDirect(true);
dict.setFloat(CONFIDENCE, word.confidence);
if ( word.lang!=null ) dict.setString(LANG, word.lang);
if ( word.dir!=null ) dict.setItem(WRITING_MODE, word.dir=="ltr" ? LTR : RTL);
csw.writeTokens(SPAN, dict, BDC);
// preferable but less reliable (produces the same error as tesseract does) :(
if ( last!=null ) cs.newLineAtOffset(word.getDistance() - last.getDistance(), 0);
last = word;
// overkill but places words at the right position
// p = word.getStart();
// cs.setTextMatrix(Matrix.getRotateInstance(Math.toRadians(line.getRotation()), (float)p.getX(), (float)p.getY()));
String text = word.text;
if ( lit.hasNext() && !text.isBlank() ) text += " ";
float zoom = glyphScale * word.getWidth() / (line.size * text.codePointCount(0, text.length()));
cs.setHorizontalScaling(zoom);
// COSString constructor with java string argument adds a BOM in the beginning, that ain't good.
csw.writeToken(new COSString(text.getBytes(UTF16BE)));
csw.writeToken(Operator.getOperator(OperatorName.SHOW_TEXT));
cs.endMarkedContent();
}
cs.endMarkedContent();
}
cs.endMarkedContent();
}
cs.endText();
cs.endMarkedContent();
}
cs.close();
// Rotation matrix a b c d e f: cos sin -sin cos 0 0
// x = a*x + c*y + e
// y = b*x + d*y + f
int rotation = page.getRotation();
final float x1 = pageBox.getLowerLeftX(), y1 = pageBox.getLowerLeftY();
final float x2 = pageBox.getUpperRightX(), y2 = pageBox.getUpperRightY();
final float s = pageBox.getWidth() / ((rotation % 180 == 0) ? formBox.getWidth() : formBox.getHeight());
Matrix m = null;
switch(rotation) {
case 0: m = new Matrix( s, 0, 0, s, x1, y1); break;
case 180: m = new Matrix(-s, 0, 0, -s, x2, x2); break;
case 90: m = new Matrix(0, s, -s, 0, x2, y1); break;
case 270: m = new Matrix(0, -s, s, 0, x1, y2); break;
}
cs = new PDPageContentStream(_doc, page, AppendMode.APPEND, true, true);
cs.transform(m);
cs.drawForm(form);
cs.close();
}
private COSName addGlyphless(PDResources target) throws IOException
{
if ( _glyphless!=null ) return target.add(_glyphless);
try (
InputStream in = PageMerger.class.getResourceAsStream("glyphless.pdf");
PDDocument template = PDDocument.load(in)
) {
PDResources source = template.getPage(0).getResources();
PDFont font = cloneFont(source, source.getFontNames().iterator().next());
COSName name = target.add(font);
_glyphless = target.getFont(name);
return name;
}
}
private PDFont cloneFont(PDResources source, COSName name) throws IOException
{
PDFont f1 = source.getFont(name);
PDFCloneUtility c = new PDFCloneUtility(_doc);
return new PDType0Font((COSDictionary)c.cloneForNewDocument(f1.getCOSObject()));
}
private static PDRectangle rectangle(Rectangle bbox)
{
return new PDRectangle(bbox.x, bbox.y, bbox.width, bbox.height);
}
@SuppressWarnings("deprecation")
private static class PDPageContentStreamAdapter extends OutputStream
{
private final PDPageContentStream stream;
PDPageContentStreamAdapter(PDPageContentStream stream) {
this.stream = stream;
}
@Override public void write(int b) throws IOException {
stream.appendRawCommands(b);
}
@Override public void write(byte[] b) throws IOException {
stream.appendRawCommands(b);
}
@Override public void write(byte[] b, int off, int len) throws IOException {
stream.appendRawCommands(off==0 && len==b.length ? b : Arrays.copyOfRange(b, off, off + len));
}
}
}
-----Ursprüngliche Nachricht-----
Von: Constantine Dokolas <cd...@gmail.com>
Gesendet: Freitag, 26. März 2021 11:38
An: users@pdfbox.apache.org
Betreff: Re: Empty cmap in TTF Files.
Hi, Gunnar,
Do you think this SO question
<https://stackoverflow.com/questions/49363954/using-arialmt-for-arabic-text-without-embedding-font-with-pdfbox>
is related? I'm the OP and the (admittedly somewhat niche) case for no-glyph (i.e. non-renderable) chars on a PDF is a "capability" that's been missing for me.
To give some context, at work I'm responsible for a library that, among other things, overlays OCRed text (from diverse sources) on images placed in PDF pages. There have been issues I've overcome (especially concerning Unicode), but "glyphless font" embedding is something that would really make a noticeable impact on PDF size. Most OCR software that produce PDFs from images do this in some way, Tesseract included.
I think PDFBox is a great library for reading and generating PDFs, and I'm seriously considering contributing as soon as possible. A big thanks to everyone working to make this project successful.
C.D.
--
There is a computer disease that anybody who works with computers knows about. It's a very serious disease and it interferes completely with the work. The trouble with computers is that you 'play' with them!
- Richard P. Feynman
On Thu, Mar 25, 2021 at 2:30 PM Gunnar Brand < Gunnar.Brand@interface-projects.de> wrote:
> Hi.
>
> The process is as follows:
> 1) For images: use the image
> For PDFs: render each page to 300 dpi (since optimized PDFs don't
> necessarily have a single big image), maybe even with text if text
> extraction returned gibberish (missing unicode mapping).
> 2) Use tesseract to OCR image/page with PDF and HOCR output. (for pages:
> create an imageless PDF). The HOCR is used for additional page layout
> information and word confidence values.
> 3) For images, use the HOCR to filter the PDF text stream and add
> layout information
> For PDFs, insert the tesseract PDF text stream into the orignal
> PDF's page (+add that glyphless font), use the HOCR to filter and add
> layout information.
>
> For step 3, I would like to use a normal PDPageContentStream to add
> the content instead of working with a raw stream. But that step fails
> since I cannot use the showText() method with a Font that has an empty cmap.
>
> I attached an empty tesseract PDF with the glyphless font. Appending
> text using the font to the single page in there will fail immediately
> with the exception due to the empty cmap. Adding the font to any other
> PDF and trying to show text using it will fail as well.
>
> I can probably get away with just creating/transfering the Tj commands
> raw, but I was wondering if the empty cmap behaviour is ok or would it
> be better to ignore empty cmaps (i.e. look for a non empty one first
> and return null if none can be found in TrueTypeFont.getUnicodeCmapImpl).
>
> Gunnar
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: Tilman Hausherr <TH...@t-online.de>
> Gesendet: Donnerstag, 25. März 2021 04:37
> An: users@pdfbox.apache.org
> Betreff: Re: Empty cmap in TTF Files.
>
> Am 24.03.2021 um 14:40 schrieb Gunnar Brand:
> > Hi.
> >
> > I am working on merging original PDFs and the PDF/HOCR output of
> Tesseract, as to create a searchable PDF. Transplanting the glyphless
> font used by tesseract was no problem, it doesn’t matter if I simply
> use the font in the original PDF or use cloneutil, when saving the
> file the font is embedded properly.
> >
> > The problem is when I show text using a content stream, I get a “No
> Glyph for …” exception. I traced this down to the glyphless font
> containing empty cmap tables. There is a CIDToGIDMap. Coincidentally
> PDFBOX-5103 just addressed this issue with a reverse mapping if the
> cmap is null. But the cmap is just empty and will return 0 for any
> character code, so this new feature will never work in this case.
> >
> > For testing I modified TrueTypeFont.getUnicodeCmapImpl(isStrict) so
> > that
> it ignores empty cmap subtables (even the fallback at the end of the
> method now being a loop). With this PDFBox will happily use the
> tesseract glyphless font. Now I lack the knowledge if empty cmaps make
> any sense at all and if they do I will simply write raw show text
> commands, but maybe it is something to consider?
> >
> > Gunnar
>
> I tried tesseract some time ago and it generates searchable PDFs out
> of the box, why not use that?
>
> Can you upload one of your files to a sharehoster so that I understand
> what this is about?
>
> Tilman
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> For additional commands, e-mail: users-help@pdfbox.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> For additional commands, e-mail: users-help@pdfbox.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
For additional commands, e-mail: users-help@pdfbox.apache.org
AW: Empty cmap in TTF Files.
Posted by Gunnar Brand <Gu...@interface-projects.de>.
Hi Constantine.
Yes, using the glyphless font of tesseract is perfect for putting invisible OCR text over an image. You bascially take an OCR word (which contains size and all in image coordiantes) and convert this into PDF coordinates and width and all and then show a text with the glyphless font at these coordinates with a proper scaled font. Since characters are usually not all the same width it is not perfect but good enough for highlighting in readers etc.
That font is quite a hack, it consists of two PDF-fonts, with the TTF file being more or less a placeholder:
A Type 0 font with a ToUnicode map, mapping cid 0 - 0xffff to unicode codepoints 0 - 0xffff. I have not yet verified how this handles non BMP characters, but the Tesseract PDF renderer c++ code is easy to understand and they claim it works. (Not sure about this. Need to read the tesseract pdfrenderer code some more.) This is what allows a client app to convert character ids in the content stream to unicode.
This parent font contains a descandant font, which is a type2 font, its fontdescriptor points to the TTF. It basically describes the size of the glyphs and all (which is exactly 1 pt high and 0.5pts wide, so fontsize translates directly). It also declares a CID to GID map, that maps all CIDs to GID 1.
[Fun fact: even this small font puts quite the strain on things. The PDFBox PDFDebugger renders pages without a resource cache and tesseract sets the font size very, very often. Which reloads the font (w/o verifying if already set) every time if rendered, which my modified debugger does. The GC can't keep up with that rate of object creation and either the debugger died with OOM after 1.5 GB had been used or it took 20 - 30 seconds. With resource cache hacked in it actually renders quickly. I tried some improvements to FontBox before realizing that the cache was missing, and I got object creation rate down so that even without resource cache it peaks a 150 MB and renders in 3 seconds. Might polish that when I have some free time and hand it in.]
So basically all info for PDFBox is there, but unfortunately the TTF contains an empty cmap. Now if the cmap was missing, PDFBox would use the other tables and work fine (in lenient mode, in strict mode it would fail).
To circumvent that one can either:
* Hack PDFBox to ignore empty cmaps (I did that for testing).
* Write the Tj command raw (alas these methods are labelled deprecated and will probably removed in the future, which would be kind of a bummer for advanced users).
* Trick PDFBox to do what we want.
How do we trick PDFBox? Yes, we make the cmap go missing!
The file I tried to append in the other message is was a 3KB Pdf consisting of an empty page OCR of tesseract. I call it template.pdf. So all it contains is the glyphless font. You'd load the font from this file if you have anything non tesseract OCR pdf that you need to merge into a PDF. Simply copy this template into a 2nd one, let's call it trickster.pdf. This you open in a hex editor and swap the p in "cmap" of the embedded TTF into a "b". Checksums are not verified, so I didn't bother fixing this. This is basically an invalid TTF file now since the cmap is required, but we only need it for tricking PDFBox.
So when working on a PDF page you want to enrich with OCR, you first load the font from trickster.pdf, and add it into the page/form resources. PDFBox will happily use the cidtogid tables and work and when you are done with the page/form your overwrite the font in the resources with the one from the template.pdf, so there is a proper TTF embedded. Ugly? Yes. But it's cheaper than to maintain a branch of PDFBox. PDFBox seems to simply copy the front from the open template/trickster PDFs into the altered PDF when saving, which is perfect. Otherwise one would have to do use clone util.
I am gonna work on this a bit more and show some code except later.
Gunnar
-----Ursprüngliche Nachricht-----
Von: Constantine Dokolas <cd...@gmail.com>
Gesendet: Freitag, 26. März 2021 11:38
An: users@pdfbox.apache.org
Betreff: Re: Empty cmap in TTF Files.
Hi, Gunnar,
Do you think this SO question
<https://stackoverflow.com/questions/49363954/using-arialmt-for-arabic-text-without-embedding-font-with-pdfbox>
is related? I'm the OP and the (admittedly somewhat niche) case for no-glyph (i.e. non-renderable) chars on a PDF is a "capability" that's been missing for me.
To give some context, at work I'm responsible for a library that, among other things, overlays OCRed text (from diverse sources) on images placed in PDF pages. There have been issues I've overcome (especially concerning Unicode), but "glyphless font" embedding is something that would really make a noticeable impact on PDF size. Most OCR software that produce PDFs from images do this in some way, Tesseract included.
I think PDFBox is a great library for reading and generating PDFs, and I'm seriously considering contributing as soon as possible. A big thanks to everyone working to make this project successful.
C.D.
--
There is a computer disease that anybody who works with computers knows about. It's a very serious disease and it interferes completely with the work. The trouble with computers is that you 'play' with them!
- Richard P. Feynman
On Thu, Mar 25, 2021 at 2:30 PM Gunnar Brand < Gunnar.Brand@interface-projects.de> wrote:
> Hi.
>
> The process is as follows:
> 1) For images: use the image
> For PDFs: render each page to 300 dpi (since optimized PDFs don't
> necessarily have a single big image), maybe even with text if text
> extraction returned gibberish (missing unicode mapping).
> 2) Use tesseract to OCR image/page with PDF and HOCR output. (for pages:
> create an imageless PDF). The HOCR is used for additional page layout
> information and word confidence values.
> 3) For images, use the HOCR to filter the PDF text stream and add
> layout information
> For PDFs, insert the tesseract PDF text stream into the orignal
> PDF's page (+add that glyphless font), use the HOCR to filter and add
> layout information.
>
> For step 3, I would like to use a normal PDPageContentStream to add
> the content instead of working with a raw stream. But that step fails
> since I cannot use the showText() method with a Font that has an empty cmap.
>
> I attached an empty tesseract PDF with the glyphless font. Appending
> text using the font to the single page in there will fail immediately
> with the exception due to the empty cmap. Adding the font to any other
> PDF and trying to show text using it will fail as well.
>
> I can probably get away with just creating/transfering the Tj commands
> raw, but I was wondering if the empty cmap behaviour is ok or would it
> be better to ignore empty cmaps (i.e. look for a non empty one first
> and return null if none can be found in TrueTypeFont.getUnicodeCmapImpl).
>
> Gunnar
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: Tilman Hausherr <TH...@t-online.de>
> Gesendet: Donnerstag, 25. März 2021 04:37
> An: users@pdfbox.apache.org
> Betreff: Re: Empty cmap in TTF Files.
>
> Am 24.03.2021 um 14:40 schrieb Gunnar Brand:
> > Hi.
> >
> > I am working on merging original PDFs and the PDF/HOCR output of
> Tesseract, as to create a searchable PDF. Transplanting the glyphless
> font used by tesseract was no problem, it doesn’t matter if I simply
> use the font in the original PDF or use cloneutil, when saving the
> file the font is embedded properly.
> >
> > The problem is when I show text using a content stream, I get a “No
> Glyph for …” exception. I traced this down to the glyphless font
> containing empty cmap tables. There is a CIDToGIDMap. Coincidentally
> PDFBOX-5103 just addressed this issue with a reverse mapping if the
> cmap is null. But the cmap is just empty and will return 0 for any
> character code, so this new feature will never work in this case.
> >
> > For testing I modified TrueTypeFont.getUnicodeCmapImpl(isStrict) so
> > that
> it ignores empty cmap subtables (even the fallback at the end of the
> method now being a loop). With this PDFBox will happily use the
> tesseract glyphless font. Now I lack the knowledge if empty cmaps make
> any sense at all and if they do I will simply write raw show text
> commands, but maybe it is something to consider?
> >
> > Gunnar
>
> I tried tesseract some time ago and it generates searchable PDFs out
> of the box, why not use that?
>
> Can you upload one of your files to a sharehoster so that I understand
> what this is about?
>
> Tilman
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> For additional commands, e-mail: users-help@pdfbox.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> For additional commands, e-mail: users-help@pdfbox.apache.org
Re: Empty cmap in TTF Files.
Posted by Constantine Dokolas <cd...@gmail.com>.
Hi, Gunnar,
Do you think this SO question
<https://stackoverflow.com/questions/49363954/using-arialmt-for-arabic-text-without-embedding-font-with-pdfbox>
is related? I'm the OP and the (admittedly somewhat niche) case for
no-glyph (i.e. non-renderable) chars on a PDF is a "capability" that's been
missing for me.
To give some context, at work I'm responsible for a library that, among
other things, overlays OCRed text (from diverse sources) on images placed
in PDF pages. There have been issues I've overcome (especially concerning
Unicode), but "glyphless font" embedding is something that would really
make a noticeable impact on PDF size. Most OCR software that produce PDFs
from images do this in some way, Tesseract included.
I think PDFBox is a great library for reading and generating PDFs, and I'm
seriously considering contributing as soon as possible. A big thanks to
everyone working to make this project successful.
C.D.
--
There is a computer disease that anybody who works with computers knows
about. It's a very serious disease and it interferes completely with the
work. The trouble with computers is that you 'play' with them!
- Richard P. Feynman
On Thu, Mar 25, 2021 at 2:30 PM Gunnar Brand <
Gunnar.Brand@interface-projects.de> wrote:
> Hi.
>
> The process is as follows:
> 1) For images: use the image
> For PDFs: render each page to 300 dpi (since optimized PDFs don't
> necessarily have a single big image), maybe even with text if text
> extraction returned gibberish (missing unicode mapping).
> 2) Use tesseract to OCR image/page with PDF and HOCR output. (for pages:
> create an imageless PDF). The HOCR is used for additional page layout
> information and word confidence values.
> 3) For images, use the HOCR to filter the PDF text stream and add layout
> information
> For PDFs, insert the tesseract PDF text stream into the orignal PDF's
> page (+add that glyphless font), use the HOCR to filter and add layout
> information.
>
> For step 3, I would like to use a normal PDPageContentStream to add the
> content instead of working with a raw stream. But that step fails since I
> cannot use the showText() method with a Font that has an empty cmap.
>
> I attached an empty tesseract PDF with the glyphless font. Appending text
> using the font to the single page in there will fail immediately with the
> exception due to the empty cmap. Adding the font to any other PDF and
> trying to show text using it will fail as well.
>
> I can probably get away with just creating/transfering the Tj commands
> raw, but I was wondering if the empty cmap behaviour is ok or would it be
> better to ignore empty cmaps (i.e. look for a non empty one first and
> return null if none can be found in TrueTypeFont.getUnicodeCmapImpl).
>
> Gunnar
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: Tilman Hausherr <TH...@t-online.de>
> Gesendet: Donnerstag, 25. März 2021 04:37
> An: users@pdfbox.apache.org
> Betreff: Re: Empty cmap in TTF Files.
>
> Am 24.03.2021 um 14:40 schrieb Gunnar Brand:
> > Hi.
> >
> > I am working on merging original PDFs and the PDF/HOCR output of
> Tesseract, as to create a searchable PDF. Transplanting the glyphless font
> used by tesseract was no problem, it doesn’t matter if I simply use the
> font in the original PDF or use cloneutil, when saving the file the font is
> embedded properly.
> >
> > The problem is when I show text using a content stream, I get a “No
> Glyph for …” exception. I traced this down to the glyphless font containing
> empty cmap tables. There is a CIDToGIDMap. Coincidentally PDFBOX-5103 just
> addressed this issue with a reverse mapping if the cmap is null. But the
> cmap is just empty and will return 0 for any character code, so this new
> feature will never work in this case.
> >
> > For testing I modified TrueTypeFont.getUnicodeCmapImpl(isStrict) so that
> it ignores empty cmap subtables (even the fallback at the end of the
> method now being a loop). With this PDFBox will happily use the tesseract
> glyphless font. Now I lack the knowledge if empty cmaps make any sense at
> all and if they do I will simply write raw show text commands, but maybe it
> is something to consider?
> >
> > Gunnar
>
> I tried tesseract some time ago and it generates searchable PDFs out of
> the box, why not use that?
>
> Can you upload one of your files to a sharehoster so that I understand
> what this is about?
>
> Tilman
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> For additional commands, e-mail: users-help@pdfbox.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> For additional commands, e-mail: users-help@pdfbox.apache.org
AW: Empty cmap in TTF Files.
Posted by Gunnar Brand <Gu...@interface-projects.de>.
Hi.
The process is as follows:
1) For images: use the image
For PDFs: render each page to 300 dpi (since optimized PDFs don't necessarily have a single big image), maybe even with text if text extraction returned gibberish (missing unicode mapping).
2) Use tesseract to OCR image/page with PDF and HOCR output. (for pages: create an imageless PDF). The HOCR is used for additional page layout information and word confidence values.
3) For images, use the HOCR to filter the PDF text stream and add layout information
For PDFs, insert the tesseract PDF text stream into the orignal PDF's page (+add that glyphless font), use the HOCR to filter and add layout information.
For step 3, I would like to use a normal PDPageContentStream to add the content instead of working with a raw stream. But that step fails since I cannot use the showText() method with a Font that has an empty cmap.
I attached an empty tesseract PDF with the glyphless font. Appending text using the font to the single page in there will fail immediately with the exception due to the empty cmap. Adding the font to any other PDF and trying to show text using it will fail as well.
I can probably get away with just creating/transfering the Tj commands raw, but I was wondering if the empty cmap behaviour is ok or would it be better to ignore empty cmaps (i.e. look for a non empty one first and return null if none can be found in TrueTypeFont.getUnicodeCmapImpl).
Gunnar
-----Ursprüngliche Nachricht-----
Von: Tilman Hausherr <TH...@t-online.de>
Gesendet: Donnerstag, 25. März 2021 04:37
An: users@pdfbox.apache.org
Betreff: Re: Empty cmap in TTF Files.
Am 24.03.2021 um 14:40 schrieb Gunnar Brand:
> Hi.
>
> I am working on merging original PDFs and the PDF/HOCR output of Tesseract, as to create a searchable PDF. Transplanting the glyphless font used by tesseract was no problem, it doesn’t matter if I simply use the font in the original PDF or use cloneutil, when saving the file the font is embedded properly.
>
> The problem is when I show text using a content stream, I get a “No Glyph for …” exception. I traced this down to the glyphless font containing empty cmap tables. There is a CIDToGIDMap. Coincidentally PDFBOX-5103 just addressed this issue with a reverse mapping if the cmap is null. But the cmap is just empty and will return 0 for any character code, so this new feature will never work in this case.
>
> For testing I modified TrueTypeFont.getUnicodeCmapImpl(isStrict) so that it ignores empty cmap subtables (even the fallback at the end of the method now being a loop). With this PDFBox will happily use the tesseract glyphless font. Now I lack the knowledge if empty cmaps make any sense at all and if they do I will simply write raw show text commands, but maybe it is something to consider?
>
> Gunnar
I tried tesseract some time ago and it generates searchable PDFs out of the box, why not use that?
Can you upload one of your files to a sharehoster so that I understand what this is about?
Tilman
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
For additional commands, e-mail: users-help@pdfbox.apache.org
Re: Empty cmap in TTF Files.
Posted by Tilman Hausherr <TH...@t-online.de>.
Am 24.03.2021 um 14:40 schrieb Gunnar Brand:
> Hi.
>
> I am working on merging original PDFs and the PDF/HOCR output of Tesseract, as to create a searchable PDF. Transplanting the glyphless font used by tesseract was no problem, it doesn’t matter if I simply use the font in the original PDF or use cloneutil, when saving the file the font is embedded properly.
>
> The problem is when I show text using a content stream, I get a “No Glyph for …” exception. I traced this down to the glyphless font containing empty cmap tables. There is a CIDToGIDMap. Coincidentally PDFBOX-5103 just addressed this issue with a reverse mapping if the cmap is null. But the cmap is just empty and will return 0 for any character code, so this new feature will never work in this case.
>
> For testing I modified TrueTypeFont.getUnicodeCmapImpl(isStrict) so that it ignores empty cmap subtables (even the fallback at the end of the method now being a loop). With this PDFBox will happily use the tesseract glyphless font. Now I lack the knowledge if empty cmaps make any sense at all and if they do I will simply write raw show text commands, but maybe it is something to consider?
>
> Gunnar
I tried tesseract some time ago and it generates searchable PDFs out of
the box, why not use that?
Can you upload one of your files to a sharehoster so that I understand
what this is about?
Tilman
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
For additional commands, e-mail: users-help@pdfbox.apache.org