You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pdfbox.apache.org by Kevin Day <ke...@trumpetinc.com> on 2021/11/02 19:06:01 UTC
Re: Guidance on heap space error when printing document
Sorry it took so long to respond - I had to get permission to share these.
I did test with 2.0.24 and it did not help.
Here are some files that cause the problem:
https://github.com/trumpetinc/boottest/blob/main/JrachvUniverse_H21.pdf
https://github.com/trumpetinc/boottest/blob/main/PP%20Opera%20Building_A53.pdf
Here is some quick code that demonstrates the problem (update for your
printer name and PDF path):
https://github.com/trumpetinc/boottest/blob/main/SimplifiedPdfBoxPdfPrinter.java
Maybe in TilingPaint#getImage(PageDrawer, PDTilingPattern, PDColorSpace,
PDColor, AffineTransform, Rectangle2D), the BufferedImage could
benefit from something similar to BigBufferedImage described in this post?
(
https://stackoverflow.com/questions/3854200/why-does-a-bufferedimage-require-so-much-memory-beyond-the-size-of-its-data-arra
). Or maybe there is something in the scaling logic?
On Thu, Oct 28, 2021 at 11:12 AM Tilman Hausherr <TH...@t-online.de>
wrote:
> Huge buffered images can happen in patterns if the pattern matrix and/or
> the current transformation matrix result in a huge scale.
>
> You could retry with 2.0.24, but I'm pessimistic. Please share the PDF
> (upload to a sharehoster).
>
> Tilman
>
> Am 28.10.2021 um 01:17 schrieb Kevin Day:
> > Using PDFBox 2.0.20
> >
> > We use the following to print the PDF:
> >
> > HashPrintRequestAttributeSet attributes = new
> > HashPrintRequestAttributeSet();
> >
> > try(PDDocument doc = PDDocument.load(pdf,
> > MemoryUsageSetting.setupMixed(5000000L))){
> > job.setPageable(new PDFPageable(doc));
> > job.print(attributes);
> > }
> >
> > This works really well for thousands of PDFs, but we've run into one
> > particular PDF that causes an OutOfMemoryException.
> >
> >
> > The problem PDF is rendered (does not contain bitmaps).
> >
> > I've provided a huge amount of heap (over 1GB now), and it is still
> failing.
> >
> > As near as I can tell from the stack trace (which I'll include below), it
> > seems like the problem is with creation of a huge buffered image.
> >
> > The physical page size in the PDF is 8.5 x 11".
> >
> > The PDFPrintable is configured as follows:
> >
> > Scaling is ACTUAL_SIZE
> > dpi is 0.0
> > subsamplingAllowed is false
> > renderingHints is null
> >
> >
> > Here is the stack trace:
> >
> > java.lang.OutOfMemoryError: Java heap space
> > at
> java.desktop/java.awt.image.DataBufferInt.<init>(DataBufferInt.java:75)
> > at java.desktop/java.awt.image.Raster.createPackedRaster(Raster.java:467)
> > at
> >
> java.desktop/java.awt.image.DirectColorModel.createCompatibleWritableRaster(DirectColorModel.java:1032)
> > at
> java.desktop/java.awt.image.BufferedImage.<init>(BufferedImage.java:333)
> > at org.apache.pdfbox.rendering.TilingPaint.getImage(TilingPaint.java:143)
> > at org.apache.pdfbox.rendering.TilingPaint.<init>(TilingPaint.java:103)
> > at
> >
> org.apache.pdfbox.rendering.TilingPaintFactory.create(TilingPaintFactory.java:60)
> > at org.apache.pdfbox.rendering.PageDrawer.getPaint(PageDrawer.java:351)
> > at
> >
> org.apache.pdfbox.rendering.PageDrawer.getNonStrokingPaint(PageDrawer.java:719)
> > at org.apache.pdfbox.rendering.PageDrawer.fillPath(PageDrawer.java:819)
> > at
> >
> org.apache.pdfbox.contentstream.operator.graphics.FillEvenOddRule.process(FillEvenOddRule.java:37)
> > at
> >
> org.apache.pdfbox.contentstream.PDFStreamEngine.processOperator(PDFStreamEngine.java:932)
> > at
> >
> org.apache.pdfbox.contentstream.PDFStreamEngine.processStreamOperators(PDFStreamEngine.java:510)
> > at
> >
> org.apache.pdfbox.contentstream.PDFStreamEngine.processStream(PDFStreamEngine.java:484)
> > at
> >
> org.apache.pdfbox.contentstream.PDFStreamEngine.showForm(PDFStreamEngine.java:187)
> > at org.apache.pdfbox.rendering.PageDrawer.showForm(PageDrawer.java:1462)
> > at
> >
> org.apache.pdfbox.contentstream.operator.graphics.DrawObject.process(DrawObject.java:86)
> > at
> >
> org.apache.pdfbox.contentstream.PDFStreamEngine.processOperator(PDFStreamEngine.java:932)
> > at
> >
> org.apache.pdfbox.contentstream.PDFStreamEngine.processStreamOperators(PDFStreamEngine.java:510)
> > at
> >
> org.apache.pdfbox.contentstream.PDFStreamEngine.processStream(PDFStreamEngine.java:484)
> > at
> >
> org.apache.pdfbox.contentstream.PDFStreamEngine.processPage(PDFStreamEngine.java:156)
> > at org.apache.pdfbox.rendering.PageDrawer.drawPage(PageDrawer.java:271)
> > at
> >
> org.apache.pdfbox.rendering.PDFRenderer.renderPageToGraphics(PDFRenderer.java:431)
> > at org.apache.pdfbox.printing.PDFPrintable.print(PDFPrintable.java:246)
> > at
> >
> java.desktop/sun.print.RasterPrinterJob.printPage(RasterPrinterJob.java:2430)
> > at
> java.desktop/sun.print.RasterPrinterJob.print(RasterPrinterJob.java:1654)
> >
> >
> >
> > Next, I tried setting subsampling to true - this made no difference.
> >
> >
> > Next, I tried hard coding the dpi to 300f - this works with no problems.
> >
> >
> > Next, I tried hard coding the dpi to 600f - this does not throw the heap
> > space error - but with strange results:
> > - I see "java.lang.OutOfMemoryError" on sys.err - it does NOT throw an
> > exception
> > - The pages come off the printer as blank pages
> >
> >
> > Does anyone have any suggestions on how we can print using the printer's
> > full resolution?
> >
> > Thanks!
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> For additional commands, e-mail: users-help@pdfbox.apache.org
>
>
Re: Guidance on heap space error when printing document
Posted by Tilman Hausherr <TH...@t-online.de>.
Am 07.11.2021 um 15:53 schrieb Tilman Hausherr:
> I tried it and it works pretty fast. The memory leak is still there
> but it comes later.
The memory leak was in PDFDebugger and it's now mostly fixed
https://issues.apache.org/jira/browse/PDFBOX-5320
So I've worked in the wrong direction.
I've now tried the two cache changes. Printing to PDF works as slow as
before. Also my assumption that it would be at 1200 dpi was wrong, it
works at scale 1 = 72 dpi, maybe because the printer uses the same unit
as PDF.
Tilman
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
For additional commands, e-mail: users-help@pdfbox.apache.org
Re: Guidance on heap space error when printing document
Posted by Kevin Day <ke...@trumpetinc.com>.
PS - PdfBox already has the sun.* stuff in it - see IOUtils::unmapper()
Kevin Day
*trumpet**p| *480.961.6003 x1002
*e| *kevin@trumpetinc.com
www.trumpetinc.com <http://trumpetinc.com/>
LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
<http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
Proud to be Great Place To Work
<https://www.greatplacetowork.com/certified-company/7012667> certified
since 2019
On Sun, Nov 7, 2021 at 8:09 AM Kevin Day <ke...@trumpetinc.com> wrote:
> ok - I think that maybe there are two things going on, then. The first is
> that we are allocating a single buffer that is bigger than the default heap
> (700MB > 500MB). I really think that the only two solutions are to move
> this off the heap or reduce the size of the allocation.
>
> Then the second issue is maybe a memory leak (I have not been able to see
> this because of the first problem unless I set my max heap to a huge number
> - and that makes memory leak analysis very difficult). I am going to clone
> the PdfBox git repo now and swap that class in then do a little bit of
> profiling with jvisualvm to see if I can help find the memory leak.
>
> Question: The Git repo for PdfBox says that it is a mirror. If I do wind
> up creating a pull request against this repo, will you be able to accept it?
>
>
> FYI - the use of sun.* classes is just to work around a long standing
> native resource leak bug in the Windows MappedByteBuffer implementation -
> On Windows, MappedByteBuffer does not release the allocated native resource
> until finalize() is called, and because the MappedByteBuffer itself does
> not take up much heap space, it can stay on the heap for quite a long time
> even after all references are gone. So the sun.* reference is just to
> force the release of those native resources. I used the same technique
> when I created the high throughput IO layer for iText. Instead of
> referencing the package directly, we can use reflection (and only use it
> for winXXX platform) - this shouldn't be a big problem.
>
> K
>
> Kevin Day
>
> *trumpet**p| *480.961.6003 x1002
> *e| *kevin@trumpetinc.com
> www.trumpetinc.com <http://trumpetinc.com/>
> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
> <http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
>
> Proud to be Great Place To Work
> <https://www.greatplacetowork.com/certified-company/7012667> certified
> since 2019
>
>
> On Sun, Nov 7, 2021 at 7:53 AM Tilman Hausherr <TH...@t-online.de>
> wrote:
>
>> I tried it and it works pretty fast. The memory leak is still there but
>> it comes later.
>>
>> Other drawbacks arethat it doesn't clean up after itself fast enough,
>> and it uses a sun.* class.
>>
>> Tilman
>>
>> Am 07.11.2021 um 14:31 schrieb Tilman Hausherr:
>> > Am 05.11.2021 um 22:09 schrieb Kevin Day:
>> >> ok - so should we be clamping the xstep in some way? Or at this
>> >> depth of
>> >> the algorithm do we not have enough context to actually tell that it
>> >> will
>> >> be outside the page/clipping region?
>> >
>> > Yes + yes, that is the problem. The context would have to include the
>> > region but also the current transformation matrix. Which could change
>> > despite using the same pattern. So it's tricky.
>> >
>> >>
>> >>
>> >> I'm beginning to think that the BigBufferedImage might be the right
>> >> solution... This is very inefficient, but honestly, the PDF is not
>> >> exactly
>> >> well formed, so if these particular files wind up being slower to
>> render
>> >> b/c they have to swap image content to disk, I think that is OK...
>> >
>> > Maybe... the good thing is that we know when the image will be "too
>> > big". The license is OK:
>> >
>> > https://www.apache.org/legal/resolved.html
>> >
>> > But I'm still wondering whether we have a memory leak. If we have,
>> > then it should be fixed.
>> >
>> > Tilman
>> >
>> >
>> >>
>> >> - K
>> >>
>> >>
>> >> Kevin Day
>> >>
>> >> *trumpet**p| *480.961.6003 x1002
>> >> *e| *kevin@trumpetinc.com
>> >> www.trumpetinc.com <http://trumpetinc.com/>
>> >> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
>> >> <http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc
>> >
>> >>
>> >> Proud to be Great Place To Work
>> >> <https://www.greatplacetowork.com/certified-company/7012667> Certified
>> >> 2020-2021
>> >>
>> >>
>> >> On Fri, Nov 5, 2021 at 12:47 PM Tilman Hausherr <THausherr@t-online.de
>> >
>> >> wrote:
>> >>
>> >>> Am 05.11.2021 um 20:38 schrieb Kevin Day:
>> >>>> I do have a bit of experience with memory leaks (side effect of
>> >>>> writing
>> >>>> high performance java code, unfortunately!)
>> >>>>
>> >>>>
>> >>>> I am pretty sure that this is not a memory leak, though. If this
>> >>>> was a
>> >>>> memory leak, we would see calls happening multiple times before
>> >>>> failure.
>> >>>>
>> >>>> The failure happens on the very first call to create the buffered
>> >>>> image:
>> >>>>
>> >>>> BufferedImage image = new BufferedImage(rasterWidth,
>> >>> rasterHeight,
>> >>>> BufferedImage.TYPE_INT_ARGB);
>> >>>>
>> >>>> Nothing is even getting put into the weakcache.
>> >>>>
>> >>> That is done in TilingPaintFactory. The cache was needed for some
>> files
>> >>> that have used the pattern several times, to avoid recreating the
>> image
>> >>> for the TexturePaint.
>> >>>
>> >>>
>> >>>> I think the core issue is that the raster data for the image is
>> >>>> just too
>> >>>> big to store on the heap (without the heap being massive). It's
>> >>>> 13152x13152x4 bytes (there is an alpha channel on this as well) -
>> >>>> that's
>> >>>> almost 700MB - just for this one pattern.
>> >>>>
>> >>>>
>> >>>> Tilman wrote: "That
>> >>>> pattern has an XStep and YStep of 23438 although the image is 2148 x
>> >>>> 440. Because of a matrix scale of 0.0673396 the image pattern size is
>> >>>> 1578 x 1578 at 72 dpi. So at 1200 dpi the size would be about 26300 x
>> >>>> 26300"
>> >>>>
>> >>>> I am fairly familiar with PDF (I contributed the parsing library to
>> >>>> iText
>> >>>> back in the day), but I'm not very familiar with rendering tile
>> >>>> patterns,
>> >>>> so the above is not intuitive to me (yet). Is there some chance that
>> >>>> PdfBox is doing a lot more work than is necessary for this particular
>> >>> PDF?
>> >>>> Like is the matrix scale just absurdly bad in this file? For an
>> image
>> >>> that
>> >>>> is originally 2148x440, it seems like requiring a 700MB raster
>> >>>> should not
>> >>>> be necessary - but I do NOT have nearly enough experience with this
>> >>>> area
>> >>> of
>> >>>> PDF...
>> >>> The ridiculous thing isn't the matrix, it's the XStep and YStep. This
>> >>> goes well outside of the page. The 0.067 matrix scaling makes it
>> >>> less bad.
>> >>>
>> >>> A weakness in PDFBox is that we don't readjust the tile image, or
>> >>> use an
>> >>> alternative painting method that doesn't use TexturePaint when the
>> >>> image
>> >>> would be painted only once.
>> >>>
>> >>> Tilman
>> >>>
>> >>>
>> >>>
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>> - Kevin
>> >>>>
>> >>>>
>> >>>> Kevin Day
>> >>>>
>> >>>> *trumpet**p| *480.961.6003 x1002
>> >>>> *e| *kevin@trumpetinc.com
>> >>>> www.trumpetinc.com <http://trumpetinc.com/>
>> >>>> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet
>> Blog
>> >>>> <http://trumpetinc.com/blog/>| Twitter
>> >>>> <https://twitter.com/trumpetinc>
>> >>>>
>> >>>> Proud to be Great Place To Work
>> >>>> <https://www.greatplacetowork.com/certified-company/7012667>
>> Certified
>> >>>> 2020-2021
>> >>>>
>> >>>>
>> >>>> On Fri, Nov 5, 2021 at 11:27 AM Tilman Hausherr
>> >>>> <TH...@t-online.de>
>> >>>> wrote:
>> >>>>
>> >>>>> If you're experienced with memory leaks then it would be nice if you
>> >>>>> could search.
>> >>>>>
>> >>>>> Things I tried that didn't help:
>> >>>>> - calling graphics.setPaint(null) after operations (to "lose"
>> >>>>> TilingPaint objects)
>> >>>>> - disabling the (weak) cache of TilingPaint objects
>> >>>>> - adding finalize to see if the TilingPaint class gets finalized
>> >>>>> (yes)
>> >>>>> - adding finalize to see if the HighResolutionImageIcon class gets
>> >>>>> finalized (yes)
>> >>>>>
>> >>>>> Tilman
>> >>>>>
>> >>>
>> >>> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
>> >>> For additional commands, e-mail: users-help@pdfbox.apache.org
>> >>>
>> >>>
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
>> > For additional commands, e-mail: users-help@pdfbox.apache.org
>> >
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
>> For additional commands, e-mail: users-help@pdfbox.apache.org
>>
>>
Re: Guidance on heap space error when printing document
Posted by Kevin Day <ke...@trumpetinc.com>.
Fair enough. All of the proposed changes are very self contained.
Feel free to reach out directly if you have any questions when you do get
time. I'm happy to be a sounding board of you want to retry the scaling
approach (instead of hard cropping), or want to discuss raster generation
during print jobs to improve performance during banded printing.
I think that I can resolve my immediate issue with the maxedge system
property (though I agree that it is a very ugly hack 😀).
On Mon, Nov 8, 2021, 11:38 AM Tilman Hausherr <TH...@t-online.de> wrote:
> Am 08.11.2021 um 14:28 schrieb Kevin Day:
> > Will you accept my code changes to have different maxedge values for 32
> bit
> > vs 64 bit?
>
> It's fine to have different initializations. We have code that works
> differently depending on the JDK version.
>
> But we're not there yet. What I want to do is to separate this into
> smaller pieces so I can understand them easier (when I have more time)
>
> Tilman
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> For additional commands, e-mail: users-help@pdfbox.apache.org
>
>
Re: Guidance on heap space error when printing document
Posted by Tilman Hausherr <TH...@t-online.de>.
Am 08.11.2021 um 14:28 schrieb Kevin Day:
> Will you accept my code changes to have different maxedge values for 32 bit
> vs 64 bit?
It's fine to have different initializations. We have code that works
differently depending on the JDK version.
But we're not there yet. What I want to do is to separate this into
smaller pieces so I can understand them easier (when I have more time)
Tilman
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
For additional commands, e-mail: users-help@pdfbox.apache.org
Re: Guidance on heap space error when printing document
Posted by Kevin Day <ke...@trumpetinc.com>.
Another thought:
As I've worked with the code, I think that maybe one possible issue is
that TilingPaint is creating its image during construction.
Would it be better if TilingPaint constructed an image on demand
in createContext()? That way you would have full information about the
clipping region. Would that work better?? Maybe it would not be necessary
to create this huge image at all if we did it that way?
- Kevin
Kevin Day
*trumpet**p| *480.961.6003 x1002
*e| *kevin@trumpetinc.com
www.trumpetinc.com <http://trumpetinc.com/>
LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
<http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
Proud to be Great Place To Work
<https://www.greatplacetowork.com/certified-company/7012667> certified
since 2019
On Sun, Nov 7, 2021 at 11:14 AM Kevin Day <ke...@trumpetinc.com> wrote:
> oh - I see a major problem (it's a bit of a reverse memory leak, actually)
> that is involved in your disk space issue (it also results in really poor
> performance for image tiling operations):
>
>
> The cache in TilingPaintFactory is based on WeakHashMap. WeakHashMap is
> designed around weak referenced *keys*, and the TilingPaintFactory.create()
> method is constructing a new key every call. This means that entries are
> not going to be in the cache for very long (especially with a generational
> garbage collector). In fact, in my testing, the cache is always empty.
> This results in a new BigBufferedImage (with a new disk cache) being
> created every time to the tile is painted.
>
> So I think at least some of the problem is that the cache is not
> implemented properly.
>
> If you are OK using an additional Apache dependency, I think that the
> cache should be changed to this (with strong keys and weak values):
> https://commons.apache.org/proper/commons-collections/apidocs/org/apache/commons/collections4/map/ReferenceMap.html
>
>
>
> Next, the print job is running in stripes - so a single page gets rendered
> many times. Right now, a new PageDrawer is being constructed for each
> stripe in PDFRenderer.renderImage(...), which is resulting in a huge number
> of BufferedImages being created. Adding a cache for PageDrawer instances
> addresses this.
>
>
> Next, the key on the TilingPant cache in TilingPaintFactory is based on
> the xform affine transformation. It needs this (I think) to handle scaling
> - but with stripes, the offset is changing but the scaling isn't. So I
> think that the TilingPaintParameter.hashCode() and equals() needs to be
> adjusted to only consider scale - not translation.
>
>
>
> When I make these changes, it dramatically reduces the disk usage.
>
>
> I have submitted a pull request with these changes for your evaluation
> (this is NOT a complete fix for the problem - it still uses
> BigBufferedImage): https://github.com/apache/pdfbox/pull/135
>
> - K
>
> Kevin Day
>
> *trumpet**p| *480.961.6003 x1002
> *e| *kevin@trumpetinc.com
> www.trumpetinc.com <http://trumpetinc.com/>
> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
> <http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
>
> Proud to be Great Place To Work
> <https://www.greatplacetowork.com/certified-company/7012667> certified
> since 2019
>
>
> On Sun, Nov 7, 2021 at 9:36 AM Kevin Day <ke...@trumpetinc.com> wrote:
>
>> Ah yes - the file deletion in BigBufferedImage only happens during
>> finalize() (or with a shutdown hook). That's why you are blowing out your
>> disk space. We will have the same problem even if we use ScratchFile.
>>
>> This means that we have to talk about disposing the BufferedImage
>> instances - but BufferedImage does not have a dispose() method on it.
>>
>> If we add dispose on the buffered image, now we also need dispose() on
>> the Paint object, etc.... It gets very ugly...
>>
>> - K
>>
>> Kevin Day
>>
>> *trumpet**p| *480.961.6003 x1002
>> *e| *kevin@trumpetinc.com
>> www.trumpetinc.com <http://trumpetinc.com/>
>> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
>> <http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
>>
>> Proud to be Great Place To Work
>> <https://www.greatplacetowork.com/certified-company/7012667> certified
>> since 2019
>>
>>
>> On Sun, Nov 7, 2021 at 8:55 AM Kevin Day <ke...@trumpetinc.com> wrote:
>>
>>> There is another issue with BigBufferedImage - it does not work well on
>>> 32 bit JVMs. The problem is that it creates a single MappedByteBuffer for
>>> the entire size of a channel's data (for the problem file, this means that
>>> each buffer has 172,975,104 bytes).
>>>
>>> I think it would be better to create a DataBuffer implementation that
>>> uses org.apache.pdfbox.io.ScratchFile and create a PdfBoxBufferedImage...
>>>
>>> - K
>>>
>>> Kevin Day
>>>
>>> *trumpet**p| *480.961.6003 x1002
>>> *e| *kevin@trumpetinc.com
>>> www.trumpetinc.com <http://trumpetinc.com/>
>>> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
>>> <http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
>>>
>>> Proud to be Great Place To Work
>>> <https://www.greatplacetowork.com/certified-company/7012667> certified
>>> since 2019
>>>
>>>
>>> On Sun, Nov 7, 2021 at 8:36 AM Tilman Hausherr <TH...@t-online.de>
>>> wrote:
>>>
>>>> Am 07.11.2021 um 16:09 schrieb Kevin Day:
>>>> > ok - I think that maybe there are two things going on, then. The
>>>> first is
>>>> > that we are allocating a single buffer that is bigger than the
>>>> default heap
>>>> > (700MB > 500MB). I really think that the only two solutions are to
>>>> move
>>>> > this off the heap or reduce the size of the allocation.
>>>> >
>>>> > Then the second issue is maybe a memory leak (I have not been able to
>>>> see
>>>> > this because of the first problem unless I set my max heap to a huge
>>>> number
>>>> > - and that makes memory leak analysis very difficult). I am going to
>>>> clone
>>>> > the PdfBox git repo now and swap that class in then do a little bit of
>>>> > profiling with jvisualvm to see if I can help find the memory leak.
>>>> >
>>>> > Question: The Git repo for PdfBox says that it is a mirror. If I do
>>>> wind
>>>> > up creating a pull request against this repo, will you be able to
>>>> accept it?
>>>>
>>>> Not directly but we can create a .diff / .patch file from the PR.
>>>>
>>>> In the meantime I ran my rendering regression checks which renders 1000
>>>> files. The hard disk C: went full after using 260 GB :-(
>>>>
>>>> Tilman
>>>>
>>>>
>>>> >
>>>> >
>>>> > FYI - the use of sun.* classes is just to work around a long standing
>>>> > native resource leak bug in the Windows MappedByteBuffer
>>>> implementation -
>>>> > On Windows, MappedByteBuffer does not release the allocated native
>>>> resource
>>>> > until finalize() is called, and because the MappedByteBuffer itself
>>>> does
>>>> > not take up much heap space, it can stay on the heap for quite a long
>>>> time
>>>> > even after all references are gone. So the sun.* reference is just to
>>>> > force the release of those native resources. I used the same
>>>> technique
>>>> > when I created the high throughput IO layer for iText. Instead of
>>>> > referencing the package directly, we can use reflection (and only use
>>>> it
>>>> > for winXXX platform) - this shouldn't be a big problem.
>>>> >
>>>> > K
>>>> >
>>>> > Kevin Day
>>>> >
>>>> > *trumpet**p| *480.961.6003 x1002
>>>> > *e| *kevin@trumpetinc.com
>>>> > www.trumpetinc.com <http://trumpetinc.com/>
>>>> > LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet
>>>> Blog
>>>> > <http://trumpetinc.com/blog/>| Twitter <
>>>> https://twitter.com/trumpetinc>
>>>> >
>>>> > Proud to be Great Place To Work
>>>> > <https://www.greatplacetowork.com/certified-company/7012667>
>>>> certified
>>>> > since 2019
>>>> >
>>>> >
>>>> > On Sun, Nov 7, 2021 at 7:53 AM Tilman Hausherr <THausherr@t-online.de
>>>> >
>>>> > wrote:
>>>> >
>>>> >> I tried it and it works pretty fast. The memory leak is still there
>>>> but
>>>> >> it comes later.
>>>> >>
>>>> >> Other drawbacks arethat it doesn't clean up after itself fast enough,
>>>> >> and it uses a sun.* class.
>>>> >>
>>>> >> Tilman
>>>> >>
>>>> >> Am 07.11.2021 um 14:31 schrieb Tilman Hausherr:
>>>> >>> Am 05.11.2021 um 22:09 schrieb Kevin Day:
>>>> >>>> ok - so should we be clamping the xstep in some way? Or at this
>>>> >>>> depth of
>>>> >>>> the algorithm do we not have enough context to actually tell that
>>>> it
>>>> >>>> will
>>>> >>>> be outside the page/clipping region?
>>>> >>> Yes + yes, that is the problem. The context would have to include
>>>> the
>>>> >>> region but also the current transformation matrix. Which could
>>>> change
>>>> >>> despite using the same pattern. So it's tricky.
>>>> >>>
>>>> >>>>
>>>> >>>> I'm beginning to think that the BigBufferedImage might be the right
>>>> >>>> solution... This is very inefficient, but honestly, the PDF is not
>>>> >>>> exactly
>>>> >>>> well formed, so if these particular files wind up being slower to
>>>> render
>>>> >>>> b/c they have to swap image content to disk, I think that is OK...
>>>> >>> Maybe... the good thing is that we know when the image will be "too
>>>> >>> big". The license is OK:
>>>> >>>
>>>> >>> https://www.apache.org/legal/resolved.html
>>>> >>>
>>>> >>> But I'm still wondering whether we have a memory leak. If we have,
>>>> >>> then it should be fixed.
>>>> >>>
>>>> >>> Tilman
>>>> >>>
>>>> >>>
>>>> >>>> - K
>>>> >>>>
>>>> >>>>
>>>> >>>> Kevin Day
>>>> >>>>
>>>> >>>> *trumpet**p| *480.961.6003 x1002
>>>> >>>> *e| *kevin@trumpetinc.com
>>>> >>>> www.trumpetinc.com <http://trumpetinc.com/>
>>>> >>>> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet
>>>> Blog
>>>> >>>> <http://trumpetinc.com/blog/>| Twitter <
>>>> https://twitter.com/trumpetinc>
>>>> >>>>
>>>> >>>> Proud to be Great Place To Work
>>>> >>>> <https://www.greatplacetowork.com/certified-company/7012667>
>>>> Certified
>>>> >>>> 2020-2021
>>>> >>>>
>>>> >>>>
>>>> >>>> On Fri, Nov 5, 2021 at 12:47 PM Tilman Hausherr <
>>>> THausherr@t-online.de>
>>>> >>>> wrote:
>>>> >>>>
>>>> >>>>> Am 05.11.2021 um 20:38 schrieb Kevin Day:
>>>> >>>>>> I do have a bit of experience with memory leaks (side effect of
>>>> >>>>>> writing
>>>> >>>>>> high performance java code, unfortunately!)
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> I am pretty sure that this is not a memory leak, though. If this
>>>> >>>>>> was a
>>>> >>>>>> memory leak, we would see calls happening multiple times before
>>>> >>>>>> failure.
>>>> >>>>>>
>>>> >>>>>> The failure happens on the very first call to create the buffered
>>>> >>>>>> image:
>>>> >>>>>>
>>>> >>>>>> BufferedImage image = new BufferedImage(rasterWidth,
>>>> >>>>> rasterHeight,
>>>> >>>>>> BufferedImage.TYPE_INT_ARGB);
>>>> >>>>>>
>>>> >>>>>> Nothing is even getting put into the weakcache.
>>>> >>>>>>
>>>> >>>>> That is done in TilingPaintFactory. The cache was needed for some
>>>> files
>>>> >>>>> that have used the pattern several times, to avoid recreating the
>>>> image
>>>> >>>>> for the TexturePaint.
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>> I think the core issue is that the raster data for the image is
>>>> >>>>>> just too
>>>> >>>>>> big to store on the heap (without the heap being massive). It's
>>>> >>>>>> 13152x13152x4 bytes (there is an alpha channel on this as well) -
>>>> >>>>>> that's
>>>> >>>>>> almost 700MB - just for this one pattern.
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> Tilman wrote: "That
>>>> >>>>>> pattern has an XStep and YStep of 23438 although the image is
>>>> 2148 x
>>>> >>>>>> 440. Because of a matrix scale of 0.0673396 the image pattern
>>>> size is
>>>> >>>>>> 1578 x 1578 at 72 dpi. So at 1200 dpi the size would be about
>>>> 26300 x
>>>> >>>>>> 26300"
>>>> >>>>>>
>>>> >>>>>> I am fairly familiar with PDF (I contributed the parsing library
>>>> to
>>>> >>>>>> iText
>>>> >>>>>> back in the day), but I'm not very familiar with rendering tile
>>>> >>>>>> patterns,
>>>> >>>>>> so the above is not intuitive to me (yet). Is there some chance
>>>> that
>>>> >>>>>> PdfBox is doing a lot more work than is necessary for this
>>>> particular
>>>> >>>>> PDF?
>>>> >>>>>> Like is the matrix scale just absurdly bad in this file? For an
>>>> image
>>>> >>>>> that
>>>> >>>>>> is originally 2148x440, it seems like requiring a 700MB raster
>>>> >>>>>> should not
>>>> >>>>>> be necessary - but I do NOT have nearly enough experience with
>>>> this
>>>> >>>>>> area
>>>> >>>>> of
>>>> >>>>>> PDF...
>>>> >>>>> The ridiculous thing isn't the matrix, it's the XStep and YStep.
>>>> This
>>>> >>>>> goes well outside of the page. The 0.067 matrix scaling makes it
>>>> >>>>> less bad.
>>>> >>>>>
>>>> >>>>> A weakness in PDFBox is that we don't readjust the tile image, or
>>>> >>>>> use an
>>>> >>>>> alternative painting method that doesn't use TexturePaint when the
>>>> >>>>> image
>>>> >>>>> would be painted only once.
>>>> >>>>>
>>>> >>>>> Tilman
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>> Thanks,
>>>> >>>>>>
>>>> >>>>>> - Kevin
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> Kevin Day
>>>> >>>>>>
>>>> >>>>>> *trumpet**p| *480.961.6003 x1002
>>>> >>>>>> *e| *kevin@trumpetinc.com
>>>> >>>>>> www.trumpetinc.com <http://trumpetinc.com/>
>>>> >>>>>> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>|
>>>> Trumpet
>>>> >> Blog
>>>> >>>>>> <http://trumpetinc.com/blog/>| Twitter
>>>> >>>>>> <https://twitter.com/trumpetinc>
>>>> >>>>>>
>>>> >>>>>> Proud to be Great Place To Work
>>>> >>>>>> <https://www.greatplacetowork.com/certified-company/7012667>
>>>> >> Certified
>>>> >>>>>> 2020-2021
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> On Fri, Nov 5, 2021 at 11:27 AM Tilman Hausherr
>>>> >>>>>> <TH...@t-online.de>
>>>> >>>>>> wrote:
>>>> >>>>>>
>>>> >>>>>>> If you're experienced with memory leaks then it would be nice
>>>> if you
>>>> >>>>>>> could search.
>>>> >>>>>>>
>>>> >>>>>>> Things I tried that didn't help:
>>>> >>>>>>> - calling graphics.setPaint(null) after operations (to "lose"
>>>> >>>>>>> TilingPaint objects)
>>>> >>>>>>> - disabling the (weak) cache of TilingPaint objects
>>>> >>>>>>> - adding finalize to see if the TilingPaint class gets finalized
>>>> >>>>>>> (yes)
>>>> >>>>>>> - adding finalize to see if the HighResolutionImageIcon class
>>>> gets
>>>> >>>>>>> finalized (yes)
>>>> >>>>>>>
>>>> >>>>>>> Tilman
>>>> >>>>>>>
>>>> >>>>>
>>>> ---------------------------------------------------------------------
>>>> >>>>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
>>>> >>>>> For additional commands, e-mail: users-help@pdfbox.apache.org
>>>> >>>>>
>>>> >>>>>
>>>> >>>
>>>> >>>
>>>> ---------------------------------------------------------------------
>>>> >>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
>>>> >>> For additional commands, e-mail: users-help@pdfbox.apache.org
>>>> >>>
>>>> >>
>>>> >> ---------------------------------------------------------------------
>>>> >> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
>>>> >> For additional commands, e-mail: users-help@pdfbox.apache.org
>>>> >>
>>>> >>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
>>>> For additional commands, e-mail: users-help@pdfbox.apache.org
>>>>
>>>>
Re: Guidance on heap space error when printing document
Posted by Kevin Day <ke...@trumpetinc.com>.
Will you accept my code changes to have different maxedge values for 32 bit
vs 64 bit?
On Mon, Nov 8, 2021, 12:51 AM Tilman Hausherr <TH...@t-online.de> wrote:
> Am 08.11.2021 um 06:38 schrieb Kevin Day:
> > I just realized what MAXEDGE does.
> >
> > You are basically saying that the raster can't be bigger than MAXEDGE x
> > MAXEDGE, and if it is, then you clamp the dimensions.
> >
> > But - are you scaling the bitmap? Or are you just cropping it? If you
> are
> > just cropping it, it seems like a better approach would be to adjust the
> > tiling matrix by the amount of the scaling required to prevent the raster
> > from being too big.
>
> It crops. It is of course sometimes a terrible solution. I did try
> scaling but wasn't successful but maybe you will.
>
> Tilman
>
>
> >
> >
> >
> > I just committed more changes to my fork:
> >
> > 1. Refined MAXEDGE clamping to scale both dimensions instead of clamping
> > the long dimension (I think this is a better way to do this - but it
> feels
> > like we should also be scaling the tiling matrix - see above)
> > 2. Added default MAXEDGE values that depend on the bitness of the JVM.
> I
> > found that MAXEDGE = 500 worked on 32 bit JVMs (but 650 does *not* work).
> > So setting MAXEDGE to 500 for 32 bit JVMs and 3000 for 64 bit JVMs seems
> > like a pretty decent compromise.
> >
> >
> > I look forward to your thoughts about whether MAXEDGE should produce
> > scaling of the tilematrix (or some other matrix). Hopefully that will be
> > an easy thing for you to comment on.
> >
> > - K
> >
> > Kevin Day
> >
> > *trumpet**p| *480.961.6003 x1002
> > *e| *kevin@trumpetinc.com
> > www.trumpetinc.com <http://trumpetinc.com/>
> > LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
> > <http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
> >
> > Proud to be Great Place To Work
> > <https://www.greatplacetowork.com/certified-company/7012667> certified
> > since 2019
> >
> >
> > On Sun, Nov 7, 2021 at 10:03 PM Kevin Day <ke...@trumpetinc.com> wrote:
> >
> >> Test procedure:
> >>
> >> Open PdfBox Debugger, load PDF, capture render time from label in lower
> >> left corner of window.
> >>
> >>
> >> With PdfBox 2.0.17 - this PDF renders in about 4.8 seconds (but is
> missing
> >> the colored pattern).
> >>
> >> With PdfBox 3 baseline (no code changes) - this PDF renders in 16.2
> >> seconds seconds.
> >>
> >> With PdfBox 3 caching changes only (no BigBufferedImage) - this PDF
> >> renders in 14.7 seconds
> >>
> >>
> >> So I have not done any damage, and I've sped it up a little bit.
> >>
> >>
> >> But that's only with a single render. Because printing can involve
> >> rendering many times for banded printing (on my test device, I am seeing
> >> well over 100 bands when printing this sample file), the page level
> caching
> >> should be substantial (the rendering on your test PDF is so slow that it
> >> almost can't be printed) - it is certainly a huge improvement for my
> test
> >> document.
> >>
> >>
> >> So I think my improvements to the caching (both the tile cache and
> adding
> >> caching at the page level) are valid.
> >>
> >> - K
> >>
> >> Kevin Day
> >>
> >> *trumpet**p| *480.961.6003 x1002
> >> *e| *kevin@trumpetinc.com
> >> www.trumpetinc.com <http://trumpetinc.com/>
> >> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
> >> <http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc
> >
> >>
> >> Proud to be Great Place To Work
> >> <https://www.greatplacetowork.com/certified-company/7012667> certified
> >> since 2019
> >>
> >>
> >> On Sun, Nov 7, 2021 at 8:34 PM Tilman Hausherr <TH...@t-online.de>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> I don't have much time right now, but please test your change with the
> >>> file from
> >>>
> >>> https://issues.apache.org/jira/browse/PDFBOX-3688
> >>>
> >>> Tilman
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> >>> For additional commands, e-mail: users-help@pdfbox.apache.org
> >>>
> >>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> For additional commands, e-mail: users-help@pdfbox.apache.org
>
>
Re: Guidance on heap space error when printing document
Posted by Tilman Hausherr <TH...@t-online.de>.
Am 08.11.2021 um 06:38 schrieb Kevin Day:
> I just realized what MAXEDGE does.
>
> You are basically saying that the raster can't be bigger than MAXEDGE x
> MAXEDGE, and if it is, then you clamp the dimensions.
>
> But - are you scaling the bitmap? Or are you just cropping it? If you are
> just cropping it, it seems like a better approach would be to adjust the
> tiling matrix by the amount of the scaling required to prevent the raster
> from being too big.
It crops. It is of course sometimes a terrible solution. I did try
scaling but wasn't successful but maybe you will.
Tilman
>
>
>
> I just committed more changes to my fork:
>
> 1. Refined MAXEDGE clamping to scale both dimensions instead of clamping
> the long dimension (I think this is a better way to do this - but it feels
> like we should also be scaling the tiling matrix - see above)
> 2. Added default MAXEDGE values that depend on the bitness of the JVM. I
> found that MAXEDGE = 500 worked on 32 bit JVMs (but 650 does *not* work).
> So setting MAXEDGE to 500 for 32 bit JVMs and 3000 for 64 bit JVMs seems
> like a pretty decent compromise.
>
>
> I look forward to your thoughts about whether MAXEDGE should produce
> scaling of the tilematrix (or some other matrix). Hopefully that will be
> an easy thing for you to comment on.
>
> - K
>
> Kevin Day
>
> *trumpet**p| *480.961.6003 x1002
> *e| *kevin@trumpetinc.com
> www.trumpetinc.com <http://trumpetinc.com/>
> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
> <http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
>
> Proud to be Great Place To Work
> <https://www.greatplacetowork.com/certified-company/7012667> certified
> since 2019
>
>
> On Sun, Nov 7, 2021 at 10:03 PM Kevin Day <ke...@trumpetinc.com> wrote:
>
>> Test procedure:
>>
>> Open PdfBox Debugger, load PDF, capture render time from label in lower
>> left corner of window.
>>
>>
>> With PdfBox 2.0.17 - this PDF renders in about 4.8 seconds (but is missing
>> the colored pattern).
>>
>> With PdfBox 3 baseline (no code changes) - this PDF renders in 16.2
>> seconds seconds.
>>
>> With PdfBox 3 caching changes only (no BigBufferedImage) - this PDF
>> renders in 14.7 seconds
>>
>>
>> So I have not done any damage, and I've sped it up a little bit.
>>
>>
>> But that's only with a single render. Because printing can involve
>> rendering many times for banded printing (on my test device, I am seeing
>> well over 100 bands when printing this sample file), the page level caching
>> should be substantial (the rendering on your test PDF is so slow that it
>> almost can't be printed) - it is certainly a huge improvement for my test
>> document.
>>
>>
>> So I think my improvements to the caching (both the tile cache and adding
>> caching at the page level) are valid.
>>
>> - K
>>
>> Kevin Day
>>
>> *trumpet**p| *480.961.6003 x1002
>> *e| *kevin@trumpetinc.com
>> www.trumpetinc.com <http://trumpetinc.com/>
>> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
>> <http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
>>
>> Proud to be Great Place To Work
>> <https://www.greatplacetowork.com/certified-company/7012667> certified
>> since 2019
>>
>>
>> On Sun, Nov 7, 2021 at 8:34 PM Tilman Hausherr <TH...@t-online.de>
>> wrote:
>>
>>> Hi,
>>>
>>> I don't have much time right now, but please test your change with the
>>> file from
>>>
>>> https://issues.apache.org/jira/browse/PDFBOX-3688
>>>
>>> Tilman
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
>>> For additional commands, e-mail: users-help@pdfbox.apache.org
>>>
>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
For additional commands, e-mail: users-help@pdfbox.apache.org
Re: Guidance on heap space error when printing document
Posted by Kevin Day <ke...@trumpetinc.com>.
To really fix the printing performance, I think you will need to either:
1. The drawer needs to have a viewport concept that filters content stream
operators better they render. I don't know if this is even possible or
reasonable. I would have thought that sharing like this would suggest be
implemented, but I couldn't find it. If it is implemented, then adjusting
the print handler to extract the band viewport would probably help
performance a lot.
Or
2. Create a private raster at the correct dpi, then render to it first,
then transfer data to the printer from that raster. I think that you would
want a disk based raster for this (similar to BigBufferedImage but better
designed)
Given the design of the Java print system, it may not be possible to
extract the band viewport, in which case option will probably be your best
bet.
On Mon, Nov 8, 2021, 12:48 AM Tilman Hausherr <TH...@t-online.de> wrote:
> Am 08.11.2021 um 06:03 schrieb Kevin Day:
> > But that's only with a single render. Because printing can involve
> > rendering many times for banded printing (on my test device, I am seeing
> > well over 100 bands when printing this sample file), the page level
> caching
> > should be substantial (the rendering on your test PDF is so slow that it
> > almost can't be printed) - it is certainly a huge improvement for my test
> > document.
>
> The printing improvement should be the first thing to test because this
> has been criticized for years. This happens with all PDFs that have
> transparencies, there are several issues about this. JPedal has the same
> problem.
>
> (It might take a few days before I test this)
>
> Tilman
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> For additional commands, e-mail: users-help@pdfbox.apache.org
>
>
Re: Guidance on heap space error when printing document
Posted by Tilman Hausherr <TH...@t-online.de>.
Am 08.11.2021 um 06:03 schrieb Kevin Day:
> But that's only with a single render. Because printing can involve
> rendering many times for banded printing (on my test device, I am seeing
> well over 100 bands when printing this sample file), the page level caching
> should be substantial (the rendering on your test PDF is so slow that it
> almost can't be printed) - it is certainly a huge improvement for my test
> document.
The printing improvement should be the first thing to test because this
has been criticized for years. This happens with all PDFs that have
transparencies, there are several issues about this. JPedal has the same
problem.
(It might take a few days before I test this)
Tilman
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
For additional commands, e-mail: users-help@pdfbox.apache.org
Re: Guidance on heap space error when printing document
Posted by Kevin Day <ke...@trumpetinc.com>.
I just realized what MAXEDGE does.
You are basically saying that the raster can't be bigger than MAXEDGE x
MAXEDGE, and if it is, then you clamp the dimensions.
But - are you scaling the bitmap? Or are you just cropping it? If you are
just cropping it, it seems like a better approach would be to adjust the
tiling matrix by the amount of the scaling required to prevent the raster
from being too big.
I just committed more changes to my fork:
1. Refined MAXEDGE clamping to scale both dimensions instead of clamping
the long dimension (I think this is a better way to do this - but it feels
like we should also be scaling the tiling matrix - see above)
2. Added default MAXEDGE values that depend on the bitness of the JVM. I
found that MAXEDGE = 500 worked on 32 bit JVMs (but 650 does *not* work).
So setting MAXEDGE to 500 for 32 bit JVMs and 3000 for 64 bit JVMs seems
like a pretty decent compromise.
I look forward to your thoughts about whether MAXEDGE should produce
scaling of the tilematrix (or some other matrix). Hopefully that will be
an easy thing for you to comment on.
- K
Kevin Day
*trumpet**p| *480.961.6003 x1002
*e| *kevin@trumpetinc.com
www.trumpetinc.com <http://trumpetinc.com/>
LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
<http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
Proud to be Great Place To Work
<https://www.greatplacetowork.com/certified-company/7012667> certified
since 2019
On Sun, Nov 7, 2021 at 10:03 PM Kevin Day <ke...@trumpetinc.com> wrote:
> Test procedure:
>
> Open PdfBox Debugger, load PDF, capture render time from label in lower
> left corner of window.
>
>
> With PdfBox 2.0.17 - this PDF renders in about 4.8 seconds (but is missing
> the colored pattern).
>
> With PdfBox 3 baseline (no code changes) - this PDF renders in 16.2
> seconds seconds.
>
> With PdfBox 3 caching changes only (no BigBufferedImage) - this PDF
> renders in 14.7 seconds
>
>
> So I have not done any damage, and I've sped it up a little bit.
>
>
> But that's only with a single render. Because printing can involve
> rendering many times for banded printing (on my test device, I am seeing
> well over 100 bands when printing this sample file), the page level caching
> should be substantial (the rendering on your test PDF is so slow that it
> almost can't be printed) - it is certainly a huge improvement for my test
> document.
>
>
> So I think my improvements to the caching (both the tile cache and adding
> caching at the page level) are valid.
>
> - K
>
> Kevin Day
>
> *trumpet**p| *480.961.6003 x1002
> *e| *kevin@trumpetinc.com
> www.trumpetinc.com <http://trumpetinc.com/>
> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
> <http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
>
> Proud to be Great Place To Work
> <https://www.greatplacetowork.com/certified-company/7012667> certified
> since 2019
>
>
> On Sun, Nov 7, 2021 at 8:34 PM Tilman Hausherr <TH...@t-online.de>
> wrote:
>
>> Hi,
>>
>> I don't have much time right now, but please test your change with the
>> file from
>>
>> https://issues.apache.org/jira/browse/PDFBOX-3688
>>
>> Tilman
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
>> For additional commands, e-mail: users-help@pdfbox.apache.org
>>
>>
Re: Guidance on heap space error when printing document
Posted by Kevin Day <ke...@trumpetinc.com>.
Test procedure:
Open PdfBox Debugger, load PDF, capture render time from label in lower
left corner of window.
With PdfBox 2.0.17 - this PDF renders in about 4.8 seconds (but is missing
the colored pattern).
With PdfBox 3 baseline (no code changes) - this PDF renders in 16.2 seconds
seconds.
With PdfBox 3 caching changes only (no BigBufferedImage) - this PDF renders
in 14.7 seconds
So I have not done any damage, and I've sped it up a little bit.
But that's only with a single render. Because printing can involve
rendering many times for banded printing (on my test device, I am seeing
well over 100 bands when printing this sample file), the page level caching
should be substantial (the rendering on your test PDF is so slow that it
almost can't be printed) - it is certainly a huge improvement for my test
document.
So I think my improvements to the caching (both the tile cache and adding
caching at the page level) are valid.
- K
Kevin Day
*trumpet**p| *480.961.6003 x1002
*e| *kevin@trumpetinc.com
www.trumpetinc.com <http://trumpetinc.com/>
LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
<http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
Proud to be Great Place To Work
<https://www.greatplacetowork.com/certified-company/7012667> certified
since 2019
On Sun, Nov 7, 2021 at 8:34 PM Tilman Hausherr <TH...@t-online.de>
wrote:
> Hi,
>
> I don't have much time right now, but please test your change with the
> file from
>
> https://issues.apache.org/jira/browse/PDFBOX-3688
>
> Tilman
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> For additional commands, e-mail: users-help@pdfbox.apache.org
>
>
Re: Guidance on heap space error when printing document
Posted by Tilman Hausherr <TH...@t-online.de>.
Hi,
I don't have much time right now, but please test your change with the
file from
https://issues.apache.org/jira/browse/PDFBOX-3688
Tilman
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
For additional commands, e-mail: users-help@pdfbox.apache.org
Re: Guidance on heap space error when printing document
Posted by Kevin Day <ke...@trumpetinc.com>.
oh - I see a major problem (it's a bit of a reverse memory leak, actually)
that is involved in your disk space issue (it also results in really poor
performance for image tiling operations):
The cache in TilingPaintFactory is based on WeakHashMap. WeakHashMap is
designed around weak referenced *keys*, and the TilingPaintFactory.create()
method is constructing a new key every call. This means that entries are
not going to be in the cache for very long (especially with a generational
garbage collector). In fact, in my testing, the cache is always empty.
This results in a new BigBufferedImage (with a new disk cache) being
created every time to the tile is painted.
So I think at least some of the problem is that the cache is not
implemented properly.
If you are OK using an additional Apache dependency, I think that the cache
should be changed to this (with strong keys and weak values):
https://commons.apache.org/proper/commons-collections/apidocs/org/apache/commons/collections4/map/ReferenceMap.html
Next, the print job is running in stripes - so a single page gets rendered
many times. Right now, a new PageDrawer is being constructed for each
stripe in PDFRenderer.renderImage(...), which is resulting in a huge number
of BufferedImages being created. Adding a cache for PageDrawer instances
addresses this.
Next, the key on the TilingPant cache in TilingPaintFactory is based on the
xform affine transformation. It needs this (I think) to handle scaling -
but with stripes, the offset is changing but the scaling isn't. So I think
that the TilingPaintParameter.hashCode() and equals() needs to be adjusted
to only consider scale - not translation.
When I make these changes, it dramatically reduces the disk usage.
I have submitted a pull request with these changes for your evaluation
(this is NOT a complete fix for the problem - it still uses
BigBufferedImage): https://github.com/apache/pdfbox/pull/135
- K
Kevin Day
*trumpet**p| *480.961.6003 x1002
*e| *kevin@trumpetinc.com
www.trumpetinc.com <http://trumpetinc.com/>
LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
<http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
Proud to be Great Place To Work
<https://www.greatplacetowork.com/certified-company/7012667> certified
since 2019
On Sun, Nov 7, 2021 at 9:36 AM Kevin Day <ke...@trumpetinc.com> wrote:
> Ah yes - the file deletion in BigBufferedImage only happens during
> finalize() (or with a shutdown hook). That's why you are blowing out your
> disk space. We will have the same problem even if we use ScratchFile.
>
> This means that we have to talk about disposing the BufferedImage
> instances - but BufferedImage does not have a dispose() method on it.
>
> If we add dispose on the buffered image, now we also need dispose() on the
> Paint object, etc.... It gets very ugly...
>
> - K
>
> Kevin Day
>
> *trumpet**p| *480.961.6003 x1002
> *e| *kevin@trumpetinc.com
> www.trumpetinc.com <http://trumpetinc.com/>
> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
> <http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
>
> Proud to be Great Place To Work
> <https://www.greatplacetowork.com/certified-company/7012667> certified
> since 2019
>
>
> On Sun, Nov 7, 2021 at 8:55 AM Kevin Day <ke...@trumpetinc.com> wrote:
>
>> There is another issue with BigBufferedImage - it does not work well on
>> 32 bit JVMs. The problem is that it creates a single MappedByteBuffer for
>> the entire size of a channel's data (for the problem file, this means that
>> each buffer has 172,975,104 bytes).
>>
>> I think it would be better to create a DataBuffer implementation that
>> uses org.apache.pdfbox.io.ScratchFile and create a PdfBoxBufferedImage...
>>
>> - K
>>
>> Kevin Day
>>
>> *trumpet**p| *480.961.6003 x1002
>> *e| *kevin@trumpetinc.com
>> www.trumpetinc.com <http://trumpetinc.com/>
>> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
>> <http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
>>
>> Proud to be Great Place To Work
>> <https://www.greatplacetowork.com/certified-company/7012667> certified
>> since 2019
>>
>>
>> On Sun, Nov 7, 2021 at 8:36 AM Tilman Hausherr <TH...@t-online.de>
>> wrote:
>>
>>> Am 07.11.2021 um 16:09 schrieb Kevin Day:
>>> > ok - I think that maybe there are two things going on, then. The
>>> first is
>>> > that we are allocating a single buffer that is bigger than the default
>>> heap
>>> > (700MB > 500MB). I really think that the only two solutions are to move
>>> > this off the heap or reduce the size of the allocation.
>>> >
>>> > Then the second issue is maybe a memory leak (I have not been able to
>>> see
>>> > this because of the first problem unless I set my max heap to a huge
>>> number
>>> > - and that makes memory leak analysis very difficult). I am going to
>>> clone
>>> > the PdfBox git repo now and swap that class in then do a little bit of
>>> > profiling with jvisualvm to see if I can help find the memory leak.
>>> >
>>> > Question: The Git repo for PdfBox says that it is a mirror. If I do
>>> wind
>>> > up creating a pull request against this repo, will you be able to
>>> accept it?
>>>
>>> Not directly but we can create a .diff / .patch file from the PR.
>>>
>>> In the meantime I ran my rendering regression checks which renders 1000
>>> files. The hard disk C: went full after using 260 GB :-(
>>>
>>> Tilman
>>>
>>>
>>> >
>>> >
>>> > FYI - the use of sun.* classes is just to work around a long standing
>>> > native resource leak bug in the Windows MappedByteBuffer
>>> implementation -
>>> > On Windows, MappedByteBuffer does not release the allocated native
>>> resource
>>> > until finalize() is called, and because the MappedByteBuffer itself
>>> does
>>> > not take up much heap space, it can stay on the heap for quite a long
>>> time
>>> > even after all references are gone. So the sun.* reference is just to
>>> > force the release of those native resources. I used the same technique
>>> > when I created the high throughput IO layer for iText. Instead of
>>> > referencing the package directly, we can use reflection (and only use
>>> it
>>> > for winXXX platform) - this shouldn't be a big problem.
>>> >
>>> > K
>>> >
>>> > Kevin Day
>>> >
>>> > *trumpet**p| *480.961.6003 x1002
>>> > *e| *kevin@trumpetinc.com
>>> > www.trumpetinc.com <http://trumpetinc.com/>
>>> > LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
>>> > <http://trumpetinc.com/blog/>| Twitter <
>>> https://twitter.com/trumpetinc>
>>> >
>>> > Proud to be Great Place To Work
>>> > <https://www.greatplacetowork.com/certified-company/7012667> certified
>>> > since 2019
>>> >
>>> >
>>> > On Sun, Nov 7, 2021 at 7:53 AM Tilman Hausherr <TH...@t-online.de>
>>> > wrote:
>>> >
>>> >> I tried it and it works pretty fast. The memory leak is still there
>>> but
>>> >> it comes later.
>>> >>
>>> >> Other drawbacks arethat it doesn't clean up after itself fast enough,
>>> >> and it uses a sun.* class.
>>> >>
>>> >> Tilman
>>> >>
>>> >> Am 07.11.2021 um 14:31 schrieb Tilman Hausherr:
>>> >>> Am 05.11.2021 um 22:09 schrieb Kevin Day:
>>> >>>> ok - so should we be clamping the xstep in some way? Or at this
>>> >>>> depth of
>>> >>>> the algorithm do we not have enough context to actually tell that it
>>> >>>> will
>>> >>>> be outside the page/clipping region?
>>> >>> Yes + yes, that is the problem. The context would have to include the
>>> >>> region but also the current transformation matrix. Which could change
>>> >>> despite using the same pattern. So it's tricky.
>>> >>>
>>> >>>>
>>> >>>> I'm beginning to think that the BigBufferedImage might be the right
>>> >>>> solution... This is very inefficient, but honestly, the PDF is not
>>> >>>> exactly
>>> >>>> well formed, so if these particular files wind up being slower to
>>> render
>>> >>>> b/c they have to swap image content to disk, I think that is OK...
>>> >>> Maybe... the good thing is that we know when the image will be "too
>>> >>> big". The license is OK:
>>> >>>
>>> >>> https://www.apache.org/legal/resolved.html
>>> >>>
>>> >>> But I'm still wondering whether we have a memory leak. If we have,
>>> >>> then it should be fixed.
>>> >>>
>>> >>> Tilman
>>> >>>
>>> >>>
>>> >>>> - K
>>> >>>>
>>> >>>>
>>> >>>> Kevin Day
>>> >>>>
>>> >>>> *trumpet**p| *480.961.6003 x1002
>>> >>>> *e| *kevin@trumpetinc.com
>>> >>>> www.trumpetinc.com <http://trumpetinc.com/>
>>> >>>> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet
>>> Blog
>>> >>>> <http://trumpetinc.com/blog/>| Twitter <
>>> https://twitter.com/trumpetinc>
>>> >>>>
>>> >>>> Proud to be Great Place To Work
>>> >>>> <https://www.greatplacetowork.com/certified-company/7012667>
>>> Certified
>>> >>>> 2020-2021
>>> >>>>
>>> >>>>
>>> >>>> On Fri, Nov 5, 2021 at 12:47 PM Tilman Hausherr <
>>> THausherr@t-online.de>
>>> >>>> wrote:
>>> >>>>
>>> >>>>> Am 05.11.2021 um 20:38 schrieb Kevin Day:
>>> >>>>>> I do have a bit of experience with memory leaks (side effect of
>>> >>>>>> writing
>>> >>>>>> high performance java code, unfortunately!)
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> I am pretty sure that this is not a memory leak, though. If this
>>> >>>>>> was a
>>> >>>>>> memory leak, we would see calls happening multiple times before
>>> >>>>>> failure.
>>> >>>>>>
>>> >>>>>> The failure happens on the very first call to create the buffered
>>> >>>>>> image:
>>> >>>>>>
>>> >>>>>> BufferedImage image = new BufferedImage(rasterWidth,
>>> >>>>> rasterHeight,
>>> >>>>>> BufferedImage.TYPE_INT_ARGB);
>>> >>>>>>
>>> >>>>>> Nothing is even getting put into the weakcache.
>>> >>>>>>
>>> >>>>> That is done in TilingPaintFactory. The cache was needed for some
>>> files
>>> >>>>> that have used the pattern several times, to avoid recreating the
>>> image
>>> >>>>> for the TexturePaint.
>>> >>>>>
>>> >>>>>
>>> >>>>>> I think the core issue is that the raster data for the image is
>>> >>>>>> just too
>>> >>>>>> big to store on the heap (without the heap being massive). It's
>>> >>>>>> 13152x13152x4 bytes (there is an alpha channel on this as well) -
>>> >>>>>> that's
>>> >>>>>> almost 700MB - just for this one pattern.
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> Tilman wrote: "That
>>> >>>>>> pattern has an XStep and YStep of 23438 although the image is
>>> 2148 x
>>> >>>>>> 440. Because of a matrix scale of 0.0673396 the image pattern
>>> size is
>>> >>>>>> 1578 x 1578 at 72 dpi. So at 1200 dpi the size would be about
>>> 26300 x
>>> >>>>>> 26300"
>>> >>>>>>
>>> >>>>>> I am fairly familiar with PDF (I contributed the parsing library
>>> to
>>> >>>>>> iText
>>> >>>>>> back in the day), but I'm not very familiar with rendering tile
>>> >>>>>> patterns,
>>> >>>>>> so the above is not intuitive to me (yet). Is there some chance
>>> that
>>> >>>>>> PdfBox is doing a lot more work than is necessary for this
>>> particular
>>> >>>>> PDF?
>>> >>>>>> Like is the matrix scale just absurdly bad in this file? For an
>>> image
>>> >>>>> that
>>> >>>>>> is originally 2148x440, it seems like requiring a 700MB raster
>>> >>>>>> should not
>>> >>>>>> be necessary - but I do NOT have nearly enough experience with
>>> this
>>> >>>>>> area
>>> >>>>> of
>>> >>>>>> PDF...
>>> >>>>> The ridiculous thing isn't the matrix, it's the XStep and YStep.
>>> This
>>> >>>>> goes well outside of the page. The 0.067 matrix scaling makes it
>>> >>>>> less bad.
>>> >>>>>
>>> >>>>> A weakness in PDFBox is that we don't readjust the tile image, or
>>> >>>>> use an
>>> >>>>> alternative painting method that doesn't use TexturePaint when the
>>> >>>>> image
>>> >>>>> would be painted only once.
>>> >>>>>
>>> >>>>> Tilman
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>> Thanks,
>>> >>>>>>
>>> >>>>>> - Kevin
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> Kevin Day
>>> >>>>>>
>>> >>>>>> *trumpet**p| *480.961.6003 x1002
>>> >>>>>> *e| *kevin@trumpetinc.com
>>> >>>>>> www.trumpetinc.com <http://trumpetinc.com/>
>>> >>>>>> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet
>>> >> Blog
>>> >>>>>> <http://trumpetinc.com/blog/>| Twitter
>>> >>>>>> <https://twitter.com/trumpetinc>
>>> >>>>>>
>>> >>>>>> Proud to be Great Place To Work
>>> >>>>>> <https://www.greatplacetowork.com/certified-company/7012667>
>>> >> Certified
>>> >>>>>> 2020-2021
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> On Fri, Nov 5, 2021 at 11:27 AM Tilman Hausherr
>>> >>>>>> <TH...@t-online.de>
>>> >>>>>> wrote:
>>> >>>>>>
>>> >>>>>>> If you're experienced with memory leaks then it would be nice if
>>> you
>>> >>>>>>> could search.
>>> >>>>>>>
>>> >>>>>>> Things I tried that didn't help:
>>> >>>>>>> - calling graphics.setPaint(null) after operations (to "lose"
>>> >>>>>>> TilingPaint objects)
>>> >>>>>>> - disabling the (weak) cache of TilingPaint objects
>>> >>>>>>> - adding finalize to see if the TilingPaint class gets finalized
>>> >>>>>>> (yes)
>>> >>>>>>> - adding finalize to see if the HighResolutionImageIcon class
>>> gets
>>> >>>>>>> finalized (yes)
>>> >>>>>>>
>>> >>>>>>> Tilman
>>> >>>>>>>
>>> >>>>>
>>> ---------------------------------------------------------------------
>>> >>>>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
>>> >>>>> For additional commands, e-mail: users-help@pdfbox.apache.org
>>> >>>>>
>>> >>>>>
>>> >>>
>>> >>> ---------------------------------------------------------------------
>>> >>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
>>> >>> For additional commands, e-mail: users-help@pdfbox.apache.org
>>> >>>
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
>>> >> For additional commands, e-mail: users-help@pdfbox.apache.org
>>> >>
>>> >>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
>>> For additional commands, e-mail: users-help@pdfbox.apache.org
>>>
>>>
Re: Guidance on heap space error when printing document
Posted by Kevin Day <ke...@trumpetinc.com>.
Ah yes - the file deletion in BigBufferedImage only happens during
finalize() (or with a shutdown hook). That's why you are blowing out your
disk space. We will have the same problem even if we use ScratchFile.
This means that we have to talk about disposing the BufferedImage instances
- but BufferedImage does not have a dispose() method on it.
If we add dispose on the buffered image, now we also need dispose() on the
Paint object, etc.... It gets very ugly...
- K
Kevin Day
*trumpet**p| *480.961.6003 x1002
*e| *kevin@trumpetinc.com
www.trumpetinc.com <http://trumpetinc.com/>
LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
<http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
Proud to be Great Place To Work
<https://www.greatplacetowork.com/certified-company/7012667> certified
since 2019
On Sun, Nov 7, 2021 at 8:55 AM Kevin Day <ke...@trumpetinc.com> wrote:
> There is another issue with BigBufferedImage - it does not work well on 32
> bit JVMs. The problem is that it creates a single MappedByteBuffer for the
> entire size of a channel's data (for the problem file, this means that each
> buffer has 172,975,104 bytes).
>
> I think it would be better to create a DataBuffer implementation that uses
> org.apache.pdfbox.io.ScratchFile and create a PdfBoxBufferedImage...
>
> - K
>
> Kevin Day
>
> *trumpet**p| *480.961.6003 x1002
> *e| *kevin@trumpetinc.com
> www.trumpetinc.com <http://trumpetinc.com/>
> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
> <http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
>
> Proud to be Great Place To Work
> <https://www.greatplacetowork.com/certified-company/7012667> certified
> since 2019
>
>
> On Sun, Nov 7, 2021 at 8:36 AM Tilman Hausherr <TH...@t-online.de>
> wrote:
>
>> Am 07.11.2021 um 16:09 schrieb Kevin Day:
>> > ok - I think that maybe there are two things going on, then. The first
>> is
>> > that we are allocating a single buffer that is bigger than the default
>> heap
>> > (700MB > 500MB). I really think that the only two solutions are to move
>> > this off the heap or reduce the size of the allocation.
>> >
>> > Then the second issue is maybe a memory leak (I have not been able to
>> see
>> > this because of the first problem unless I set my max heap to a huge
>> number
>> > - and that makes memory leak analysis very difficult). I am going to
>> clone
>> > the PdfBox git repo now and swap that class in then do a little bit of
>> > profiling with jvisualvm to see if I can help find the memory leak.
>> >
>> > Question: The Git repo for PdfBox says that it is a mirror. If I do
>> wind
>> > up creating a pull request against this repo, will you be able to
>> accept it?
>>
>> Not directly but we can create a .diff / .patch file from the PR.
>>
>> In the meantime I ran my rendering regression checks which renders 1000
>> files. The hard disk C: went full after using 260 GB :-(
>>
>> Tilman
>>
>>
>> >
>> >
>> > FYI - the use of sun.* classes is just to work around a long standing
>> > native resource leak bug in the Windows MappedByteBuffer implementation
>> -
>> > On Windows, MappedByteBuffer does not release the allocated native
>> resource
>> > until finalize() is called, and because the MappedByteBuffer itself does
>> > not take up much heap space, it can stay on the heap for quite a long
>> time
>> > even after all references are gone. So the sun.* reference is just to
>> > force the release of those native resources. I used the same technique
>> > when I created the high throughput IO layer for iText. Instead of
>> > referencing the package directly, we can use reflection (and only use it
>> > for winXXX platform) - this shouldn't be a big problem.
>> >
>> > K
>> >
>> > Kevin Day
>> >
>> > *trumpet**p| *480.961.6003 x1002
>> > *e| *kevin@trumpetinc.com
>> > www.trumpetinc.com <http://trumpetinc.com/>
>> > LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
>> > <http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc
>> >
>> >
>> > Proud to be Great Place To Work
>> > <https://www.greatplacetowork.com/certified-company/7012667> certified
>> > since 2019
>> >
>> >
>> > On Sun, Nov 7, 2021 at 7:53 AM Tilman Hausherr <TH...@t-online.de>
>> > wrote:
>> >
>> >> I tried it and it works pretty fast. The memory leak is still there but
>> >> it comes later.
>> >>
>> >> Other drawbacks arethat it doesn't clean up after itself fast enough,
>> >> and it uses a sun.* class.
>> >>
>> >> Tilman
>> >>
>> >> Am 07.11.2021 um 14:31 schrieb Tilman Hausherr:
>> >>> Am 05.11.2021 um 22:09 schrieb Kevin Day:
>> >>>> ok - so should we be clamping the xstep in some way? Or at this
>> >>>> depth of
>> >>>> the algorithm do we not have enough context to actually tell that it
>> >>>> will
>> >>>> be outside the page/clipping region?
>> >>> Yes + yes, that is the problem. The context would have to include the
>> >>> region but also the current transformation matrix. Which could change
>> >>> despite using the same pattern. So it's tricky.
>> >>>
>> >>>>
>> >>>> I'm beginning to think that the BigBufferedImage might be the right
>> >>>> solution... This is very inefficient, but honestly, the PDF is not
>> >>>> exactly
>> >>>> well formed, so if these particular files wind up being slower to
>> render
>> >>>> b/c they have to swap image content to disk, I think that is OK...
>> >>> Maybe... the good thing is that we know when the image will be "too
>> >>> big". The license is OK:
>> >>>
>> >>> https://www.apache.org/legal/resolved.html
>> >>>
>> >>> But I'm still wondering whether we have a memory leak. If we have,
>> >>> then it should be fixed.
>> >>>
>> >>> Tilman
>> >>>
>> >>>
>> >>>> - K
>> >>>>
>> >>>>
>> >>>> Kevin Day
>> >>>>
>> >>>> *trumpet**p| *480.961.6003 x1002
>> >>>> *e| *kevin@trumpetinc.com
>> >>>> www.trumpetinc.com <http://trumpetinc.com/>
>> >>>> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet
>> Blog
>> >>>> <http://trumpetinc.com/blog/>| Twitter <
>> https://twitter.com/trumpetinc>
>> >>>>
>> >>>> Proud to be Great Place To Work
>> >>>> <https://www.greatplacetowork.com/certified-company/7012667>
>> Certified
>> >>>> 2020-2021
>> >>>>
>> >>>>
>> >>>> On Fri, Nov 5, 2021 at 12:47 PM Tilman Hausherr <
>> THausherr@t-online.de>
>> >>>> wrote:
>> >>>>
>> >>>>> Am 05.11.2021 um 20:38 schrieb Kevin Day:
>> >>>>>> I do have a bit of experience with memory leaks (side effect of
>> >>>>>> writing
>> >>>>>> high performance java code, unfortunately!)
>> >>>>>>
>> >>>>>>
>> >>>>>> I am pretty sure that this is not a memory leak, though. If this
>> >>>>>> was a
>> >>>>>> memory leak, we would see calls happening multiple times before
>> >>>>>> failure.
>> >>>>>>
>> >>>>>> The failure happens on the very first call to create the buffered
>> >>>>>> image:
>> >>>>>>
>> >>>>>> BufferedImage image = new BufferedImage(rasterWidth,
>> >>>>> rasterHeight,
>> >>>>>> BufferedImage.TYPE_INT_ARGB);
>> >>>>>>
>> >>>>>> Nothing is even getting put into the weakcache.
>> >>>>>>
>> >>>>> That is done in TilingPaintFactory. The cache was needed for some
>> files
>> >>>>> that have used the pattern several times, to avoid recreating the
>> image
>> >>>>> for the TexturePaint.
>> >>>>>
>> >>>>>
>> >>>>>> I think the core issue is that the raster data for the image is
>> >>>>>> just too
>> >>>>>> big to store on the heap (without the heap being massive). It's
>> >>>>>> 13152x13152x4 bytes (there is an alpha channel on this as well) -
>> >>>>>> that's
>> >>>>>> almost 700MB - just for this one pattern.
>> >>>>>>
>> >>>>>>
>> >>>>>> Tilman wrote: "That
>> >>>>>> pattern has an XStep and YStep of 23438 although the image is 2148
>> x
>> >>>>>> 440. Because of a matrix scale of 0.0673396 the image pattern size
>> is
>> >>>>>> 1578 x 1578 at 72 dpi. So at 1200 dpi the size would be about
>> 26300 x
>> >>>>>> 26300"
>> >>>>>>
>> >>>>>> I am fairly familiar with PDF (I contributed the parsing library to
>> >>>>>> iText
>> >>>>>> back in the day), but I'm not very familiar with rendering tile
>> >>>>>> patterns,
>> >>>>>> so the above is not intuitive to me (yet). Is there some chance
>> that
>> >>>>>> PdfBox is doing a lot more work than is necessary for this
>> particular
>> >>>>> PDF?
>> >>>>>> Like is the matrix scale just absurdly bad in this file? For an
>> image
>> >>>>> that
>> >>>>>> is originally 2148x440, it seems like requiring a 700MB raster
>> >>>>>> should not
>> >>>>>> be necessary - but I do NOT have nearly enough experience with this
>> >>>>>> area
>> >>>>> of
>> >>>>>> PDF...
>> >>>>> The ridiculous thing isn't the matrix, it's the XStep and YStep.
>> This
>> >>>>> goes well outside of the page. The 0.067 matrix scaling makes it
>> >>>>> less bad.
>> >>>>>
>> >>>>> A weakness in PDFBox is that we don't readjust the tile image, or
>> >>>>> use an
>> >>>>> alternative painting method that doesn't use TexturePaint when the
>> >>>>> image
>> >>>>> would be painted only once.
>> >>>>>
>> >>>>> Tilman
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>> Thanks,
>> >>>>>>
>> >>>>>> - Kevin
>> >>>>>>
>> >>>>>>
>> >>>>>> Kevin Day
>> >>>>>>
>> >>>>>> *trumpet**p| *480.961.6003 x1002
>> >>>>>> *e| *kevin@trumpetinc.com
>> >>>>>> www.trumpetinc.com <http://trumpetinc.com/>
>> >>>>>> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet
>> >> Blog
>> >>>>>> <http://trumpetinc.com/blog/>| Twitter
>> >>>>>> <https://twitter.com/trumpetinc>
>> >>>>>>
>> >>>>>> Proud to be Great Place To Work
>> >>>>>> <https://www.greatplacetowork.com/certified-company/7012667>
>> >> Certified
>> >>>>>> 2020-2021
>> >>>>>>
>> >>>>>>
>> >>>>>> On Fri, Nov 5, 2021 at 11:27 AM Tilman Hausherr
>> >>>>>> <TH...@t-online.de>
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>>> If you're experienced with memory leaks then it would be nice if
>> you
>> >>>>>>> could search.
>> >>>>>>>
>> >>>>>>> Things I tried that didn't help:
>> >>>>>>> - calling graphics.setPaint(null) after operations (to "lose"
>> >>>>>>> TilingPaint objects)
>> >>>>>>> - disabling the (weak) cache of TilingPaint objects
>> >>>>>>> - adding finalize to see if the TilingPaint class gets finalized
>> >>>>>>> (yes)
>> >>>>>>> - adding finalize to see if the HighResolutionImageIcon class gets
>> >>>>>>> finalized (yes)
>> >>>>>>>
>> >>>>>>> Tilman
>> >>>>>>>
>> >>>>>
>> ---------------------------------------------------------------------
>> >>>>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
>> >>>>> For additional commands, e-mail: users-help@pdfbox.apache.org
>> >>>>>
>> >>>>>
>> >>>
>> >>> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
>> >>> For additional commands, e-mail: users-help@pdfbox.apache.org
>> >>>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
>> >> For additional commands, e-mail: users-help@pdfbox.apache.org
>> >>
>> >>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
>> For additional commands, e-mail: users-help@pdfbox.apache.org
>>
>>
Re: Guidance on heap space error when printing document
Posted by Kevin Day <ke...@trumpetinc.com>.
There is another issue with BigBufferedImage - it does not work well on 32
bit JVMs. The problem is that it creates a single MappedByteBuffer for the
entire size of a channel's data (for the problem file, this means that each
buffer has 172,975,104 bytes).
I think it would be better to create a DataBuffer implementation that uses
org.apache.pdfbox.io.ScratchFile and create a PdfBoxBufferedImage...
- K
Kevin Day
*trumpet**p| *480.961.6003 x1002
*e| *kevin@trumpetinc.com
www.trumpetinc.com <http://trumpetinc.com/>
LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
<http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
Proud to be Great Place To Work
<https://www.greatplacetowork.com/certified-company/7012667> certified
since 2019
On Sun, Nov 7, 2021 at 8:36 AM Tilman Hausherr <TH...@t-online.de>
wrote:
> Am 07.11.2021 um 16:09 schrieb Kevin Day:
> > ok - I think that maybe there are two things going on, then. The first
> is
> > that we are allocating a single buffer that is bigger than the default
> heap
> > (700MB > 500MB). I really think that the only two solutions are to move
> > this off the heap or reduce the size of the allocation.
> >
> > Then the second issue is maybe a memory leak (I have not been able to see
> > this because of the first problem unless I set my max heap to a huge
> number
> > - and that makes memory leak analysis very difficult). I am going to
> clone
> > the PdfBox git repo now and swap that class in then do a little bit of
> > profiling with jvisualvm to see if I can help find the memory leak.
> >
> > Question: The Git repo for PdfBox says that it is a mirror. If I do
> wind
> > up creating a pull request against this repo, will you be able to accept
> it?
>
> Not directly but we can create a .diff / .patch file from the PR.
>
> In the meantime I ran my rendering regression checks which renders 1000
> files. The hard disk C: went full after using 260 GB :-(
>
> Tilman
>
>
> >
> >
> > FYI - the use of sun.* classes is just to work around a long standing
> > native resource leak bug in the Windows MappedByteBuffer implementation -
> > On Windows, MappedByteBuffer does not release the allocated native
> resource
> > until finalize() is called, and because the MappedByteBuffer itself does
> > not take up much heap space, it can stay on the heap for quite a long
> time
> > even after all references are gone. So the sun.* reference is just to
> > force the release of those native resources. I used the same technique
> > when I created the high throughput IO layer for iText. Instead of
> > referencing the package directly, we can use reflection (and only use it
> > for winXXX platform) - this shouldn't be a big problem.
> >
> > K
> >
> > Kevin Day
> >
> > *trumpet**p| *480.961.6003 x1002
> > *e| *kevin@trumpetinc.com
> > www.trumpetinc.com <http://trumpetinc.com/>
> > LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
> > <http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
> >
> > Proud to be Great Place To Work
> > <https://www.greatplacetowork.com/certified-company/7012667> certified
> > since 2019
> >
> >
> > On Sun, Nov 7, 2021 at 7:53 AM Tilman Hausherr <TH...@t-online.de>
> > wrote:
> >
> >> I tried it and it works pretty fast. The memory leak is still there but
> >> it comes later.
> >>
> >> Other drawbacks arethat it doesn't clean up after itself fast enough,
> >> and it uses a sun.* class.
> >>
> >> Tilman
> >>
> >> Am 07.11.2021 um 14:31 schrieb Tilman Hausherr:
> >>> Am 05.11.2021 um 22:09 schrieb Kevin Day:
> >>>> ok - so should we be clamping the xstep in some way? Or at this
> >>>> depth of
> >>>> the algorithm do we not have enough context to actually tell that it
> >>>> will
> >>>> be outside the page/clipping region?
> >>> Yes + yes, that is the problem. The context would have to include the
> >>> region but also the current transformation matrix. Which could change
> >>> despite using the same pattern. So it's tricky.
> >>>
> >>>>
> >>>> I'm beginning to think that the BigBufferedImage might be the right
> >>>> solution... This is very inefficient, but honestly, the PDF is not
> >>>> exactly
> >>>> well formed, so if these particular files wind up being slower to
> render
> >>>> b/c they have to swap image content to disk, I think that is OK...
> >>> Maybe... the good thing is that we know when the image will be "too
> >>> big". The license is OK:
> >>>
> >>> https://www.apache.org/legal/resolved.html
> >>>
> >>> But I'm still wondering whether we have a memory leak. If we have,
> >>> then it should be fixed.
> >>>
> >>> Tilman
> >>>
> >>>
> >>>> - K
> >>>>
> >>>>
> >>>> Kevin Day
> >>>>
> >>>> *trumpet**p| *480.961.6003 x1002
> >>>> *e| *kevin@trumpetinc.com
> >>>> www.trumpetinc.com <http://trumpetinc.com/>
> >>>> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet
> Blog
> >>>> <http://trumpetinc.com/blog/>| Twitter <
> https://twitter.com/trumpetinc>
> >>>>
> >>>> Proud to be Great Place To Work
> >>>> <https://www.greatplacetowork.com/certified-company/7012667>
> Certified
> >>>> 2020-2021
> >>>>
> >>>>
> >>>> On Fri, Nov 5, 2021 at 12:47 PM Tilman Hausherr <
> THausherr@t-online.de>
> >>>> wrote:
> >>>>
> >>>>> Am 05.11.2021 um 20:38 schrieb Kevin Day:
> >>>>>> I do have a bit of experience with memory leaks (side effect of
> >>>>>> writing
> >>>>>> high performance java code, unfortunately!)
> >>>>>>
> >>>>>>
> >>>>>> I am pretty sure that this is not a memory leak, though. If this
> >>>>>> was a
> >>>>>> memory leak, we would see calls happening multiple times before
> >>>>>> failure.
> >>>>>>
> >>>>>> The failure happens on the very first call to create the buffered
> >>>>>> image:
> >>>>>>
> >>>>>> BufferedImage image = new BufferedImage(rasterWidth,
> >>>>> rasterHeight,
> >>>>>> BufferedImage.TYPE_INT_ARGB);
> >>>>>>
> >>>>>> Nothing is even getting put into the weakcache.
> >>>>>>
> >>>>> That is done in TilingPaintFactory. The cache was needed for some
> files
> >>>>> that have used the pattern several times, to avoid recreating the
> image
> >>>>> for the TexturePaint.
> >>>>>
> >>>>>
> >>>>>> I think the core issue is that the raster data for the image is
> >>>>>> just too
> >>>>>> big to store on the heap (without the heap being massive). It's
> >>>>>> 13152x13152x4 bytes (there is an alpha channel on this as well) -
> >>>>>> that's
> >>>>>> almost 700MB - just for this one pattern.
> >>>>>>
> >>>>>>
> >>>>>> Tilman wrote: "That
> >>>>>> pattern has an XStep and YStep of 23438 although the image is 2148 x
> >>>>>> 440. Because of a matrix scale of 0.0673396 the image pattern size
> is
> >>>>>> 1578 x 1578 at 72 dpi. So at 1200 dpi the size would be about 26300
> x
> >>>>>> 26300"
> >>>>>>
> >>>>>> I am fairly familiar with PDF (I contributed the parsing library to
> >>>>>> iText
> >>>>>> back in the day), but I'm not very familiar with rendering tile
> >>>>>> patterns,
> >>>>>> so the above is not intuitive to me (yet). Is there some chance
> that
> >>>>>> PdfBox is doing a lot more work than is necessary for this
> particular
> >>>>> PDF?
> >>>>>> Like is the matrix scale just absurdly bad in this file? For an
> image
> >>>>> that
> >>>>>> is originally 2148x440, it seems like requiring a 700MB raster
> >>>>>> should not
> >>>>>> be necessary - but I do NOT have nearly enough experience with this
> >>>>>> area
> >>>>> of
> >>>>>> PDF...
> >>>>> The ridiculous thing isn't the matrix, it's the XStep and YStep. This
> >>>>> goes well outside of the page. The 0.067 matrix scaling makes it
> >>>>> less bad.
> >>>>>
> >>>>> A weakness in PDFBox is that we don't readjust the tile image, or
> >>>>> use an
> >>>>> alternative painting method that doesn't use TexturePaint when the
> >>>>> image
> >>>>> would be painted only once.
> >>>>>
> >>>>> Tilman
> >>>>>
> >>>>>
> >>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> - Kevin
> >>>>>>
> >>>>>>
> >>>>>> Kevin Day
> >>>>>>
> >>>>>> *trumpet**p| *480.961.6003 x1002
> >>>>>> *e| *kevin@trumpetinc.com
> >>>>>> www.trumpetinc.com <http://trumpetinc.com/>
> >>>>>> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet
> >> Blog
> >>>>>> <http://trumpetinc.com/blog/>| Twitter
> >>>>>> <https://twitter.com/trumpetinc>
> >>>>>>
> >>>>>> Proud to be Great Place To Work
> >>>>>> <https://www.greatplacetowork.com/certified-company/7012667>
> >> Certified
> >>>>>> 2020-2021
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Nov 5, 2021 at 11:27 AM Tilman Hausherr
> >>>>>> <TH...@t-online.de>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> If you're experienced with memory leaks then it would be nice if
> you
> >>>>>>> could search.
> >>>>>>>
> >>>>>>> Things I tried that didn't help:
> >>>>>>> - calling graphics.setPaint(null) after operations (to "lose"
> >>>>>>> TilingPaint objects)
> >>>>>>> - disabling the (weak) cache of TilingPaint objects
> >>>>>>> - adding finalize to see if the TilingPaint class gets finalized
> >>>>>>> (yes)
> >>>>>>> - adding finalize to see if the HighResolutionImageIcon class gets
> >>>>>>> finalized (yes)
> >>>>>>>
> >>>>>>> Tilman
> >>>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> >>>>> For additional commands, e-mail: users-help@pdfbox.apache.org
> >>>>>
> >>>>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> >>> For additional commands, e-mail: users-help@pdfbox.apache.org
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> >> For additional commands, e-mail: users-help@pdfbox.apache.org
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> For additional commands, e-mail: users-help@pdfbox.apache.org
>
>
Re: Guidance on heap space error when printing document
Posted by Tilman Hausherr <TH...@t-online.de>.
Am 07.11.2021 um 16:09 schrieb Kevin Day:
> ok - I think that maybe there are two things going on, then. The first is
> that we are allocating a single buffer that is bigger than the default heap
> (700MB > 500MB). I really think that the only two solutions are to move
> this off the heap or reduce the size of the allocation.
>
> Then the second issue is maybe a memory leak (I have not been able to see
> this because of the first problem unless I set my max heap to a huge number
> - and that makes memory leak analysis very difficult). I am going to clone
> the PdfBox git repo now and swap that class in then do a little bit of
> profiling with jvisualvm to see if I can help find the memory leak.
>
> Question: The Git repo for PdfBox says that it is a mirror. If I do wind
> up creating a pull request against this repo, will you be able to accept it?
Not directly but we can create a .diff / .patch file from the PR.
In the meantime I ran my rendering regression checks which renders 1000
files. The hard disk C: went full after using 260 GB :-(
Tilman
>
>
> FYI - the use of sun.* classes is just to work around a long standing
> native resource leak bug in the Windows MappedByteBuffer implementation -
> On Windows, MappedByteBuffer does not release the allocated native resource
> until finalize() is called, and because the MappedByteBuffer itself does
> not take up much heap space, it can stay on the heap for quite a long time
> even after all references are gone. So the sun.* reference is just to
> force the release of those native resources. I used the same technique
> when I created the high throughput IO layer for iText. Instead of
> referencing the package directly, we can use reflection (and only use it
> for winXXX platform) - this shouldn't be a big problem.
>
> K
>
> Kevin Day
>
> *trumpet**p| *480.961.6003 x1002
> *e| *kevin@trumpetinc.com
> www.trumpetinc.com <http://trumpetinc.com/>
> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
> <http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
>
> Proud to be Great Place To Work
> <https://www.greatplacetowork.com/certified-company/7012667> certified
> since 2019
>
>
> On Sun, Nov 7, 2021 at 7:53 AM Tilman Hausherr <TH...@t-online.de>
> wrote:
>
>> I tried it and it works pretty fast. The memory leak is still there but
>> it comes later.
>>
>> Other drawbacks arethat it doesn't clean up after itself fast enough,
>> and it uses a sun.* class.
>>
>> Tilman
>>
>> Am 07.11.2021 um 14:31 schrieb Tilman Hausherr:
>>> Am 05.11.2021 um 22:09 schrieb Kevin Day:
>>>> ok - so should we be clamping the xstep in some way? Or at this
>>>> depth of
>>>> the algorithm do we not have enough context to actually tell that it
>>>> will
>>>> be outside the page/clipping region?
>>> Yes + yes, that is the problem. The context would have to include the
>>> region but also the current transformation matrix. Which could change
>>> despite using the same pattern. So it's tricky.
>>>
>>>>
>>>> I'm beginning to think that the BigBufferedImage might be the right
>>>> solution... This is very inefficient, but honestly, the PDF is not
>>>> exactly
>>>> well formed, so if these particular files wind up being slower to render
>>>> b/c they have to swap image content to disk, I think that is OK...
>>> Maybe... the good thing is that we know when the image will be "too
>>> big". The license is OK:
>>>
>>> https://www.apache.org/legal/resolved.html
>>>
>>> But I'm still wondering whether we have a memory leak. If we have,
>>> then it should be fixed.
>>>
>>> Tilman
>>>
>>>
>>>> - K
>>>>
>>>>
>>>> Kevin Day
>>>>
>>>> *trumpet**p| *480.961.6003 x1002
>>>> *e| *kevin@trumpetinc.com
>>>> www.trumpetinc.com <http://trumpetinc.com/>
>>>> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
>>>> <http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
>>>>
>>>> Proud to be Great Place To Work
>>>> <https://www.greatplacetowork.com/certified-company/7012667> Certified
>>>> 2020-2021
>>>>
>>>>
>>>> On Fri, Nov 5, 2021 at 12:47 PM Tilman Hausherr <TH...@t-online.de>
>>>> wrote:
>>>>
>>>>> Am 05.11.2021 um 20:38 schrieb Kevin Day:
>>>>>> I do have a bit of experience with memory leaks (side effect of
>>>>>> writing
>>>>>> high performance java code, unfortunately!)
>>>>>>
>>>>>>
>>>>>> I am pretty sure that this is not a memory leak, though. If this
>>>>>> was a
>>>>>> memory leak, we would see calls happening multiple times before
>>>>>> failure.
>>>>>>
>>>>>> The failure happens on the very first call to create the buffered
>>>>>> image:
>>>>>>
>>>>>> BufferedImage image = new BufferedImage(rasterWidth,
>>>>> rasterHeight,
>>>>>> BufferedImage.TYPE_INT_ARGB);
>>>>>>
>>>>>> Nothing is even getting put into the weakcache.
>>>>>>
>>>>> That is done in TilingPaintFactory. The cache was needed for some files
>>>>> that have used the pattern several times, to avoid recreating the image
>>>>> for the TexturePaint.
>>>>>
>>>>>
>>>>>> I think the core issue is that the raster data for the image is
>>>>>> just too
>>>>>> big to store on the heap (without the heap being massive). It's
>>>>>> 13152x13152x4 bytes (there is an alpha channel on this as well) -
>>>>>> that's
>>>>>> almost 700MB - just for this one pattern.
>>>>>>
>>>>>>
>>>>>> Tilman wrote: "That
>>>>>> pattern has an XStep and YStep of 23438 although the image is 2148 x
>>>>>> 440. Because of a matrix scale of 0.0673396 the image pattern size is
>>>>>> 1578 x 1578 at 72 dpi. So at 1200 dpi the size would be about 26300 x
>>>>>> 26300"
>>>>>>
>>>>>> I am fairly familiar with PDF (I contributed the parsing library to
>>>>>> iText
>>>>>> back in the day), but I'm not very familiar with rendering tile
>>>>>> patterns,
>>>>>> so the above is not intuitive to me (yet). Is there some chance that
>>>>>> PdfBox is doing a lot more work than is necessary for this particular
>>>>> PDF?
>>>>>> Like is the matrix scale just absurdly bad in this file? For an image
>>>>> that
>>>>>> is originally 2148x440, it seems like requiring a 700MB raster
>>>>>> should not
>>>>>> be necessary - but I do NOT have nearly enough experience with this
>>>>>> area
>>>>> of
>>>>>> PDF...
>>>>> The ridiculous thing isn't the matrix, it's the XStep and YStep. This
>>>>> goes well outside of the page. The 0.067 matrix scaling makes it
>>>>> less bad.
>>>>>
>>>>> A weakness in PDFBox is that we don't readjust the tile image, or
>>>>> use an
>>>>> alternative painting method that doesn't use TexturePaint when the
>>>>> image
>>>>> would be painted only once.
>>>>>
>>>>> Tilman
>>>>>
>>>>>
>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> - Kevin
>>>>>>
>>>>>>
>>>>>> Kevin Day
>>>>>>
>>>>>> *trumpet**p| *480.961.6003 x1002
>>>>>> *e| *kevin@trumpetinc.com
>>>>>> www.trumpetinc.com <http://trumpetinc.com/>
>>>>>> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet
>> Blog
>>>>>> <http://trumpetinc.com/blog/>| Twitter
>>>>>> <https://twitter.com/trumpetinc>
>>>>>>
>>>>>> Proud to be Great Place To Work
>>>>>> <https://www.greatplacetowork.com/certified-company/7012667>
>> Certified
>>>>>> 2020-2021
>>>>>>
>>>>>>
>>>>>> On Fri, Nov 5, 2021 at 11:27 AM Tilman Hausherr
>>>>>> <TH...@t-online.de>
>>>>>> wrote:
>>>>>>
>>>>>>> If you're experienced with memory leaks then it would be nice if you
>>>>>>> could search.
>>>>>>>
>>>>>>> Things I tried that didn't help:
>>>>>>> - calling graphics.setPaint(null) after operations (to "lose"
>>>>>>> TilingPaint objects)
>>>>>>> - disabling the (weak) cache of TilingPaint objects
>>>>>>> - adding finalize to see if the TilingPaint class gets finalized
>>>>>>> (yes)
>>>>>>> - adding finalize to see if the HighResolutionImageIcon class gets
>>>>>>> finalized (yes)
>>>>>>>
>>>>>>> Tilman
>>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
>>>>> For additional commands, e-mail: users-help@pdfbox.apache.org
>>>>>
>>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
>>> For additional commands, e-mail: users-help@pdfbox.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
>> For additional commands, e-mail: users-help@pdfbox.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
For additional commands, e-mail: users-help@pdfbox.apache.org
Re: Guidance on heap space error when printing document
Posted by Kevin Day <ke...@trumpetinc.com>.
ok - I think that maybe there are two things going on, then. The first is
that we are allocating a single buffer that is bigger than the default heap
(700MB > 500MB). I really think that the only two solutions are to move
this off the heap or reduce the size of the allocation.
Then the second issue is maybe a memory leak (I have not been able to see
this because of the first problem unless I set my max heap to a huge number
- and that makes memory leak analysis very difficult). I am going to clone
the PdfBox git repo now and swap that class in then do a little bit of
profiling with jvisualvm to see if I can help find the memory leak.
Question: The Git repo for PdfBox says that it is a mirror. If I do wind
up creating a pull request against this repo, will you be able to accept it?
FYI - the use of sun.* classes is just to work around a long standing
native resource leak bug in the Windows MappedByteBuffer implementation -
On Windows, MappedByteBuffer does not release the allocated native resource
until finalize() is called, and because the MappedByteBuffer itself does
not take up much heap space, it can stay on the heap for quite a long time
even after all references are gone. So the sun.* reference is just to
force the release of those native resources. I used the same technique
when I created the high throughput IO layer for iText. Instead of
referencing the package directly, we can use reflection (and only use it
for winXXX platform) - this shouldn't be a big problem.
K
Kevin Day
*trumpet**p| *480.961.6003 x1002
*e| *kevin@trumpetinc.com
www.trumpetinc.com <http://trumpetinc.com/>
LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
<http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
Proud to be Great Place To Work
<https://www.greatplacetowork.com/certified-company/7012667> certified
since 2019
On Sun, Nov 7, 2021 at 7:53 AM Tilman Hausherr <TH...@t-online.de>
wrote:
> I tried it and it works pretty fast. The memory leak is still there but
> it comes later.
>
> Other drawbacks arethat it doesn't clean up after itself fast enough,
> and it uses a sun.* class.
>
> Tilman
>
> Am 07.11.2021 um 14:31 schrieb Tilman Hausherr:
> > Am 05.11.2021 um 22:09 schrieb Kevin Day:
> >> ok - so should we be clamping the xstep in some way? Or at this
> >> depth of
> >> the algorithm do we not have enough context to actually tell that it
> >> will
> >> be outside the page/clipping region?
> >
> > Yes + yes, that is the problem. The context would have to include the
> > region but also the current transformation matrix. Which could change
> > despite using the same pattern. So it's tricky.
> >
> >>
> >>
> >> I'm beginning to think that the BigBufferedImage might be the right
> >> solution... This is very inefficient, but honestly, the PDF is not
> >> exactly
> >> well formed, so if these particular files wind up being slower to render
> >> b/c they have to swap image content to disk, I think that is OK...
> >
> > Maybe... the good thing is that we know when the image will be "too
> > big". The license is OK:
> >
> > https://www.apache.org/legal/resolved.html
> >
> > But I'm still wondering whether we have a memory leak. If we have,
> > then it should be fixed.
> >
> > Tilman
> >
> >
> >>
> >> - K
> >>
> >>
> >> Kevin Day
> >>
> >> *trumpet**p| *480.961.6003 x1002
> >> *e| *kevin@trumpetinc.com
> >> www.trumpetinc.com <http://trumpetinc.com/>
> >> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
> >> <http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
> >>
> >> Proud to be Great Place To Work
> >> <https://www.greatplacetowork.com/certified-company/7012667> Certified
> >> 2020-2021
> >>
> >>
> >> On Fri, Nov 5, 2021 at 12:47 PM Tilman Hausherr <TH...@t-online.de>
> >> wrote:
> >>
> >>> Am 05.11.2021 um 20:38 schrieb Kevin Day:
> >>>> I do have a bit of experience with memory leaks (side effect of
> >>>> writing
> >>>> high performance java code, unfortunately!)
> >>>>
> >>>>
> >>>> I am pretty sure that this is not a memory leak, though. If this
> >>>> was a
> >>>> memory leak, we would see calls happening multiple times before
> >>>> failure.
> >>>>
> >>>> The failure happens on the very first call to create the buffered
> >>>> image:
> >>>>
> >>>> BufferedImage image = new BufferedImage(rasterWidth,
> >>> rasterHeight,
> >>>> BufferedImage.TYPE_INT_ARGB);
> >>>>
> >>>> Nothing is even getting put into the weakcache.
> >>>>
> >>> That is done in TilingPaintFactory. The cache was needed for some files
> >>> that have used the pattern several times, to avoid recreating the image
> >>> for the TexturePaint.
> >>>
> >>>
> >>>> I think the core issue is that the raster data for the image is
> >>>> just too
> >>>> big to store on the heap (without the heap being massive). It's
> >>>> 13152x13152x4 bytes (there is an alpha channel on this as well) -
> >>>> that's
> >>>> almost 700MB - just for this one pattern.
> >>>>
> >>>>
> >>>> Tilman wrote: "That
> >>>> pattern has an XStep and YStep of 23438 although the image is 2148 x
> >>>> 440. Because of a matrix scale of 0.0673396 the image pattern size is
> >>>> 1578 x 1578 at 72 dpi. So at 1200 dpi the size would be about 26300 x
> >>>> 26300"
> >>>>
> >>>> I am fairly familiar with PDF (I contributed the parsing library to
> >>>> iText
> >>>> back in the day), but I'm not very familiar with rendering tile
> >>>> patterns,
> >>>> so the above is not intuitive to me (yet). Is there some chance that
> >>>> PdfBox is doing a lot more work than is necessary for this particular
> >>> PDF?
> >>>> Like is the matrix scale just absurdly bad in this file? For an image
> >>> that
> >>>> is originally 2148x440, it seems like requiring a 700MB raster
> >>>> should not
> >>>> be necessary - but I do NOT have nearly enough experience with this
> >>>> area
> >>> of
> >>>> PDF...
> >>> The ridiculous thing isn't the matrix, it's the XStep and YStep. This
> >>> goes well outside of the page. The 0.067 matrix scaling makes it
> >>> less bad.
> >>>
> >>> A weakness in PDFBox is that we don't readjust the tile image, or
> >>> use an
> >>> alternative painting method that doesn't use TexturePaint when the
> >>> image
> >>> would be painted only once.
> >>>
> >>> Tilman
> >>>
> >>>
> >>>
> >>>>
> >>>> Thanks,
> >>>>
> >>>> - Kevin
> >>>>
> >>>>
> >>>> Kevin Day
> >>>>
> >>>> *trumpet**p| *480.961.6003 x1002
> >>>> *e| *kevin@trumpetinc.com
> >>>> www.trumpetinc.com <http://trumpetinc.com/>
> >>>> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet
> Blog
> >>>> <http://trumpetinc.com/blog/>| Twitter
> >>>> <https://twitter.com/trumpetinc>
> >>>>
> >>>> Proud to be Great Place To Work
> >>>> <https://www.greatplacetowork.com/certified-company/7012667>
> Certified
> >>>> 2020-2021
> >>>>
> >>>>
> >>>> On Fri, Nov 5, 2021 at 11:27 AM Tilman Hausherr
> >>>> <TH...@t-online.de>
> >>>> wrote:
> >>>>
> >>>>> If you're experienced with memory leaks then it would be nice if you
> >>>>> could search.
> >>>>>
> >>>>> Things I tried that didn't help:
> >>>>> - calling graphics.setPaint(null) after operations (to "lose"
> >>>>> TilingPaint objects)
> >>>>> - disabling the (weak) cache of TilingPaint objects
> >>>>> - adding finalize to see if the TilingPaint class gets finalized
> >>>>> (yes)
> >>>>> - adding finalize to see if the HighResolutionImageIcon class gets
> >>>>> finalized (yes)
> >>>>>
> >>>>> Tilman
> >>>>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> >>> For additional commands, e-mail: users-help@pdfbox.apache.org
> >>>
> >>>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> > For additional commands, e-mail: users-help@pdfbox.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> For additional commands, e-mail: users-help@pdfbox.apache.org
>
>
Re: Guidance on heap space error when printing document
Posted by Tilman Hausherr <TH...@t-online.de>.
I tried it and it works pretty fast. The memory leak is still there but
it comes later.
Other drawbacks arethat it doesn't clean up after itself fast enough,
and it uses a sun.* class.
Tilman
Am 07.11.2021 um 14:31 schrieb Tilman Hausherr:
> Am 05.11.2021 um 22:09 schrieb Kevin Day:
>> ok - so should we be clamping the xstep in some way? Or at this
>> depth of
>> the algorithm do we not have enough context to actually tell that it
>> will
>> be outside the page/clipping region?
>
> Yes + yes, that is the problem. The context would have to include the
> region but also the current transformation matrix. Which could change
> despite using the same pattern. So it's tricky.
>
>>
>>
>> I'm beginning to think that the BigBufferedImage might be the right
>> solution... This is very inefficient, but honestly, the PDF is not
>> exactly
>> well formed, so if these particular files wind up being slower to render
>> b/c they have to swap image content to disk, I think that is OK...
>
> Maybe... the good thing is that we know when the image will be "too
> big". The license is OK:
>
> https://www.apache.org/legal/resolved.html
>
> But I'm still wondering whether we have a memory leak. If we have,
> then it should be fixed.
>
> Tilman
>
>
>>
>> - K
>>
>>
>> Kevin Day
>>
>> *trumpet**p| *480.961.6003 x1002
>> *e| *kevin@trumpetinc.com
>> www.trumpetinc.com <http://trumpetinc.com/>
>> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
>> <http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
>>
>> Proud to be Great Place To Work
>> <https://www.greatplacetowork.com/certified-company/7012667> Certified
>> 2020-2021
>>
>>
>> On Fri, Nov 5, 2021 at 12:47 PM Tilman Hausherr <TH...@t-online.de>
>> wrote:
>>
>>> Am 05.11.2021 um 20:38 schrieb Kevin Day:
>>>> I do have a bit of experience with memory leaks (side effect of
>>>> writing
>>>> high performance java code, unfortunately!)
>>>>
>>>>
>>>> I am pretty sure that this is not a memory leak, though. If this
>>>> was a
>>>> memory leak, we would see calls happening multiple times before
>>>> failure.
>>>>
>>>> The failure happens on the very first call to create the buffered
>>>> image:
>>>>
>>>> Â Â Â Â Â Â Â Â Â BufferedImage image = new BufferedImage(rasterWidth,
>>> rasterHeight,
>>>> BufferedImage.TYPE_INT_ARGB);
>>>>
>>>> Nothing is even getting put into the weakcache.
>>>>
>>> That is done in TilingPaintFactory. The cache was needed for some files
>>> that have used the pattern several times, to avoid recreating the image
>>> for the TexturePaint.
>>>
>>>
>>>> I think the core issue is that the raster data for the image is
>>>> just too
>>>> big to store on the heap (without the heap being massive). It's
>>>> 13152x13152x4 bytes (there is an alpha channel on this as well) -
>>>> that's
>>>> almost 700MB - just for this one pattern.
>>>>
>>>>
>>>> Tilman wrote:Â "That
>>>> pattern has an XStep and YStep of 23438 although the image is 2148 x
>>>> 440. Because of a matrix scale of 0.0673396 the image pattern size is
>>>> 1578 x 1578 at 72 dpi. So at 1200 dpi the size would be about 26300 x
>>>> 26300"
>>>>
>>>> I am fairly familiar with PDF (I contributed the parsing library to
>>>> iText
>>>> back in the day), but I'm not very familiar with rendering tile
>>>> patterns,
>>>> so the above is not intuitive to me (yet). Is there some chance that
>>>> PdfBox is doing a lot more work than is necessary for this particular
>>> PDF?
>>>> Like is the matrix scale just absurdly bad in this file? For an image
>>> that
>>>> is originally 2148x440, it seems like requiring a 700MB raster
>>>> should not
>>>> be necessary - but I do NOT have nearly enough experience with this
>>>> area
>>> of
>>>> PDF...
>>> The ridiculous thing isn't the matrix, it's the XStep and YStep. This
>>> goes well outside of the page. The 0.067 matrix scaling makes it
>>> less bad.
>>>
>>> A weakness in PDFBox is that we don't readjust the tile image, or
>>> use an
>>> alternative painting method that doesn't use TexturePaint when the
>>> image
>>> would be painted only once.
>>>
>>> Tilman
>>>
>>>
>>>
>>>>
>>>> Thanks,
>>>>
>>>> - Kevin
>>>>
>>>>
>>>> Kevin Day
>>>>
>>>> *trumpet**p| *480.961.6003 x1002
>>>> *e| *kevin@trumpetinc.com
>>>> www.trumpetinc.com <http://trumpetinc.com/>
>>>> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
>>>> <http://trumpetinc.com/blog/>| Twitter
>>>> <https://twitter.com/trumpetinc>
>>>>
>>>> Proud to be Great Place To Work
>>>> <https://www.greatplacetowork.com/certified-company/7012667> Certified
>>>> 2020-2021
>>>>
>>>>
>>>> On Fri, Nov 5, 2021 at 11:27 AM Tilman Hausherr
>>>> <TH...@t-online.de>
>>>> wrote:
>>>>
>>>>> If you're experienced with memory leaks then it would be nice if you
>>>>> could search.
>>>>>
>>>>> Things I tried that didn't help:
>>>>> - calling graphics.setPaint(null) after operations (to "lose"
>>>>> TilingPaint objects)
>>>>> - disabling the (weak) cache of TilingPaint objects
>>>>> - adding finalize to see if the TilingPaint class gets finalized
>>>>> (yes)
>>>>> - adding finalize to see if the HighResolutionImageIcon class gets
>>>>> finalized (yes)
>>>>>
>>>>> Tilman
>>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
>>> For additional commands, e-mail: users-help@pdfbox.apache.org
>>>
>>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> For additional commands, e-mail: users-help@pdfbox.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
For additional commands, e-mail: users-help@pdfbox.apache.org
Re: Guidance on heap space error when printing document
Posted by Tilman Hausherr <TH...@t-online.de>.
Am 05.11.2021 um 22:09 schrieb Kevin Day:
> ok - so should we be clamping the xstep in some way? Or at this depth of
> the algorithm do we not have enough context to actually tell that it will
> be outside the page/clipping region?
Yes + yes, that is the problem. The context would have to include the
region but also the current transformation matrix. Which could change
despite using the same pattern. So it's tricky.
>
>
> I'm beginning to think that the BigBufferedImage might be the right
> solution... This is very inefficient, but honestly, the PDF is not exactly
> well formed, so if these particular files wind up being slower to render
> b/c they have to swap image content to disk, I think that is OK...
Maybe... the good thing is that we know when the image will be "too
big". The license is OK:
https://www.apache.org/legal/resolved.html
But I'm still wondering whether we have a memory leak. If we have, then
it should be fixed.
Tilman
>
> - K
>
>
> Kevin Day
>
> *trumpet**p| *480.961.6003 x1002
> *e| *kevin@trumpetinc.com
> www.trumpetinc.com <http://trumpetinc.com/>
> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
> <http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
>
> Proud to be Great Place To Work
> <https://www.greatplacetowork.com/certified-company/7012667> Certified
> 2020-2021
>
>
> On Fri, Nov 5, 2021 at 12:47 PM Tilman Hausherr <TH...@t-online.de>
> wrote:
>
>> Am 05.11.2021 um 20:38 schrieb Kevin Day:
>>> I do have a bit of experience with memory leaks (side effect of writing
>>> high performance java code, unfortunately!)
>>>
>>>
>>> I am pretty sure that this is not a memory leak, though. If this was a
>>> memory leak, we would see calls happening multiple times before failure.
>>>
>>> The failure happens on the very first call to create the buffered image:
>>>
>>> BufferedImage image = new BufferedImage(rasterWidth,
>> rasterHeight,
>>> BufferedImage.TYPE_INT_ARGB);
>>>
>>> Nothing is even getting put into the weakcache.
>>>
>> That is done in TilingPaintFactory. The cache was needed for some files
>> that have used the pattern several times, to avoid recreating the image
>> for the TexturePaint.
>>
>>
>>> I think the core issue is that the raster data for the image is just too
>>> big to store on the heap (without the heap being massive). It's
>>> 13152x13152x4 bytes (there is an alpha channel on this as well) - that's
>>> almost 700MB - just for this one pattern.
>>>
>>>
>>> Tilman wrote: "That
>>> pattern has an XStep and YStep of 23438 although the image is 2148 x
>>> 440. Because of a matrix scale of 0.0673396 the image pattern size is
>>> 1578 x 1578 at 72 dpi. So at 1200 dpi the size would be about 26300 x
>>> 26300"
>>>
>>> I am fairly familiar with PDF (I contributed the parsing library to iText
>>> back in the day), but I'm not very familiar with rendering tile patterns,
>>> so the above is not intuitive to me (yet). Is there some chance that
>>> PdfBox is doing a lot more work than is necessary for this particular
>> PDF?
>>> Like is the matrix scale just absurdly bad in this file? For an image
>> that
>>> is originally 2148x440, it seems like requiring a 700MB raster should not
>>> be necessary - but I do NOT have nearly enough experience with this area
>> of
>>> PDF...
>> The ridiculous thing isn't the matrix, it's the XStep and YStep. This
>> goes well outside of the page. The 0.067 matrix scaling makes it less bad.
>>
>> A weakness in PDFBox is that we don't readjust the tile image, or use an
>> alternative painting method that doesn't use TexturePaint when the image
>> would be painted only once.
>>
>> Tilman
>>
>>
>>
>>>
>>> Thanks,
>>>
>>> - Kevin
>>>
>>>
>>> Kevin Day
>>>
>>> *trumpet**p| *480.961.6003 x1002
>>> *e| *kevin@trumpetinc.com
>>> www.trumpetinc.com <http://trumpetinc.com/>
>>> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
>>> <http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
>>>
>>> Proud to be Great Place To Work
>>> <https://www.greatplacetowork.com/certified-company/7012667> Certified
>>> 2020-2021
>>>
>>>
>>> On Fri, Nov 5, 2021 at 11:27 AM Tilman Hausherr <TH...@t-online.de>
>>> wrote:
>>>
>>>> If you're experienced with memory leaks then it would be nice if you
>>>> could search.
>>>>
>>>> Things I tried that didn't help:
>>>> - calling graphics.setPaint(null) after operations (to "lose"
>>>> TilingPaint objects)
>>>> - disabling the (weak) cache of TilingPaint objects
>>>> - adding finalize to see if the TilingPaint class gets finalized (yes)
>>>> - adding finalize to see if the HighResolutionImageIcon class gets
>>>> finalized (yes)
>>>>
>>>> Tilman
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
>> For additional commands, e-mail: users-help@pdfbox.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
For additional commands, e-mail: users-help@pdfbox.apache.org
Re: Guidance on heap space error when printing document
Posted by Kevin Day <ke...@trumpetinc.com>.
ok - so should we be clamping the xstep in some way? Or at this depth of
the algorithm do we not have enough context to actually tell that it will
be outside the page/clipping region?
I'm beginning to think that the BigBufferedImage might be the right
solution... This is very inefficient, but honestly, the PDF is not exactly
well formed, so if these particular files wind up being slower to render
b/c they have to swap image content to disk, I think that is OK...
- K
Kevin Day
*trumpet**p| *480.961.6003 x1002
*e| *kevin@trumpetinc.com
www.trumpetinc.com <http://trumpetinc.com/>
LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
<http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
Proud to be Great Place To Work
<https://www.greatplacetowork.com/certified-company/7012667> Certified
2020-2021
On Fri, Nov 5, 2021 at 12:47 PM Tilman Hausherr <TH...@t-online.de>
wrote:
> Am 05.11.2021 um 20:38 schrieb Kevin Day:
> > I do have a bit of experience with memory leaks (side effect of writing
> > high performance java code, unfortunately!)
> >
> >
> > I am pretty sure that this is not a memory leak, though. If this was a
> > memory leak, we would see calls happening multiple times before failure.
> >
> > The failure happens on the very first call to create the buffered image:
> >
> > BufferedImage image = new BufferedImage(rasterWidth,
> rasterHeight,
> > BufferedImage.TYPE_INT_ARGB);
> >
> > Nothing is even getting put into the weakcache.
> >
> That is done in TilingPaintFactory. The cache was needed for some files
> that have used the pattern several times, to avoid recreating the image
> for the TexturePaint.
>
>
> > I think the core issue is that the raster data for the image is just too
> > big to store on the heap (without the heap being massive). It's
> > 13152x13152x4 bytes (there is an alpha channel on this as well) - that's
> > almost 700MB - just for this one pattern.
> >
> >
> > Tilman wrote: "That
> > pattern has an XStep and YStep of 23438 although the image is 2148 x
> > 440. Because of a matrix scale of 0.0673396 the image pattern size is
> > 1578 x 1578 at 72 dpi. So at 1200 dpi the size would be about 26300 x
> > 26300"
> >
> > I am fairly familiar with PDF (I contributed the parsing library to iText
> > back in the day), but I'm not very familiar with rendering tile patterns,
> > so the above is not intuitive to me (yet). Is there some chance that
> > PdfBox is doing a lot more work than is necessary for this particular
> PDF?
> > Like is the matrix scale just absurdly bad in this file? For an image
> that
> > is originally 2148x440, it seems like requiring a 700MB raster should not
> > be necessary - but I do NOT have nearly enough experience with this area
> of
> > PDF...
>
> The ridiculous thing isn't the matrix, it's the XStep and YStep. This
> goes well outside of the page. The 0.067 matrix scaling makes it less bad.
>
> A weakness in PDFBox is that we don't readjust the tile image, or use an
> alternative painting method that doesn't use TexturePaint when the image
> would be painted only once.
>
> Tilman
>
>
>
> >
> >
> > Thanks,
> >
> > - Kevin
> >
> >
> > Kevin Day
> >
> > *trumpet**p| *480.961.6003 x1002
> > *e| *kevin@trumpetinc.com
> > www.trumpetinc.com <http://trumpetinc.com/>
> > LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
> > <http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
> >
> > Proud to be Great Place To Work
> > <https://www.greatplacetowork.com/certified-company/7012667> Certified
> > 2020-2021
> >
> >
> > On Fri, Nov 5, 2021 at 11:27 AM Tilman Hausherr <TH...@t-online.de>
> > wrote:
> >
> >> If you're experienced with memory leaks then it would be nice if you
> >> could search.
> >>
> >> Things I tried that didn't help:
> >> - calling graphics.setPaint(null) after operations (to "lose"
> >> TilingPaint objects)
> >> - disabling the (weak) cache of TilingPaint objects
> >> - adding finalize to see if the TilingPaint class gets finalized (yes)
> >> - adding finalize to see if the HighResolutionImageIcon class gets
> >> finalized (yes)
> >>
> >> Tilman
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> For additional commands, e-mail: users-help@pdfbox.apache.org
>
>
Re: Guidance on heap space error when printing document
Posted by Tilman Hausherr <TH...@t-online.de>.
Am 05.11.2021 um 20:38 schrieb Kevin Day:
> I do have a bit of experience with memory leaks (side effect of writing
> high performance java code, unfortunately!)
>
>
> I am pretty sure that this is not a memory leak, though. If this was a
> memory leak, we would see calls happening multiple times before failure.
>
> The failure happens on the very first call to create the buffered image:
>
> BufferedImage image = new BufferedImage(rasterWidth, rasterHeight,
> BufferedImage.TYPE_INT_ARGB);
>
> Nothing is even getting put into the weakcache.
>
That is done in TilingPaintFactory. The cache was needed for some files
that have used the pattern several times, to avoid recreating the image
for the TexturePaint.
> I think the core issue is that the raster data for the image is just too
> big to store on the heap (without the heap being massive). It's
> 13152x13152x4 bytes (there is an alpha channel on this as well) - that's
> almost 700MB - just for this one pattern.
>
>
> Tilman wrote: "That
> pattern has an XStep and YStep of 23438 although the image is 2148 x
> 440. Because of a matrix scale of 0.0673396 the image pattern size is
> 1578 x 1578 at 72 dpi. So at 1200 dpi the size would be about 26300 x
> 26300"
>
> I am fairly familiar with PDF (I contributed the parsing library to iText
> back in the day), but I'm not very familiar with rendering tile patterns,
> so the above is not intuitive to me (yet). Is there some chance that
> PdfBox is doing a lot more work than is necessary for this particular PDF?
> Like is the matrix scale just absurdly bad in this file? For an image that
> is originally 2148x440, it seems like requiring a 700MB raster should not
> be necessary - but I do NOT have nearly enough experience with this area of
> PDF...
The ridiculous thing isn't the matrix, it's the XStep and YStep. This
goes well outside of the page. The 0.067 matrix scaling makes it less bad.
A weakness in PDFBox is that we don't readjust the tile image, or use an
alternative painting method that doesn't use TexturePaint when the image
would be painted only once.
Tilman
>
>
> Thanks,
>
> - Kevin
>
>
> Kevin Day
>
> *trumpet**p| *480.961.6003 x1002
> *e| *kevin@trumpetinc.com
> www.trumpetinc.com <http://trumpetinc.com/>
> LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
> <http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
>
> Proud to be Great Place To Work
> <https://www.greatplacetowork.com/certified-company/7012667> Certified
> 2020-2021
>
>
> On Fri, Nov 5, 2021 at 11:27 AM Tilman Hausherr <TH...@t-online.de>
> wrote:
>
>> If you're experienced with memory leaks then it would be nice if you
>> could search.
>>
>> Things I tried that didn't help:
>> - calling graphics.setPaint(null) after operations (to "lose"
>> TilingPaint objects)
>> - disabling the (weak) cache of TilingPaint objects
>> - adding finalize to see if the TilingPaint class gets finalized (yes)
>> - adding finalize to see if the HighResolutionImageIcon class gets
>> finalized (yes)
>>
>> Tilman
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
For additional commands, e-mail: users-help@pdfbox.apache.org
Re: Guidance on heap space error when printing document
Posted by Kevin Day <ke...@trumpetinc.com>.
I do have a bit of experience with memory leaks (side effect of writing
high performance java code, unfortunately!)
I am pretty sure that this is not a memory leak, though. If this was a
memory leak, we would see calls happening multiple times before failure.
The failure happens on the very first call to create the buffered image:
BufferedImage image = new BufferedImage(rasterWidth, rasterHeight,
BufferedImage.TYPE_INT_ARGB);
Nothing is even getting put into the weakcache.
I think the core issue is that the raster data for the image is just too
big to store on the heap (without the heap being massive). It's
13152x13152x4 bytes (there is an alpha channel on this as well) - that's
almost 700MB - just for this one pattern.
Tilman wrote: "That
pattern has an XStep and YStep of 23438 although the image is 2148 x
440. Because of a matrix scale of 0.0673396 the image pattern size is
1578 x 1578 at 72 dpi. So at 1200 dpi the size would be about 26300 x
26300"
I am fairly familiar with PDF (I contributed the parsing library to iText
back in the day), but I'm not very familiar with rendering tile patterns,
so the above is not intuitive to me (yet). Is there some chance that
PdfBox is doing a lot more work than is necessary for this particular PDF?
Like is the matrix scale just absurdly bad in this file? For an image that
is originally 2148x440, it seems like requiring a 700MB raster should not
be necessary - but I do NOT have nearly enough experience with this area of
PDF...
Thanks,
- Kevin
Kevin Day
*trumpet**p| *480.961.6003 x1002
*e| *kevin@trumpetinc.com
www.trumpetinc.com <http://trumpetinc.com/>
LinkedIn <https://www.linkedin.com/company/trumpet-inc.>| Trumpet Blog
<http://trumpetinc.com/blog/>| Twitter <https://twitter.com/trumpetinc>
Proud to be Great Place To Work
<https://www.greatplacetowork.com/certified-company/7012667> Certified
2020-2021
On Fri, Nov 5, 2021 at 11:27 AM Tilman Hausherr <TH...@t-online.de>
wrote:
> If you're experienced with memory leaks then it would be nice if you
> could search.
>
> Things I tried that didn't help:
> - calling graphics.setPaint(null) after operations (to "lose"
> TilingPaint objects)
> - disabling the (weak) cache of TilingPaint objects
> - adding finalize to see if the TilingPaint class gets finalized (yes)
> - adding finalize to see if the HighResolutionImageIcon class gets
> finalized (yes)
>
> Tilman
>
>
>
> Am 05.11.2021 um 06:37 schrieb Kevin Day:
> > Thanks for the analysis...
> >
> > So, how do we proceed? We are dealing with lots of different files from
> > lots of different sources, so custom tuning maxedge makes me nervous.
> >
> >
> > Is it worth trying BigBufferedImage? (I.e. would this even be accepted
> into
> > the code base?) "Effective Handling of Big Images in Java"
> >
> http://jobcardsystems.com/index.php/blog/29-effective-handling-of-big-images-in-java
> .
> > - license is Creative Commons CCO. If licensing is a problem, then
> > providing a hook to allow the BufferedImage creation to be overridden
> would
> > be needed.
> >
> >
> > Or should we be trying to profile to track down a memory leak?
> >
> > Awhile back, I dealt with native memory leaks in Java MappedByteBuffer
> > under windows, but this was a native resource leak, and that doesn't seem
> > like what we have here. And I don't know what the backing data store in
> > ByteBuffer actually is.
> >
> > K
> >
> >
> > On Thu, Nov 4, 2021, 9:52 PM Tilman Hausherr <TH...@t-online.de>
> wrote:
> >
> >> Thanks for the files.
> >>
> >> I wonder if there is some memory leak, in PDFBox or in java itself. The
> >> H21 file has a pattern that contains an image (the company logo). That
> >> pattern has an XStep and YStep of 23438 although the image is 2148 x
> >> 440. Because of a matrix scale of 0.0673396 the image pattern size is
> >> 1578 x 1578 at 72 dpi. So at 1200 dpi the size would be about 26300 x
> >> 26300.
> >>
> >> You can limit this by setting pdfbox.rendering.tilingpaint.maxedge to a
> >> smaller value than the default of 3000. However this may have a bad
> >> influence on other files.
> >>
> >> I tried to show the H21 file in PDFDebugger at 1000% which would be 720
> >> dpi. This worked until the last image. But no problem when displaying
> >> the last image first. That is why I suspect a memory leak. I remember we
> >> noticed something similar years ago, BufferedImage objects that stayed
> >> despite nobody using them.
> >>
> >> Tilman
> >>
> >> Am 02.11.2021 um 20:06 schrieb Kevin Day:
> >>> Sorry it took so long to respond - I had to get permission to share
> >> these.
> >>> I did test with 2.0.24 and it did not help.
> >>>
> >>>
> >>> Here are some files that cause the problem:
> >>>
> >>>
> https://github.com/trumpetinc/boottest/blob/main/JrachvUniverse_H21.pdf
> >>>
> >>
> https://github.com/trumpetinc/boottest/blob/main/PP%20Opera%20Building_A53.pdf
> >>>
> >>> Here is some quick code that demonstrates the problem (update for your
> >>> printer name and PDF path):
> >>>
> >>>
> >>
> https://github.com/trumpetinc/boottest/blob/main/SimplifiedPdfBoxPdfPrinter.java
> >>>
> >>>
> >>> Maybe in TilingPaint#getImage(PageDrawer, PDTilingPattern,
> PDColorSpace,
> >>> PDColor, AffineTransform, Rectangle2D), the BufferedImage could
> >>> benefit from something similar to BigBufferedImage described in this
> >> post?
> >>> (
> >>>
> >>
> https://stackoverflow.com/questions/3854200/why-does-a-bufferedimage-require-so-much-memory-beyond-the-size-of-its-data-arra
> >>> ). Or maybe there is something in the scaling logic?
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Thu, Oct 28, 2021 at 11:12 AM Tilman Hausherr <
> THausherr@t-online.de>
> >>> wrote:
> >>>
> >>>> Huge buffered images can happen in patterns if the pattern matrix
> and/or
> >>>> the current transformation matrix result in a huge scale.
> >>>>
> >>>> You could retry with 2.0.24, but I'm pessimistic. Please share the PDF
> >>>> (upload to a sharehoster).
> >>>>
> >>>> Tilman
> >>>>
> >>>> Am 28.10.2021 um 01:17 schrieb Kevin Day:
> >>>>> Using PDFBox 2.0.20
> >>>>>
> >>>>> We use the following to print the PDF:
> >>>>>
> >>>>> HashPrintRequestAttributeSet attributes = new
> >>>>> HashPrintRequestAttributeSet();
> >>>>>
> >>>>> try(PDDocument doc = PDDocument.load(pdf,
> >>>>> MemoryUsageSetting.setupMixed(5000000L))){
> >>>>> job.setPageable(new PDFPageable(doc));
> >>>>> job.print(attributes);
> >>>>> }
> >>>>>
> >>>>> This works really well for thousands of PDFs, but we've run into one
> >>>>> particular PDF that causes an OutOfMemoryException.
> >>>>>
> >>>>>
> >>>>> The problem PDF is rendered (does not contain bitmaps).
> >>>>>
> >>>>> I've provided a huge amount of heap (over 1GB now), and it is still
> >>>> failing.
> >>>>> As near as I can tell from the stack trace (which I'll include
> below),
> >> it
> >>>>> seems like the problem is with creation of a huge buffered image.
> >>>>>
> >>>>> The physical page size in the PDF is 8.5 x 11".
> >>>>>
> >>>>> The PDFPrintable is configured as follows:
> >>>>>
> >>>>> Scaling is ACTUAL_SIZE
> >>>>> dpi is 0.0
> >>>>> subsamplingAllowed is false
> >>>>> renderingHints is null
> >>>>>
> >>>>>
> >>>>> Here is the stack trace:
> >>>>>
> >>>>> java.lang.OutOfMemoryError: Java heap space
> >>>>> at
> >>>>
> java.desktop/java.awt.image.DataBufferInt.<init>(DataBufferInt.java:75)
> >>>>> at
> >> java.desktop/java.awt.image.Raster.createPackedRaster(Raster.java:467)
> >>>>> at
> >>>>>
> >>
> java.desktop/java.awt.image.DirectColorModel.createCompatibleWritableRaster(DirectColorModel.java:1032)
> >>>>> at
> >>>>
> java.desktop/java.awt.image.BufferedImage.<init>(BufferedImage.java:333)
> >>>>> at
> >> org.apache.pdfbox.rendering.TilingPaint.getImage(TilingPaint.java:143)
> >>>>> at
> org.apache.pdfbox.rendering.TilingPaint.<init>(TilingPaint.java:103)
> >>>>> at
> >>>>>
> >>
> org.apache.pdfbox.rendering.TilingPaintFactory.create(TilingPaintFactory.java:60)
> >>>>> at
> org.apache.pdfbox.rendering.PageDrawer.getPaint(PageDrawer.java:351)
> >>>>> at
> >>>>>
> >>
> org.apache.pdfbox.rendering.PageDrawer.getNonStrokingPaint(PageDrawer.java:719)
> >>>>> at
> org.apache.pdfbox.rendering.PageDrawer.fillPath(PageDrawer.java:819)
> >>>>> at
> >>>>>
> >>
> org.apache.pdfbox.contentstream.operator.graphics.FillEvenOddRule.process(FillEvenOddRule.java:37)
> >>>>> at
> >>>>>
> >>
> org.apache.pdfbox.contentstream.PDFStreamEngine.processOperator(PDFStreamEngine.java:932)
> >>>>> at
> >>>>>
> >>
> org.apache.pdfbox.contentstream.PDFStreamEngine.processStreamOperators(PDFStreamEngine.java:510)
> >>>>> at
> >>>>>
> >>
> org.apache.pdfbox.contentstream.PDFStreamEngine.processStream(PDFStreamEngine.java:484)
> >>>>> at
> >>>>>
> >>
> org.apache.pdfbox.contentstream.PDFStreamEngine.showForm(PDFStreamEngine.java:187)
> >>>>> at
> >> org.apache.pdfbox.rendering.PageDrawer.showForm(PageDrawer.java:1462)
> >>>>> at
> >>>>>
> >>
> org.apache.pdfbox.contentstream.operator.graphics.DrawObject.process(DrawObject.java:86)
> >>>>> at
> >>>>>
> >>
> org.apache.pdfbox.contentstream.PDFStreamEngine.processOperator(PDFStreamEngine.java:932)
> >>>>> at
> >>>>>
> >>
> org.apache.pdfbox.contentstream.PDFStreamEngine.processStreamOperators(PDFStreamEngine.java:510)
> >>>>> at
> >>>>>
> >>
> org.apache.pdfbox.contentstream.PDFStreamEngine.processStream(PDFStreamEngine.java:484)
> >>>>> at
> >>>>>
> >>
> org.apache.pdfbox.contentstream.PDFStreamEngine.processPage(PDFStreamEngine.java:156)
> >>>>> at
> org.apache.pdfbox.rendering.PageDrawer.drawPage(PageDrawer.java:271)
> >>>>> at
> >>>>>
> >>
> org.apache.pdfbox.rendering.PDFRenderer.renderPageToGraphics(PDFRenderer.java:431)
> >>>>> at
> org.apache.pdfbox.printing.PDFPrintable.print(PDFPrintable.java:246)
> >>>>> at
> >>>>>
> >>
> java.desktop/sun.print.RasterPrinterJob.printPage(RasterPrinterJob.java:2430)
> >>>>> at
> >>
> java.desktop/sun.print.RasterPrinterJob.print(RasterPrinterJob.java:1654)
> >>>>>
> >>>>> Next, I tried setting subsampling to true - this made no difference.
> >>>>>
> >>>>>
> >>>>> Next, I tried hard coding the dpi to 300f - this works with no
> >> problems.
> >>>>>
> >>>>> Next, I tried hard coding the dpi to 600f - this does not throw the
> >> heap
> >>>>> space error - but with strange results:
> >>>>> - I see "java.lang.OutOfMemoryError" on sys.err - it does NOT throw
> an
> >>>>> exception
> >>>>> - The pages come off the printer as blank pages
> >>>>>
> >>>>>
> >>>>> Does anyone have any suggestions on how we can print using the
> >> printer's
> >>>>> full resolution?
> >>>>>
> >>>>> Thanks!
> >>>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> >>>> For additional commands, e-mail: users-help@pdfbox.apache.org
> >>>>
> >>>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> >> For additional commands, e-mail: users-help@pdfbox.apache.org
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> For additional commands, e-mail: users-help@pdfbox.apache.org
>
>
Re: Guidance on heap space error when printing document
Posted by Tilman Hausherr <TH...@t-online.de>.
If you're experienced with memory leaks then it would be nice if you
could search.
Things I tried that didn't help:
- calling graphics.setPaint(null) after operations (to "lose"
TilingPaint objects)
- disabling the (weak) cache of TilingPaint objects
- adding finalize to see if the TilingPaint class gets finalized (yes)
- adding finalize to see if the HighResolutionImageIcon class gets
finalized (yes)
Tilman
Am 05.11.2021 um 06:37 schrieb Kevin Day:
> Thanks for the analysis...
>
> So, how do we proceed? We are dealing with lots of different files from
> lots of different sources, so custom tuning maxedge makes me nervous.
>
>
> Is it worth trying BigBufferedImage? (I.e. would this even be accepted into
> the code base?) "Effective Handling of Big Images in Java"
> http://jobcardsystems.com/index.php/blog/29-effective-handling-of-big-images-in-java.
> - license is Creative Commons CCO. If licensing is a problem, then
> providing a hook to allow the BufferedImage creation to be overridden would
> be needed.
>
>
> Or should we be trying to profile to track down a memory leak?
>
> Awhile back, I dealt with native memory leaks in Java MappedByteBuffer
> under windows, but this was a native resource leak, and that doesn't seem
> like what we have here. And I don't know what the backing data store in
> ByteBuffer actually is.
>
> K
>
>
> On Thu, Nov 4, 2021, 9:52 PM Tilman Hausherr <TH...@t-online.de> wrote:
>
>> Thanks for the files.
>>
>> I wonder if there is some memory leak, in PDFBox or in java itself. The
>> H21 file has a pattern that contains an image (the company logo). That
>> pattern has an XStep and YStep of 23438 although the image is 2148 x
>> 440. Because of a matrix scale of 0.0673396 the image pattern size is
>> 1578 x 1578 at 72 dpi. So at 1200 dpi the size would be about 26300 x
>> 26300.
>>
>> You can limit this by setting pdfbox.rendering.tilingpaint.maxedge to a
>> smaller value than the default of 3000. However this may have a bad
>> influence on other files.
>>
>> I tried to show the H21 file in PDFDebugger at 1000% which would be 720
>> dpi. This worked until the last image. But no problem when displaying
>> the last image first. That is why I suspect a memory leak. I remember we
>> noticed something similar years ago, BufferedImage objects that stayed
>> despite nobody using them.
>>
>> Tilman
>>
>> Am 02.11.2021 um 20:06 schrieb Kevin Day:
>>> Sorry it took so long to respond - I had to get permission to share
>> these.
>>> I did test with 2.0.24 and it did not help.
>>>
>>>
>>> Here are some files that cause the problem:
>>>
>>> https://github.com/trumpetinc/boottest/blob/main/JrachvUniverse_H21.pdf
>>>
>> https://github.com/trumpetinc/boottest/blob/main/PP%20Opera%20Building_A53.pdf
>>>
>>> Here is some quick code that demonstrates the problem (update for your
>>> printer name and PDF path):
>>>
>>>
>> https://github.com/trumpetinc/boottest/blob/main/SimplifiedPdfBoxPdfPrinter.java
>>>
>>>
>>> Maybe in TilingPaint#getImage(PageDrawer, PDTilingPattern, PDColorSpace,
>>> PDColor, AffineTransform, Rectangle2D), the BufferedImage could
>>> benefit from something similar to BigBufferedImage described in this
>> post?
>>> (
>>>
>> https://stackoverflow.com/questions/3854200/why-does-a-bufferedimage-require-so-much-memory-beyond-the-size-of-its-data-arra
>>> ). Or maybe there is something in the scaling logic?
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Oct 28, 2021 at 11:12 AM Tilman Hausherr <TH...@t-online.de>
>>> wrote:
>>>
>>>> Huge buffered images can happen in patterns if the pattern matrix and/or
>>>> the current transformation matrix result in a huge scale.
>>>>
>>>> You could retry with 2.0.24, but I'm pessimistic. Please share the PDF
>>>> (upload to a sharehoster).
>>>>
>>>> Tilman
>>>>
>>>> Am 28.10.2021 um 01:17 schrieb Kevin Day:
>>>>> Using PDFBox 2.0.20
>>>>>
>>>>> We use the following to print the PDF:
>>>>>
>>>>> HashPrintRequestAttributeSet attributes = new
>>>>> HashPrintRequestAttributeSet();
>>>>>
>>>>> try(PDDocument doc = PDDocument.load(pdf,
>>>>> MemoryUsageSetting.setupMixed(5000000L))){
>>>>> job.setPageable(new PDFPageable(doc));
>>>>> job.print(attributes);
>>>>> }
>>>>>
>>>>> This works really well for thousands of PDFs, but we've run into one
>>>>> particular PDF that causes an OutOfMemoryException.
>>>>>
>>>>>
>>>>> The problem PDF is rendered (does not contain bitmaps).
>>>>>
>>>>> I've provided a huge amount of heap (over 1GB now), and it is still
>>>> failing.
>>>>> As near as I can tell from the stack trace (which I'll include below),
>> it
>>>>> seems like the problem is with creation of a huge buffered image.
>>>>>
>>>>> The physical page size in the PDF is 8.5 x 11".
>>>>>
>>>>> The PDFPrintable is configured as follows:
>>>>>
>>>>> Scaling is ACTUAL_SIZE
>>>>> dpi is 0.0
>>>>> subsamplingAllowed is false
>>>>> renderingHints is null
>>>>>
>>>>>
>>>>> Here is the stack trace:
>>>>>
>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>> at
>>>> java.desktop/java.awt.image.DataBufferInt.<init>(DataBufferInt.java:75)
>>>>> at
>> java.desktop/java.awt.image.Raster.createPackedRaster(Raster.java:467)
>>>>> at
>>>>>
>> java.desktop/java.awt.image.DirectColorModel.createCompatibleWritableRaster(DirectColorModel.java:1032)
>>>>> at
>>>> java.desktop/java.awt.image.BufferedImage.<init>(BufferedImage.java:333)
>>>>> at
>> org.apache.pdfbox.rendering.TilingPaint.getImage(TilingPaint.java:143)
>>>>> at org.apache.pdfbox.rendering.TilingPaint.<init>(TilingPaint.java:103)
>>>>> at
>>>>>
>> org.apache.pdfbox.rendering.TilingPaintFactory.create(TilingPaintFactory.java:60)
>>>>> at org.apache.pdfbox.rendering.PageDrawer.getPaint(PageDrawer.java:351)
>>>>> at
>>>>>
>> org.apache.pdfbox.rendering.PageDrawer.getNonStrokingPaint(PageDrawer.java:719)
>>>>> at org.apache.pdfbox.rendering.PageDrawer.fillPath(PageDrawer.java:819)
>>>>> at
>>>>>
>> org.apache.pdfbox.contentstream.operator.graphics.FillEvenOddRule.process(FillEvenOddRule.java:37)
>>>>> at
>>>>>
>> org.apache.pdfbox.contentstream.PDFStreamEngine.processOperator(PDFStreamEngine.java:932)
>>>>> at
>>>>>
>> org.apache.pdfbox.contentstream.PDFStreamEngine.processStreamOperators(PDFStreamEngine.java:510)
>>>>> at
>>>>>
>> org.apache.pdfbox.contentstream.PDFStreamEngine.processStream(PDFStreamEngine.java:484)
>>>>> at
>>>>>
>> org.apache.pdfbox.contentstream.PDFStreamEngine.showForm(PDFStreamEngine.java:187)
>>>>> at
>> org.apache.pdfbox.rendering.PageDrawer.showForm(PageDrawer.java:1462)
>>>>> at
>>>>>
>> org.apache.pdfbox.contentstream.operator.graphics.DrawObject.process(DrawObject.java:86)
>>>>> at
>>>>>
>> org.apache.pdfbox.contentstream.PDFStreamEngine.processOperator(PDFStreamEngine.java:932)
>>>>> at
>>>>>
>> org.apache.pdfbox.contentstream.PDFStreamEngine.processStreamOperators(PDFStreamEngine.java:510)
>>>>> at
>>>>>
>> org.apache.pdfbox.contentstream.PDFStreamEngine.processStream(PDFStreamEngine.java:484)
>>>>> at
>>>>>
>> org.apache.pdfbox.contentstream.PDFStreamEngine.processPage(PDFStreamEngine.java:156)
>>>>> at org.apache.pdfbox.rendering.PageDrawer.drawPage(PageDrawer.java:271)
>>>>> at
>>>>>
>> org.apache.pdfbox.rendering.PDFRenderer.renderPageToGraphics(PDFRenderer.java:431)
>>>>> at org.apache.pdfbox.printing.PDFPrintable.print(PDFPrintable.java:246)
>>>>> at
>>>>>
>> java.desktop/sun.print.RasterPrinterJob.printPage(RasterPrinterJob.java:2430)
>>>>> at
>> java.desktop/sun.print.RasterPrinterJob.print(RasterPrinterJob.java:1654)
>>>>>
>>>>> Next, I tried setting subsampling to true - this made no difference.
>>>>>
>>>>>
>>>>> Next, I tried hard coding the dpi to 300f - this works with no
>> problems.
>>>>>
>>>>> Next, I tried hard coding the dpi to 600f - this does not throw the
>> heap
>>>>> space error - but with strange results:
>>>>> - I see "java.lang.OutOfMemoryError" on sys.err - it does NOT throw an
>>>>> exception
>>>>> - The pages come off the printer as blank pages
>>>>>
>>>>>
>>>>> Does anyone have any suggestions on how we can print using the
>> printer's
>>>>> full resolution?
>>>>>
>>>>> Thanks!
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
>>>> For additional commands, e-mail: users-help@pdfbox.apache.org
>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
>> For additional commands, e-mail: users-help@pdfbox.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
For additional commands, e-mail: users-help@pdfbox.apache.org
Re: Guidance on heap space error when printing document
Posted by "sahyoun@fileaffairs.de" <sa...@fileaffairs.de>.
HiAm Donnerstag, dem 04.11.2021 um 22:37 -0700 schrieb Kevin Day:
> Thanks for the analysis...
>
> So, how do we proceed? We are dealing with lots of different files
> from
> lots of different sources, so custom tuning maxedge makes me nervous.
>
>
> Is it worth trying BigBufferedImage? (I.e. would this even be
> accepted into
> the code base?) "Effective Handling of Big Images in Java"
> http://jobcardsystems.com/index.php/blog/29-effective-handling-of-big-images-in-java
> .
> - license is Creative Commons CCO. If licensing is a problem, then
> providing a hook to allow the BufferedImage creation to be overridden
> would
> be needed.
>
>
> Or should we be trying to profile to track down a memory leak?
IMHO we should try tracking down the leak or rule that out before
looking at an alternative to BufferedImage - if the leak is in PDFBox
we would still have the leak when using the replacement only kicking in
at a later stage.
BR
Maruan
>
> Awhile back, I dealt with native memory leaks in Java
> MappedByteBuffer
> under windows, but this was a native resource leak, and that doesn't
> seem
> like what we have here. And I don't know what the backing data store
> in
> ByteBuffer actually is.
>
> K
>
>
> On Thu, Nov 4, 2021, 9:52 PM Tilman Hausherr <TH...@t-online.de>
> wrote:
>
> > Thanks for the files.
> >
> > I wonder if there is some memory leak, in PDFBox or in java itself.
> > The
> > H21 file has a pattern that contains an image (the company logo).
> > That
> > pattern has an XStep and YStep of 23438 although the image is 2148
> > x
> > 440. Because of a matrix scale of 0.0673396 the image pattern size
> > is
> > 1578 x 1578 at 72 dpi. So at 1200 dpi the size would be about 26300
> > x
> > 26300.
> >
> > You can limit this by setting pdfbox.rendering.tilingpaint.maxedge
> > to a
> > smaller value than the default of 3000. However this may have a bad
> > influence on other files.
> >
> > I tried to show the H21 file in PDFDebugger at 1000% which would be
> > 720
> > dpi. This worked until the last image. But no problem when
> > displaying
> > the last image first. That is why I suspect a memory leak. I
> > remember we
> > noticed something similar years ago, BufferedImage objects that
> > stayed
> > despite nobody using them.
> >
> > Tilman
> >
> > Am 02.11.2021 um 20:06 schrieb Kevin Day:
> > > Sorry it took so long to respond - I had to get permission to
> > > share
> > these.
> > >
> > > I did test with 2.0.24 and it did not help.
> > >
> > >
> > > Here are some files that cause the problem:
> > >
> > > https://github.com/trumpetinc/boottest/blob/main/JrachvUniverse_H21.pdf
> > >
> > https://github.com/trumpetinc/boottest/blob/main/PP%20Opera%20Building_A53.pdf
> > >
> > >
> > > Here is some quick code that demonstrates the problem (update for
> > > your
> > > printer name and PDF path):
> > >
> > >
> > https://github.com/trumpetinc/boottest/blob/main/SimplifiedPdfBoxPdfPrinter.java
> > >
> > >
> > >
> > > Maybe in TilingPaint#getImage(PageDrawer, PDTilingPattern,
> > > PDColorSpace,
> > > PDColor, AffineTransform, Rectangle2D), the BufferedImage could
> > > benefit from something similar to BigBufferedImage described in
> > > this
> > post?
> > > (
> > >
> > https://stackoverflow.com/questions/3854200/why-does-a-bufferedimage-require-so-much-memory-beyond-the-size-of-its-data-arra
> > > ). Or maybe there is something in the scaling logic?
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Oct 28, 2021 at 11:12 AM Tilman Hausherr
> > > <TH...@t-online.de>
> > > wrote:
> > >
> > > > Huge buffered images can happen in patterns if the pattern
> > > > matrix and/or
> > > > the current transformation matrix result in a huge scale.
> > > >
> > > > You could retry with 2.0.24, but I'm pessimistic. Please share
> > > > the PDF
> > > > (upload to a sharehoster).
> > > >
> > > > Tilman
> > > >
> > > > Am 28.10.2021 um 01:17 schrieb Kevin Day:
> > > > > Using PDFBox 2.0.20
> > > > >
> > > > > We use the following to print the PDF:
> > > > >
> > > > > Â Â Â Â Â HashPrintRequestAttributeSet attributes = new
> > > > > HashPrintRequestAttributeSet();
> > > > >
> > > > > Â Â Â Â Â try(PDDocument doc = PDDocument.load(pdf,
> > > > > MemoryUsageSetting.setupMixed(5000000L))){
> > > > > Â Â Â Â Â Â Â Â Â job.setPageable(new PDFPageable(doc));
> > > > > Â Â Â Â Â Â Â Â Â job.print(attributes);
> > > > > Â Â Â Â Â }
> > > > >
> > > > > This works really well for thousands of PDFs, but we've run
> > > > > into one
> > > > > particular PDF that causes an OutOfMemoryException.
> > > > >
> > > > >
> > > > > The problem PDF is rendered (does not contain bitmaps).
> > > > >
> > > > > I've provided a huge amount of heap (over 1GB now), and it is
> > > > > still
> > > > failing.
> > > > > As near as I can tell from the stack trace (which I'll
> > > > > include below),
> > it
> > > > > seems like the problem is with creation of a huge buffered
> > > > > image.
> > > > >
> > > > > The physical page size in the PDF is 8.5 x 11".
> > > > >
> > > > > The PDFPrintable is configured as follows:
> > > > >
> > > > > Scaling is ACTUAL_SIZE
> > > > > dpi is 0.0
> > > > > subsamplingAllowed is false
> > > > > renderingHints is null
> > > > >
> > > > >
> > > > > Here is the stack trace:
> > > > >
> > > > > java.lang.OutOfMemoryError: Java heap space
> > > > > at
> > > > java.desktop/java.awt.image.DataBufferInt.<init>(DataBufferInt.
> > > > java:75)
> > > > > at
> > java.desktop/java.awt.image.Raster.createPackedRaster(Raster.java:4
> > 67)
> > > > > at
> > > > >
> > > >
> > java.desktop/java.awt.image.DirectColorModel.createCompatibleWritab
> > leRaster(DirectColorModel.java:1032)
> > > > > at
> > > > java.desktop/java.awt.image.BufferedImage.<init>(BufferedImage.
> > > > java:333)
> > > > > at
> > org.apache.pdfbox.rendering.TilingPaint.getImage(TilingPaint.java:1
> > 43)
> > > > > at
> > > > > org.apache.pdfbox.rendering.TilingPaint.<init>(TilingPaint.ja
> > > > > va:103)
> > > > > at
> > > > >
> > > >
> > org.apache.pdfbox.rendering.TilingPaintFactory.create(TilingPaintFa
> > ctory.java:60)
> > > > > at
> > > > > org.apache.pdfbox.rendering.PageDrawer.getPaint(PageDrawer.ja
> > > > > va:351)
> > > > > at
> > > > >
> > > >
> > org.apache.pdfbox.rendering.PageDrawer.getNonStrokingPaint(PageDraw
> > er.java:719)
> > > > > at
> > > > > org.apache.pdfbox.rendering.PageDrawer.fillPath(PageDrawer.ja
> > > > > va:819)
> > > > > at
> > > > >
> > > >
> > org.apache.pdfbox.contentstream.operator.graphics.FillEvenOddRule.p
> > rocess(FillEvenOddRule.java:37)
> > > > > at
> > > > >
> > > >
> > org.apache.pdfbox.contentstream.PDFStreamEngine.processOperator(PDF
> > StreamEngine.java:932)
> > > > > at
> > > > >
> > > >
> > org.apache.pdfbox.contentstream.PDFStreamEngine.processStreamOperat
> > ors(PDFStreamEngine.java:510)
> > > > > at
> > > > >
> > > >
> > org.apache.pdfbox.contentstream.PDFStreamEngine.processStream(PDFSt
> > reamEngine.java:484)
> > > > > at
> > > > >
> > > >
> > org.apache.pdfbox.contentstream.PDFStreamEngine.showForm(PDFStreamE
> > ngine.java:187)
> > > > > at
> > org.apache.pdfbox.rendering.PageDrawer.showForm(PageDrawer.java:146
> > 2)
> > > > > at
> > > > >
> > > >
> > org.apache.pdfbox.contentstream.operator.graphics.DrawObject.proces
> > s(DrawObject.java:86)
> > > > > at
> > > > >
> > > >
> > org.apache.pdfbox.contentstream.PDFStreamEngine.processOperator(PDF
> > StreamEngine.java:932)
> > > > > at
> > > > >
> > > >
> > org.apache.pdfbox.contentstream.PDFStreamEngine.processStreamOperat
> > ors(PDFStreamEngine.java:510)
> > > > > at
> > > > >
> > > >
> > org.apache.pdfbox.contentstream.PDFStreamEngine.processStream(PDFSt
> > reamEngine.java:484)
> > > > > at
> > > > >
> > > >
> > org.apache.pdfbox.contentstream.PDFStreamEngine.processPage(PDFStre
> > amEngine.java:156)
> > > > > at
> > > > > org.apache.pdfbox.rendering.PageDrawer.drawPage(PageDrawer.ja
> > > > > va:271)
> > > > > at
> > > > >
> > > >
> > org.apache.pdfbox.rendering.PDFRenderer.renderPageToGraphics(PDFRen
> > derer.java:431)
> > > > > at
> > > > > org.apache.pdfbox.printing.PDFPrintable.print(PDFPrintable.ja
> > > > > va:246)
> > > > > at
> > > > >
> > > >
> > java.desktop/sun.print.RasterPrinterJob.printPage(RasterPrinterJob.
> > java:2430)
> > > > > at
> > > >
> > java.desktop/sun.print.RasterPrinterJob.print(RasterPrinterJob.java
> > :1654)
> > > > >
> > > > >
> > > > > Next, I tried setting subsampling to true - this made no
> > > > > difference.
> > > > >
> > > > >
> > > > > Next, I tried hard coding the dpi to 300f - this works with
> > > > > no
> > problems.
> > > > >
> > > > >
> > > > > Next, I tried hard coding the dpi to 600f - this does not
> > > > > throw the
> > heap
> > > > > space error - but with strange results:
> > > > > - I see "java.lang.OutOfMemoryError" on sys.err - it does NOT
> > > > > throw an
> > > > > exception
> > > > > - The pages come off the printer as blank pages
> > > > >
> > > > >
> > > > > Does anyone have any suggestions on how we can print using
> > > > > the
> > printer's
> > > > > full resolution?
> > > > >
> > > > > Thanks!
> > > > >
> > > >
> > > > ---------------------------------------------------------------
> > > > ------
> > > > To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> > > > For additional commands, e-mail: users-help@pdfbox.apache.org
> > > >
> > > >
> >
> >
> > -------------------------------------------------------------------
> > --
> > To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> > For additional commands, e-mail: users-help@pdfbox.apache.org
> >
> >
--
--Â
Maruan Sahyoun
FileAffairs GmbH
Josef-Schappe-Straße 21
40882 Ratingen
Tel: +49 (2102) 89497 88
Fax: +49 (2102) 89497 91
sahyoun@fileaffairs.de
www.fileaffairs.de
Geschäftsführer: Maruan Sahyoun
Handelsregister: AG Düsseldorf, HRB 53837
UST.-ID: DE248275827
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
For additional commands, e-mail: users-help@pdfbox.apache.org
Re: Guidance on heap space error when printing document
Posted by Kevin Day <ke...@trumpetinc.com>.
Thanks for the analysis...
So, how do we proceed? We are dealing with lots of different files from
lots of different sources, so custom tuning maxedge makes me nervous.
Is it worth trying BigBufferedImage? (I.e. would this even be accepted into
the code base?) "Effective Handling of Big Images in Java"
http://jobcardsystems.com/index.php/blog/29-effective-handling-of-big-images-in-java.
- license is Creative Commons CCO. If licensing is a problem, then
providing a hook to allow the BufferedImage creation to be overridden would
be needed.
Or should we be trying to profile to track down a memory leak?
Awhile back, I dealt with native memory leaks in Java MappedByteBuffer
under windows, but this was a native resource leak, and that doesn't seem
like what we have here. And I don't know what the backing data store in
ByteBuffer actually is.
K
On Thu, Nov 4, 2021, 9:52 PM Tilman Hausherr <TH...@t-online.de> wrote:
> Thanks for the files.
>
> I wonder if there is some memory leak, in PDFBox or in java itself. The
> H21 file has a pattern that contains an image (the company logo). That
> pattern has an XStep and YStep of 23438 although the image is 2148 x
> 440. Because of a matrix scale of 0.0673396 the image pattern size is
> 1578 x 1578 at 72 dpi. So at 1200 dpi the size would be about 26300 x
> 26300.
>
> You can limit this by setting pdfbox.rendering.tilingpaint.maxedge to a
> smaller value than the default of 3000. However this may have a bad
> influence on other files.
>
> I tried to show the H21 file in PDFDebugger at 1000% which would be 720
> dpi. This worked until the last image. But no problem when displaying
> the last image first. That is why I suspect a memory leak. I remember we
> noticed something similar years ago, BufferedImage objects that stayed
> despite nobody using them.
>
> Tilman
>
> Am 02.11.2021 um 20:06 schrieb Kevin Day:
> > Sorry it took so long to respond - I had to get permission to share
> these.
> >
> > I did test with 2.0.24 and it did not help.
> >
> >
> > Here are some files that cause the problem:
> >
> > https://github.com/trumpetinc/boottest/blob/main/JrachvUniverse_H21.pdf
> >
> https://github.com/trumpetinc/boottest/blob/main/PP%20Opera%20Building_A53.pdf
> >
> >
> > Here is some quick code that demonstrates the problem (update for your
> > printer name and PDF path):
> >
> >
> https://github.com/trumpetinc/boottest/blob/main/SimplifiedPdfBoxPdfPrinter.java
> >
> >
> >
> > Maybe in TilingPaint#getImage(PageDrawer, PDTilingPattern, PDColorSpace,
> > PDColor, AffineTransform, Rectangle2D), the BufferedImage could
> > benefit from something similar to BigBufferedImage described in this
> post?
> > (
> >
> https://stackoverflow.com/questions/3854200/why-does-a-bufferedimage-require-so-much-memory-beyond-the-size-of-its-data-arra
> > ). Or maybe there is something in the scaling logic?
> >
> >
> >
> >
> >
> >
> > On Thu, Oct 28, 2021 at 11:12 AM Tilman Hausherr <TH...@t-online.de>
> > wrote:
> >
> >> Huge buffered images can happen in patterns if the pattern matrix and/or
> >> the current transformation matrix result in a huge scale.
> >>
> >> You could retry with 2.0.24, but I'm pessimistic. Please share the PDF
> >> (upload to a sharehoster).
> >>
> >> Tilman
> >>
> >> Am 28.10.2021 um 01:17 schrieb Kevin Day:
> >>> Using PDFBox 2.0.20
> >>>
> >>> We use the following to print the PDF:
> >>>
> >>> HashPrintRequestAttributeSet attributes = new
> >>> HashPrintRequestAttributeSet();
> >>>
> >>> try(PDDocument doc = PDDocument.load(pdf,
> >>> MemoryUsageSetting.setupMixed(5000000L))){
> >>> job.setPageable(new PDFPageable(doc));
> >>> job.print(attributes);
> >>> }
> >>>
> >>> This works really well for thousands of PDFs, but we've run into one
> >>> particular PDF that causes an OutOfMemoryException.
> >>>
> >>>
> >>> The problem PDF is rendered (does not contain bitmaps).
> >>>
> >>> I've provided a huge amount of heap (over 1GB now), and it is still
> >> failing.
> >>> As near as I can tell from the stack trace (which I'll include below),
> it
> >>> seems like the problem is with creation of a huge buffered image.
> >>>
> >>> The physical page size in the PDF is 8.5 x 11".
> >>>
> >>> The PDFPrintable is configured as follows:
> >>>
> >>> Scaling is ACTUAL_SIZE
> >>> dpi is 0.0
> >>> subsamplingAllowed is false
> >>> renderingHints is null
> >>>
> >>>
> >>> Here is the stack trace:
> >>>
> >>> java.lang.OutOfMemoryError: Java heap space
> >>> at
> >> java.desktop/java.awt.image.DataBufferInt.<init>(DataBufferInt.java:75)
> >>> at
> java.desktop/java.awt.image.Raster.createPackedRaster(Raster.java:467)
> >>> at
> >>>
> >>
> java.desktop/java.awt.image.DirectColorModel.createCompatibleWritableRaster(DirectColorModel.java:1032)
> >>> at
> >> java.desktop/java.awt.image.BufferedImage.<init>(BufferedImage.java:333)
> >>> at
> org.apache.pdfbox.rendering.TilingPaint.getImage(TilingPaint.java:143)
> >>> at org.apache.pdfbox.rendering.TilingPaint.<init>(TilingPaint.java:103)
> >>> at
> >>>
> >>
> org.apache.pdfbox.rendering.TilingPaintFactory.create(TilingPaintFactory.java:60)
> >>> at org.apache.pdfbox.rendering.PageDrawer.getPaint(PageDrawer.java:351)
> >>> at
> >>>
> >>
> org.apache.pdfbox.rendering.PageDrawer.getNonStrokingPaint(PageDrawer.java:719)
> >>> at org.apache.pdfbox.rendering.PageDrawer.fillPath(PageDrawer.java:819)
> >>> at
> >>>
> >>
> org.apache.pdfbox.contentstream.operator.graphics.FillEvenOddRule.process(FillEvenOddRule.java:37)
> >>> at
> >>>
> >>
> org.apache.pdfbox.contentstream.PDFStreamEngine.processOperator(PDFStreamEngine.java:932)
> >>> at
> >>>
> >>
> org.apache.pdfbox.contentstream.PDFStreamEngine.processStreamOperators(PDFStreamEngine.java:510)
> >>> at
> >>>
> >>
> org.apache.pdfbox.contentstream.PDFStreamEngine.processStream(PDFStreamEngine.java:484)
> >>> at
> >>>
> >>
> org.apache.pdfbox.contentstream.PDFStreamEngine.showForm(PDFStreamEngine.java:187)
> >>> at
> org.apache.pdfbox.rendering.PageDrawer.showForm(PageDrawer.java:1462)
> >>> at
> >>>
> >>
> org.apache.pdfbox.contentstream.operator.graphics.DrawObject.process(DrawObject.java:86)
> >>> at
> >>>
> >>
> org.apache.pdfbox.contentstream.PDFStreamEngine.processOperator(PDFStreamEngine.java:932)
> >>> at
> >>>
> >>
> org.apache.pdfbox.contentstream.PDFStreamEngine.processStreamOperators(PDFStreamEngine.java:510)
> >>> at
> >>>
> >>
> org.apache.pdfbox.contentstream.PDFStreamEngine.processStream(PDFStreamEngine.java:484)
> >>> at
> >>>
> >>
> org.apache.pdfbox.contentstream.PDFStreamEngine.processPage(PDFStreamEngine.java:156)
> >>> at org.apache.pdfbox.rendering.PageDrawer.drawPage(PageDrawer.java:271)
> >>> at
> >>>
> >>
> org.apache.pdfbox.rendering.PDFRenderer.renderPageToGraphics(PDFRenderer.java:431)
> >>> at org.apache.pdfbox.printing.PDFPrintable.print(PDFPrintable.java:246)
> >>> at
> >>>
> >>
> java.desktop/sun.print.RasterPrinterJob.printPage(RasterPrinterJob.java:2430)
> >>> at
> >>
> java.desktop/sun.print.RasterPrinterJob.print(RasterPrinterJob.java:1654)
> >>>
> >>>
> >>> Next, I tried setting subsampling to true - this made no difference.
> >>>
> >>>
> >>> Next, I tried hard coding the dpi to 300f - this works with no
> problems.
> >>>
> >>>
> >>> Next, I tried hard coding the dpi to 600f - this does not throw the
> heap
> >>> space error - but with strange results:
> >>> - I see "java.lang.OutOfMemoryError" on sys.err - it does NOT throw an
> >>> exception
> >>> - The pages come off the printer as blank pages
> >>>
> >>>
> >>> Does anyone have any suggestions on how we can print using the
> printer's
> >>> full resolution?
> >>>
> >>> Thanks!
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> >> For additional commands, e-mail: users-help@pdfbox.apache.org
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> For additional commands, e-mail: users-help@pdfbox.apache.org
>
>
Re: Guidance on heap space error when printing document
Posted by Tilman Hausherr <TH...@t-online.de>.
Thanks for the files.
I wonder if there is some memory leak, in PDFBox or in java itself. The
H21 file has a pattern that contains an image (the company logo). That
pattern has an XStep and YStep of 23438 although the image is 2148 x
440. Because of a matrix scale of 0.0673396 the image pattern size is
1578 x 1578 at 72 dpi. So at 1200 dpi the size would be about 26300 x 26300.
You can limit this by setting pdfbox.rendering.tilingpaint.maxedge to a
smaller value than the default of 3000. However this may have a bad
influence on other files.
I tried to show the H21 file in PDFDebugger at 1000% which would be 720
dpi. This worked until the last image. But no problem when displaying
the last image first. That is why I suspect a memory leak. I remember we
noticed something similar years ago, BufferedImage objects that stayed
despite nobody using them.
Tilman
Am 02.11.2021 um 20:06 schrieb Kevin Day:
> Sorry it took so long to respond - I had to get permission to share these.
>
> I did test with 2.0.24 and it did not help.
>
>
> Here are some files that cause the problem:
>
> https://github.com/trumpetinc/boottest/blob/main/JrachvUniverse_H21.pdf
> https://github.com/trumpetinc/boottest/blob/main/PP%20Opera%20Building_A53.pdf
>
>
> Here is some quick code that demonstrates the problem (update for your
> printer name and PDF path):
>
> https://github.com/trumpetinc/boottest/blob/main/SimplifiedPdfBoxPdfPrinter.java
>
>
>
> Maybe in TilingPaint#getImage(PageDrawer, PDTilingPattern, PDColorSpace,
> PDColor, AffineTransform, Rectangle2D), the BufferedImage could
> benefit from something similar to BigBufferedImage described in this post?
> (
> https://stackoverflow.com/questions/3854200/why-does-a-bufferedimage-require-so-much-memory-beyond-the-size-of-its-data-arra
> ). Or maybe there is something in the scaling logic?
>
>
>
>
>
>
> On Thu, Oct 28, 2021 at 11:12 AM Tilman Hausherr <TH...@t-online.de>
> wrote:
>
>> Huge buffered images can happen in patterns if the pattern matrix and/or
>> the current transformation matrix result in a huge scale.
>>
>> You could retry with 2.0.24, but I'm pessimistic. Please share the PDF
>> (upload to a sharehoster).
>>
>> Tilman
>>
>> Am 28.10.2021 um 01:17 schrieb Kevin Day:
>>> Using PDFBox 2.0.20
>>>
>>> We use the following to print the PDF:
>>>
>>> HashPrintRequestAttributeSet attributes = new
>>> HashPrintRequestAttributeSet();
>>>
>>> try(PDDocument doc = PDDocument.load(pdf,
>>> MemoryUsageSetting.setupMixed(5000000L))){
>>> job.setPageable(new PDFPageable(doc));
>>> job.print(attributes);
>>> }
>>>
>>> This works really well for thousands of PDFs, but we've run into one
>>> particular PDF that causes an OutOfMemoryException.
>>>
>>>
>>> The problem PDF is rendered (does not contain bitmaps).
>>>
>>> I've provided a huge amount of heap (over 1GB now), and it is still
>> failing.
>>> As near as I can tell from the stack trace (which I'll include below), it
>>> seems like the problem is with creation of a huge buffered image.
>>>
>>> The physical page size in the PDF is 8.5 x 11".
>>>
>>> The PDFPrintable is configured as follows:
>>>
>>> Scaling is ACTUAL_SIZE
>>> dpi is 0.0
>>> subsamplingAllowed is false
>>> renderingHints is null
>>>
>>>
>>> Here is the stack trace:
>>>
>>> java.lang.OutOfMemoryError: Java heap space
>>> at
>> java.desktop/java.awt.image.DataBufferInt.<init>(DataBufferInt.java:75)
>>> at java.desktop/java.awt.image.Raster.createPackedRaster(Raster.java:467)
>>> at
>>>
>> java.desktop/java.awt.image.DirectColorModel.createCompatibleWritableRaster(DirectColorModel.java:1032)
>>> at
>> java.desktop/java.awt.image.BufferedImage.<init>(BufferedImage.java:333)
>>> at org.apache.pdfbox.rendering.TilingPaint.getImage(TilingPaint.java:143)
>>> at org.apache.pdfbox.rendering.TilingPaint.<init>(TilingPaint.java:103)
>>> at
>>>
>> org.apache.pdfbox.rendering.TilingPaintFactory.create(TilingPaintFactory.java:60)
>>> at org.apache.pdfbox.rendering.PageDrawer.getPaint(PageDrawer.java:351)
>>> at
>>>
>> org.apache.pdfbox.rendering.PageDrawer.getNonStrokingPaint(PageDrawer.java:719)
>>> at org.apache.pdfbox.rendering.PageDrawer.fillPath(PageDrawer.java:819)
>>> at
>>>
>> org.apache.pdfbox.contentstream.operator.graphics.FillEvenOddRule.process(FillEvenOddRule.java:37)
>>> at
>>>
>> org.apache.pdfbox.contentstream.PDFStreamEngine.processOperator(PDFStreamEngine.java:932)
>>> at
>>>
>> org.apache.pdfbox.contentstream.PDFStreamEngine.processStreamOperators(PDFStreamEngine.java:510)
>>> at
>>>
>> org.apache.pdfbox.contentstream.PDFStreamEngine.processStream(PDFStreamEngine.java:484)
>>> at
>>>
>> org.apache.pdfbox.contentstream.PDFStreamEngine.showForm(PDFStreamEngine.java:187)
>>> at org.apache.pdfbox.rendering.PageDrawer.showForm(PageDrawer.java:1462)
>>> at
>>>
>> org.apache.pdfbox.contentstream.operator.graphics.DrawObject.process(DrawObject.java:86)
>>> at
>>>
>> org.apache.pdfbox.contentstream.PDFStreamEngine.processOperator(PDFStreamEngine.java:932)
>>> at
>>>
>> org.apache.pdfbox.contentstream.PDFStreamEngine.processStreamOperators(PDFStreamEngine.java:510)
>>> at
>>>
>> org.apache.pdfbox.contentstream.PDFStreamEngine.processStream(PDFStreamEngine.java:484)
>>> at
>>>
>> org.apache.pdfbox.contentstream.PDFStreamEngine.processPage(PDFStreamEngine.java:156)
>>> at org.apache.pdfbox.rendering.PageDrawer.drawPage(PageDrawer.java:271)
>>> at
>>>
>> org.apache.pdfbox.rendering.PDFRenderer.renderPageToGraphics(PDFRenderer.java:431)
>>> at org.apache.pdfbox.printing.PDFPrintable.print(PDFPrintable.java:246)
>>> at
>>>
>> java.desktop/sun.print.RasterPrinterJob.printPage(RasterPrinterJob.java:2430)
>>> at
>> java.desktop/sun.print.RasterPrinterJob.print(RasterPrinterJob.java:1654)
>>>
>>>
>>> Next, I tried setting subsampling to true - this made no difference.
>>>
>>>
>>> Next, I tried hard coding the dpi to 300f - this works with no problems.
>>>
>>>
>>> Next, I tried hard coding the dpi to 600f - this does not throw the heap
>>> space error - but with strange results:
>>> - I see "java.lang.OutOfMemoryError" on sys.err - it does NOT throw an
>>> exception
>>> - The pages come off the printer as blank pages
>>>
>>>
>>> Does anyone have any suggestions on how we can print using the printer's
>>> full resolution?
>>>
>>> Thanks!
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
>> For additional commands, e-mail: users-help@pdfbox.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
For additional commands, e-mail: users-help@pdfbox.apache.org